feat(devcontainer): bake velero CLI into the toolbox, pinned to prod server v1.18.2

The velero CLI and server are version-coupled, so VELERO_VERSION follows the
server running on prod (docker.io/velero/velero:v1.18.2, chart 12.2.3,
verified 2026-11-07). It has no renovate marker on purpose: bump it together
with the chart's appVersion. The verify stage asserts that
`velero version --client-only` reports the pin. The build arg reaches CI
through build-image.sh --print-build-args, so no .gitea/ or ci/ change is
needed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

.gitea/workflows/README.md

@@ -115,14 +115,59 @@ CI. That is a real coverage gap, not a phase.
 | variable | `REQUIRE_CLUSTER_TESTS` | `'true'` (Phase 1+) makes unverified cluster-needing touched reqs a hard block |
 | secret | `CONFORMANCE_TRIGGER_KUBECONFIG` | narrow creds to create/await the in-cluster Chainsaw Job (not cluster-admin) |
 
+## How images are built
+
+This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`,
+built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built,
+verified and published by [`images.yaml`](images.yaml), never from a workstation
+(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh).
+
+**The builder is rootless BuildKit** ([`platform/buildkit/`](../../platform/buildkit/README.md)),
+reached over mTLS with a pinned `buildx` — no job here touches a Docker daemon, privileged or
+otherwise (operator decision on PR #81). The client certificate reaches jobs as the Actions
+secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT`, `BUILDKIT_CLIENT_KEY`.
+
+**Verification is a build target.** Each Containerfile ends in a `verify` stage that runs every
+in-image assertion at build time as the non-root uid with `RUN --network=none` (tool presence and
+pinned versions, `PM_SELFTEST=1 pm-dispatch`, the pm-worker probe, the `conformance-clone`
+refusals, no writable path outside the workspace). The pipeline builds `<image>-verify` before it
+builds, let alone pushes, the image. Build args come from each image's own
+`build-image.sh --print-build-args`, which is also the local podman path — so the recipe and the
+proof exist once.
+
+| Trigger | What happens |
+|---|---|
+| pull request touching an image's inputs | build + verify only. Nothing is pushed and **no registry credential is referenced** |
+| push to `main` touching an image's inputs | build + verify + publish to `gitea.home.keithjackson.dev/home-network/<image>` as `sha-<commit>` (immutable) and `main` (moving); the **registry-returned** digest is in the job summary |
+| `workflow_dispatch` | the same as a push when run on `main` (all images by default); verify-only elsewhere |
+
+Only images whose inputs changed are built. The publish job **never writes to git**: Renovate
+follows each image's `main` tag through a `# renovate-digest:` marker next to every digest
+reference (`platform/pm-agent/`, `infra/agent/versions.env`, `conformance/versions.env`) and
+opens the bump PR, which goes through the gate like anything else. A PR opened with the Actions
+token would not trigger the required gate, which is why it is Renovate's job and not this one's.
+
+Credential: the repository secret **`REGISTRY_PUSH_TOKEN`** (scope `write:package`, owned by an
+account with package write on the org). The automatic Actions token cannot push in Gitea 1.24.3
+— the reasoning, with source citations, is in the header of `images.yaml`. Retention:
+[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator
+action, keeps every digest git pins plus the newest tags.
+
+Cost: the job shares the single act_runner slot with the gate, but the build itself runs in the
+`buildkitd` pod, not in DinD, so the runner's disk and memory are untouched. Until
+`platform/buildkit` is deployed and its client certificate copied
+(`infra/gitea/buildkit-client-secret.sh`), the job fails loudly at its builder preflight.
+
 ## Files
 
 ```
 .gitea/workflows/
   merge-gate.yaml            the pipeline (lint → l0 → l1 → conformance → decision)
+  images.yaml                build/verify/publish this repo's container images (see above)
   scripts/
     install-tools.sh         installs the pinned tools per stage (from ci/versions.env)
     decide.sh                the R-TEST-03 merge decision over job results
+    images.sh                the image pipeline's driver (what to build, base, push, digest)
   README.md                  this file
 ```
 

.gitea/workflows/gate.yml

@@ -80,7 +80,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with:
           fetch-depth: 0        # full history so ci/touched-requirements.sh can diff vs main
 

.gitea/workflows/images.yaml

@@ -0,0 +1,185 @@
+# Container images — built, verified and published by CI, never from a workstation.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY THIS FILE EXISTS (operator rule, 2026-10-29)
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# "Container image creation, hosting and lifecycle must be part of CI/CD and the codebase, never a
+# one-off build/push from a workstation." Before this workflow, three images existed only as
+# workstation builds and NOTHING had ever been pushed to the Gitea registry: the Home-Network org's
+# package list was empty, platform/pm-agent/ pinned an all-zero digest and sat in ImagePullBackOff,
+# and conformance/versions.env carried an empty digest. Both build scripts said "publishing is an
+# operator action" — a step with no log, no recorded inputs and no verification anyone could read.
+#
+# WHAT IT DOES — the logic is in scripts/images.sh; this file only wires triggers and secrets.
+#   pull_request -> build + VERIFY only. No push, and no registry credential is referenced.
+#   push to main -> build + verify + PUBLISH to gitea.home.keithjackson.dev/home-network/<image>
+#   dispatch     -> same as a push when run on main; verify-only on any other ref.
+# Images: toolbox (infra/devcontainer), agent-runner (infra/agent, FROM toolbox) and
+# conformance-runner (conformance/incluster). Only images whose inputs changed are built.
+#
+# THE DIGEST LIFECYCLE — this workflow NEVER writes to git.
+#   1. publish pushes `sha-<commit>` (immutable) and `main` (moving), and prints the digest the
+#      REGISTRY returns in the job summary.
+#   2. Renovate (renovate.yaml) follows the `main` tag of each image through the
+#      `# renovate-digest:` markers next to every digest reference, and opens the bump PR.
+#   3. That PR goes through the gate like any other change, and Flux deploys it.
+# The bump PR is Renovate's and not this job's because a PR opened with the Actions token does not
+# trigger workflows, so it could never pass the REQUIRED gate check; Renovate's token is a real
+# user's.
+#
+# THE PUSH CREDENTIAL — `REGISTRY_PUSH_TOKEN`, a repository Actions secret, referenced BY NAME.
+# The automatic Actions token CANNOT be used. Gitea 1.24.3 source, verified:
+#   * services/auth/basic.go (Basic.Verify) turns a task token into the pseudo-user
+#     `gitea-actions` (ActionsUserID -2) and sets NO token scope;
+#   * routers/api/packages/container/container.go (Authenticate, the /v2/token exchange every
+#     docker client uses) then requires read:package / write:package / all scope and answers 401;
+#   * services/context/package.go (determineAccessMode) has `// TODO: ActionUser permission check`
+#     and no branch for that user, so even past login it would get at most READ on a public org.
+# So: a personal access token with scope `write:package`, minted by an account that has package
+# WRITE on the Home-Network org (org owner, or a team with the Packages unit at write). A dedicated
+# bot account in a packages-only team is the least-privilege choice. The username sent with it is
+# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
+# is optional and only labels the login.
+#
+# THE BUILDER — rootless BuildKit (operator decision on PR #81, 2026-10-29).
+# The act runner's job containers get NO Docker daemon (act_runner 0.2.11 forces docker_host "-"
+# for a tcp:// DinD, and gitea/act strips every bind when valid_volumes is empty — measured on
+# this PR's first run), and handing them the privileged DinD daemon would be root on the node for
+# anyone who can push a branch. So images are built by platform/buildkit/: a ROOTLESS buildkitd
+# (not privileged; uid 1000 in a user namespace), reached over mTLS at
+# buildkitd.buildkit.svc.cluster.local:1234 with a pinned `buildx` (remote driver). Its client
+# certificate reaches jobs as three Actions secrets — BUILDKIT_CA, BUILDKIT_CLIENT_CERT,
+# BUILDKIT_CLIENT_KEY — copied from cert-manager by infra/gitea/buildkit-client-secret.sh.
+# Pull requests get the client certificate (they must build) but never a registry credential.
+#
+# UNTIL platform/buildkit IS DEPLOYED (Flux applies it only after merge) THIS JOB FAILS, LOUDLY,
+# at its builder preflight. A skipped Gitea job reports `success`, so it is red on purpose.
+#
+# VERIFICATION is each Containerfile's `verify` stage, built before anything is pushed.
+#
+name: images
+
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - infra/devcontainer/Containerfile
+      - infra/devcontainer/versions.env
+      - infra/devcontainer/build-image.sh
+      - infra/agent/**
+      - conformance/incluster/Containerfile
+      - conformance/incluster/conformance-clone
+      - conformance/incluster/build-image.sh
+      - .gitea/workflows/images.yaml
+      - .gitea/workflows/scripts/images.sh
+      - platform/buildkit/versions.env
+  push:
+    branches: [main]
+    paths:
+      - infra/devcontainer/Containerfile
+      - infra/devcontainer/versions.env
+      - infra/devcontainer/build-image.sh
+      - infra/agent/**
+      - conformance/incluster/Containerfile
+      - conformance/incluster/conformance-clone
+      - conformance/incluster/build-image.sh
+      - .gitea/workflows/images.yaml
+      - .gitea/workflows/scripts/images.sh
+      - platform/buildkit/versions.env
+  workflow_dispatch:
+    inputs:
+      images:
+        description: "Which images: 'all', or a space-separated subset of: toolbox agent-runner conformance-runner"
+        required: false
+        default: "all"
+
+concurrency:
+  # Never cancel a publish half-way: a pushed `sha-` tag with no matching `main` move is
+  # recoverable (a rerun skips what is already published) but pointless to provoke.
+  group: images-${{ github.ref }}
+  cancel-in-progress: false
+
+jobs:
+  images:
+    runs-on: ubuntu-latest
+    # Measured on the PR that introduced this file — see its decisions entry. Generous because a
+    # cold DinD store downloads the Fedora base and every pinned tool on each run.
+    timeout-minutes: 75
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+        with:
+          fetch-depth: 0   # the per-image change detection diffs against the base commit
+
+      # ── pull requests: build + verify, NO credential ───────────────────────────────────────
+      - name: Build and verify (pull request — nothing is pushed)
+        if: github.event_name == 'pull_request'
+        env:
+          MODE: verify
+          IMAGES: auto
+          PR_BASE_REF: ${{ github.base_ref }}
+          # The BUILDER's client certificate (platform/buildkit). Not a registry credential:
+          # a pull request can build, never push.
+          BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
+          BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
+          BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
+        run: |
+          set -euo pipefail
+          BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)"
+          export BASE_SHA
+          ./.gitea/workflows/scripts/images.sh
+
+      # ── non-main dispatch: verify only ──────────────────────────────────────────────────────
+      - name: Build and verify (dispatch off main — nothing is pushed)
+        if: github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main'
+        env:
+          MODE: verify
+          IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }}
+          BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
+          BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
+          BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
+        run: ./.gitea/workflows/scripts/images.sh
+
+      # ── main: publish ───────────────────────────────────────────────────────────────────────
+      - name: Preflight - the push token must exist, and its absence must be LOUD
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        env:
+          REGISTRY_PUSH_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${REGISTRY_PUSH_TOKEN:-}" ]; then
+            echo "::error::REGISTRY_PUSH_TOKEN is not set on this repository." >&2
+            cat >&2 <<'MSG'
+
+          Images cannot be published without a registry credential, and this step fails rather
+          than skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would
+          make "nothing was published" look exactly like "published". That is how this repo went
+          a month with images that existed only on one workstation.
+
+          WHY NOT THE BUILT-IN ACTIONS TOKEN: Gitea 1.24.3 gives it no package scope at the
+          registry's /v2/token exchange (routers/api/packages/container/container.go,
+          Authenticate) — see the header of .gitea/workflows/images.yaml.
+
+          TO FIX: as an account with package WRITE on the Home-Network org (ideally a dedicated
+          bot in a packages-only team), create an access token with scope `write:package` and
+          nothing else, then add it to this repository as the Actions secret
+          REGISTRY_PUSH_TOKEN. Optionally set the Actions variable REGISTRY_PUSH_USER to that
+          account's name. Do not commit the token anywhere. Then re-run this workflow
+          (workflow_dispatch on main, images: all).
+          MSG
+            exit 1
+          fi
+          echo "REGISTRY_PUSH_TOKEN is present (value not printed)."
+
+      - name: Build, verify and publish (main)
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        env:
+          MODE: publish
+          IMAGES: ${{ github.event_name == 'workflow_dispatch' && (inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images) || 'auto' }}
+          BASE_SHA: ${{ github.event.before }}
+          REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
+          REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }}
+          BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
+          BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
+          BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
+        run: ./.gitea/workflows/scripts/images.sh

.gitea/workflows/merge-gate.yaml

@@ -60,9 +60,9 @@ jobs:
   spec-immutability:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4                 # the implementation repo
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1                 # v7                 # the implementation repo
       - name: Check out the read-only spec (sibling path)
-        uses: actions/checkout@v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with:
           repository: ${{ vars.SPEC_REPO_SLUG }}  # e.g. Home-Network/home-network-k3s (set in Gitea)
           path: ../home-network-k3s
@@ -74,7 +74,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [spec-immutability]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }          # full history so touched-requirement diff works
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh lint
@@ -85,7 +85,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [lint]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l0
@@ -96,7 +96,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [l0]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l1
@@ -107,7 +107,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [render-golden]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l1
@@ -115,7 +115,7 @@ jobs:
         run: make -C ci l1
       - name: Upload L1 artifacts
         if: always()
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
         with:
           name: l1-junit-coverage
           path: |
@@ -131,7 +131,7 @@ jobs:
     needs: [l1]
     if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh conformance
@@ -142,7 +142,7 @@ jobs:
         run: make -C ci conformance
       - name: Upload L2/L3 artifacts
         if: always()
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
         with:
           name: conformance-junit-coverage
           path: |
@@ -160,7 +160,7 @@ jobs:
     needs: [lint, l0, render-golden, l1, conformance]
     if: always()
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
       - name: Decide
         env:
           R_LINT: ${{ needs.lint.result }}

.gitea/workflows/renovate.yaml

@@ -0,0 +1,318 @@
+# Renovate — the dependency/image update automation. SCHEDULED, self-hosted against this Gitea.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY THIS FILE EXISTS: renovate.json HAS BEEN IN THIS REPO AND HAS NEVER ONCE RUN.
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# Measured 2026-10-17: `renovate.json` is present, complete and thoughtfully written — it declares
+# managers, schedules, automerge classes and grouping. Nothing executes it. There is no Renovate
+# workflow (this is the first), no Renovate CronJob or Deployment in the cluster
+# (`kubectl get cronjob,deploy,job -A | grep -i renovate` -> nothing), and no bot account doing it
+# from outside. Meanwhile SEVEN files carry comments asserting "Renovate maintains this pin"
+# (platform/homepage/deployment.yaml:40, runbooks/serving/deployment.yaml:87,
+# platform/crafty/deployment.yaml:63, platform/vpn-ingress/daemonset.yaml:259,
+# platform/keycloak/postgres-cluster.yaml:75, apps/wave-0/nextcloud-db.yaml:302, and
+# platform/headlamp/helmrepository.yaml:2). Those comments have been false for the life of the file.
+#
+# The cost is measurable. With nothing bumping anything: the six per-app oauth2-proxy auth sidecars
+# sit on v7.8.1 while the platform oauth2-proxy in the SAME cluster runs v7.15.5; Keycloak (the IdP)
+# runs 26.0.6, a November-2024 release, 8 minors behind; Velero runs a RELEASE CANDIDATE
+# (velero-plugin-for-aws v1.14.3-rc.2) with a stable available; the CNPG Postgres image is 16.4,
+# missing every 16.5+ security release. A pin nothing maintains is not a pin, it is a fossil.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY RENOVATE AND NOT FLUX IMAGE-AUTOMATION (the decision, and it reverses ADR-0009's split)
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# ADR-0009 divided the work: Renovate owns Helm chart/dependency bumps, Flux image-automation owns
+# per-service container image TAGS, deliberately non-overlapping. The second half of that split
+# cannot be delivered on this repo's terms, for three measured reasons:
+#
+#   1. IT WOULD BYPASS THE MERGE GATE, WHICH CLAUDE.md §1 FORBIDS. ImageUpdateAutomation does not
+#      open pull requests; it COMMITS AND PUSHES. Its default target is the source's own branch —
+#      `main` — so it would write straight past the PR flow. It can be pointed at a side branch
+#      instead, but then nothing opens the PR and bumps silently accumulate on a branch nobody
+#      merges. Either way a human or a second bot has to close the loop.
+#   2. ON THIS CLUSTER IT CANNOT EVEN DO THAT. Measured 2026-10-17: `main` is protected with
+#      `user_can_push: false`, and the `flux-system` GitRepository has NO `spec.secretRef` at all —
+#      the source is anonymous and read-only. Enabling write-back means minting an in-cluster git
+#      WRITE credential for the repository that defines the cluster, which is a materially larger
+#      blast radius than a CI-scoped token.
+#   3. RENOVATE ALREADY OPENS PRs NATIVELY, so every change it proposes goes through the gate with
+#      no extra machinery.
+#
+# So: Renovate owns BOTH halves, and Flux image-automation stays disabled. Note this is a change of
+# plan, not a change of state — image-automation has been commented out at
+# platform/kustomization.yaml:52 since it was authored, and the cluster has ZERO
+# ImageRepository/ImagePolicy/ImageUpdateAutomation objects (measured). Its two controllers ARE
+# installed and have idled for 43 days. Nothing is being torn down here; a path that was never
+# taken is being formally closed, and renovate.json's `docker`/`kubernetes` managers — disabled
+# purely to avoid fighting a controller that never ran — are switched on to fill the gap.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# THIS WORKFLOW MUST NOT BE ABLE TO SUCCEED SILENTLY
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# A SKIPPED Gitea job reports `success` (CLAUDE.md §2), so "green" is not evidence this ran. Two
+# guards:
+#   * The preflight step FAILS HARD (exit 1) if RENOVATE_TOKEN is absent, rather than skipping.
+#     A missing credential on a dependency bot is indistinguishable from "nothing to update", and
+#     that is precisely how renovate.json stayed inert for months.
+#   * The summary step records what ran into the job summary unconditionally.
+# What this still does NOT give you is an alert when the SCHEDULE stops firing. See the handoff
+# entry's "What I did NOT verify" — that gap is stated, not papered over.
+#
+# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
+#   `RENOVATE_TOKEN` — a Gitea access token for the bot account (`renovate-bot`) with FIVE scopes.
+#   Each is required by a call Renovate 44.x makes against Gitea 1.24.3, and Gitea refuses a
+#   token missing one with `403 token does not have at least one of required scope(s)`:
+#     read:user          GET /user                        (platform init: initPlatform)
+#     read:organization  GET /orgs/{owner}, /orgs/{o}/labels  (initRepo -> isOrg, then org labels)
+#     write:repository   clone/push branches, /pulls, /statuses
+#     write:issue        /issues, /labels, /issues/{n}/labels  (dependency dashboard, PR labels;
+#                        Gitea files all of these under the ISSUE scope, not repository)
+#     read:package       the Gitea container registry (RENOVATE_HOST_RULES below)
+#   CORRECTED 2026-10-30. This list used to name only write:repository + read:user (+ read:package),
+#   and the operator minted exactly that. Run 350 then failed at
+#   `GET /api/v1/orgs/Home-Network 403 ... required=[read:organization], token scope=read:package,
+#   write:repository,read:user`; write:issue would have been the next wall. The preflight below now
+#   PROBES these scopes, so a short token fails in seconds with the missing scope named.
+#   Set it as an Actions secret on this repository. Nothing in this repo mints,
+#   stores or references its value; it is consumed by name only.
+#   Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
+#   visible red is the correct state for "automation configured but not credentialed".
+#
+# THE FIRST RUN WILL WANT TO OPEN A LOT OF PRs. ~30 dependencies are behind. renovate.json caps the
+# blast radius (`prConcurrentLimit: 5`, `prHourlyLimit: 2`), but run it manually with
+# `dry_run: true` FIRST and read the log before letting the schedule loose. That is what
+# workflow_dispatch's default is for.
+name: renovate
+
+on:
+  schedule:
+    # 08:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` with
+    # `"timezone": "America/New_York"`, and THAT inner schedule is authoritative — Renovate re-checks
+    # it and DECLINES TO OPEN PRs outside it. So this cron must fire INSIDE the local window, not
+    # merely near it.
+    #
+    # WAS `15 3 * * *`, WHICH NEVER WOULD HAVE WORKED (fixed 2026-10-18, before the first scheduled
+    # run). Gitea Actions cron is UTC, so 03:15 UTC is 23:15 the PREVIOUS EVENING in New York --
+    # 3h45m BEFORE the window opens, not "slightly ahead" of it as the original comment claimed.
+    # Every nightly run would have started, logged "not within schedule", opened zero PRs and
+    # exited SUCCESSFULLY. A green tick forever, and no updates ever: the exact "failure mode is a
+    # green tick" shape this repo keeps getting burned by, and indistinguishable from "nothing
+    # needed updating".
+    #
+    # THE WINDOW IN UTC MOVES WITH US DST, and a cron cannot follow it:
+    #   EDT (UTC-4, Mar-Nov): local 03:00-06:00 == 07:00-10:00 UTC
+    #   EST (UTC-5, Nov-Mar): local 03:00-06:00 == 08:00-11:00 UTC
+    # 08:15 UTC is the only quarter-hour comfortably inside BOTH (04:15 EDT / 03:15 EST), so the
+    # schedule keeps working across both switchovers without an edit. Do not "tidy" this back to an
+    # early-morning-UTC value without redoing this arithmetic.
+    - cron: "15 8 * * *"
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: "Dry run (no PRs, no branches). Leave true for the first run."
+        required: false
+        default: "true"
+      log_level:
+        description: "Renovate LOG_LEVEL (info | debug)"
+        required: false
+        default: "info"
+
+concurrency:
+  # Never two Renovate runs at once: concurrent runs race on the same branches and can open
+  # duplicate PRs. Not cancel-in-progress — a half-finished Renovate run can leave a branch pushed
+  # with no PR, so let the first one finish.
+  group: renovate
+  cancel-in-progress: false
+
+jobs:
+  renovate:
+    runs-on: ubuntu-latest
+    # RENOVATE RUNS AS THE JOB CONTAINER, NOT THROUGH renovatebot/github-action. FIXED 2026-10-30.
+    # That action does exactly one thing: `docker run ghcr.io/renovatebot/renovate:<version>`. On
+    # this runner a job has NO Docker daemon, by decision (decisions/2026-10-29-image-pipeline.md:
+    # act_runner sets container.docker_host to `-` when DOCKER_HOST is tcp://, and the DinD daemon
+    # is privileged, so a job that could reach it would be root on the node). So every run of the
+    # action died before Renovate started. Dispatch run 343 (2026-10-30), verbatim:
+    #   [command]/usr/bin/docker run --env LOG_LEVEL ... --env RENOVATE_TOKEN=*** ...
+    #   failed to connect to the docker API at unix:///var/run/docker.sock; ... no such file or directory
+    #   ::error::Error: The process '/usr/bin/docker' failed with exit code 1
+    # No auth, scope, config or network question was ever reached. Making the Renovate image the
+    # job container needs no daemon: act starts it through the runner's own DinD exactly as it
+    # starts the default ubuntu image, then `exec`s each step inside it.
+    #
+    # PINNED here (was the action's `renovate-version: 44.167.0`). renovate.json5 is validated
+    # against this exact tag — keep them in step. Renovate's github-actions manager reads
+    # `.gitea/workflows` and extracts `container.image`, so this pin is still self-maintained.
+    #
+    # The image runs as uid 12021 (`ubuntu`), and that is fine under act: the step command files
+    # ($GITHUB_STEP_SUMMARY etc.) are created mode 0666 (gitea/act v0.261.2 pkg/runner/step.go),
+    # HOME comes from the image env, and no step writes the workspace. There is deliberately NO
+    # actions/checkout: Renovate clones the repo itself from RENOVATE_ENDPOINT, and as uid 12021
+    # checkout could not `git init` the root-owned workspace volume anyway.
+    container:
+      image: ghcr.io/renovatebot/renovate:44.167.0
+    # Renovate scans every manager across the whole repo; 40m is generous for this repo's size and
+    # well under a runner's patience. Without a timeout a wedged run holds the `renovate`
+    # concurrency group and silently blocks every subsequent scheduled run.
+    timeout-minutes: 40
+    steps:
+      - name: Preflight - the token must exist, and its absence must be LOUD
+        shell: bash
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${RENOVATE_TOKEN:-}" ]; then
+            echo "::error::RENOVATE_TOKEN is not set on this repository." >&2
+            cat >&2 <<'MSG'
+
+          Renovate cannot run without a Gitea access token, and this step fails rather than
+          skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would make
+          "dependency automation is not running" look exactly like "everything is up to date".
+          That is the failure mode that let renovate.json sit inert in this repo for months while
+          seven files claimed "Renovate maintains this pin".
+
+          TO FIX: create a Gitea access token for the bot account (renovate-bot) with scopes
+          read:user, read:organization, write:repository, write:issue and read:package (the
+          header of .gitea/workflows/renovate.yaml says which call needs which), then add it to
+          this repository as the Actions secret RENOVATE_TOKEN. Do not commit it anywhere.
+          MSG
+            exit 1
+          fi
+          echo "RENOVATE_TOKEN is present (value not printed)."
+          # Proves the step is running INSIDE the pinned Renovate image, not the default job image.
+          echo "renovate $(renovate --version)"
+
+      - name: Preflight - the token has every scope Renovate needs
+        shell: bash
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+        # ADDED 2026-10-30. Renovate reports EVERY platform-init failure as the bare
+        # "Authentication failure" (lib/modules/platform/gitea/index.ts initPlatform swallows the
+        # HTTP error), and a later scope gap surfaces as "Repository has unknown error". Two
+        # dispatches were spent learning one missing scope at a time. These probes are GETs only,
+        # and they name the missing scope. The token never reaches argv: curl reads it from stdin
+        # (`--config -`), so it is not in `ps` output or in the log.
+        run: |
+          set -uo pipefail
+          GITEA=https://gitea.home.keithjackson.dev
+          REPO=Home-Network/home-network-k3s-impl
+          fail=0
+          body=$(mktemp)
+          probe() { # <scope it proves> <path> <auth: bearer|basic>
+            local code
+            if [ "$3" = basic ]; then
+              code=$(printf 'user = "renovate:%s"\n' "$RENOVATE_TOKEN" \
+                | curl -sS --config - -o "$body" -w '%{http_code}' "$GITEA$2") || code=000
+            else
+              code=$(printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
+                | curl -sS --config - -o "$body" -w '%{http_code}' "$GITEA$2") || code=000
+            fi
+            if [ "$code" = 200 ]; then
+              echo "ok    $1  (GET ${2%%\?*} -> 200)"
+              return
+            fi
+            fail=1
+            echo "::error::scope $1: GET ${2%%\?*} -> $code"
+            case "$code" in
+              401) echo "      401 = Gitea rejected the token VALUE (wrong, expired or revoked)." ;;
+              403) if grep -q 'Access denied' "$body"; then
+                     echo "      403 'Access denied' = a Cilium/Envoy policy denial, not Gitea."
+                   else
+                     echo "      Gitea: $(jq -r '.message // empty' "$body" 2>/dev/null | head -c 300)"
+                   fi ;;
+              *)   echo "      body: $(head -c 200 "$body")" ;;
+            esac
+          }
+          probe read:user          "/api/v1/user"                          bearer
+          probe read:organization  "/api/v1/orgs/${REPO%%/*}"              bearer
+          probe read:repository    "/api/v1/repos/$REPO"                   bearer
+          probe read:issue         "/api/v1/repos/$REPO/labels?limit=1"    bearer
+          probe read:package       "/v2/token?service=container_registry&scope=repository:home-network/toolbox:pull" basic
+          # WRITE scopes cannot be proven by a GET. Gitea's scope-refusal message, however, lists
+          # the token's scopes (names only, never the value): ask for an admin route this bot
+          # cannot have, and read `token scope=` from the 403. Advisory if the format changes.
+          printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
+            | curl -sS --config - -o "$body" "$GITEA/api/v1/admin/orgs?limit=1" || true
+          scopes=$(jq -r '.message // empty' "$body" 2>/dev/null | sed -n 's/.*token scope=//p')
+          if [ -n "$scopes" ]; then
+            echo "token scopes (from Gitea): $scopes"
+            for want in write:repository write:issue; do
+              case ",$scopes," in
+                *",$want,"*|*",all,"*) echo "ok    $want" ;;
+                *) echo "::error::scope $want is missing (token scope=$scopes)"; fail=1 ;;
+              esac
+            done
+          else
+            echo "::warning::could not read the token's scope list; write:repository and write:issue are UNVERIFIED"
+          fi
+          rm -f "$body"
+          if [ "$fail" -ne 0 ]; then
+            echo "::error::RENOVATE_TOKEN failed the checks above (a 401 is the value, a 403 is a scope). Re-mint it for renovate-bot with read:user, read:organization, write:repository, write:issue, read:package and overwrite the secret."
+            exit 1
+          fi
+
+      - name: Run Renovate
+        shell: bash
+        # The repo config is discovered by Renovate from the cloned repo (renovate.json5,
+        # RENOVATE_REQUIRE_CONFIG=required). The action used to ALSO load it as the global config
+        # via RENOVATE_CONFIG_FILE; that double-loading is gone. ci/lint.sh's `configurationFile:`
+        # check therefore has nothing to resolve in this file, and passes.
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+          # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
+          # header said were "set at runtime by the operator's renovate runner" — this is that
+          # runner, so they are set here rather than in the repo config, which keeps renovate.json
+          # portable and free of this instance's URL.
+          RENOVATE_PLATFORM: gitea
+          RENOVATE_ENDPOINT: https://gitea.home.keithjackson.dev/api/v1
+          RENOVATE_REPOSITORIES: Home-Network/home-network-k3s-impl
+          # Renovate must never push to a protected branch; it works on its own branches and opens
+          # PRs against this base. Stated explicitly rather than inherited, because this is the
+          # property that keeps the whole mechanism inside CLAUDE.md §1's branch -> PR -> gate path.
+          RENOVATE_BASE_BRANCHES: main
+          RENOVATE_ONBOARDING: "false"
+          RENOVATE_REQUIRE_CONFIG: "required"
+          RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
+          LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
+          # ADDED 2026-10-29: registry read access for this repo's OWN images (toolbox,
+          # agent-runner, conformance-runner — published by .gitea/workflows/images.yaml and
+          # followed through `# renovate-digest:` markers). The Gitea registry answers 401 to
+          # anonymous reads, and Renovate scopes the platform token to hostType `gitea`, which a
+          # docker lookup never matches — so without this rule every digest lookup fails. Same
+          # token; it therefore needs scope `read:package` in addition to the ones listed in the
+          # preflight above. Gitea ignores the username for token auth (services/auth/basic.go).
+          RENOVATE_HOST_RULES: '[{"hostType":"docker","matchHost":"gitea.home.keithjackson.dev","username":"renovate","password":"${{ secrets.RENOVATE_TOKEN }}"}]'
+          # OPTIONAL, ADDED 2026-10-30. A github.com token (a fine-grained PAT with NO permissions,
+          # public-repo read only) for lookups of GitHub-hosted dependencies. Without it Renovate
+          # SKIPS them with `github-token-required` — measured in a local dry run: 17 deps, i.e.
+          # every `uses: actions/*` here plus kubevirt/kubevirt, kubevirt/containerized-data-importer
+          # and intel/intel-device-plugins-for-kubernetes. Unset is safe: the step below drops the
+          # empty variable, and those deps simply stay unmaintained, with a WARN in the log.
+          GITHUB_COM_TOKEN: ${{ secrets.RENOVATE_GITHUB_COM_TOKEN }}
+        run: |
+          set -euo pipefail
+          # An expression that evaluates to '' still EXPORTS the variable, as an empty string.
+          # Drop those so an unset value means "Renovate's default", never "set to empty".
+          for v in RENOVATE_DRY_RUN GITHUB_COM_TOKEN; do
+            [ -n "${!v:-}" ] || unset "$v"
+          done
+          # Through the image's entrypoint, as `docker run` would have gone: it sources the
+          # containerbase env and initialises the bundled tools before exec'ing Renovate.
+          exec renovate-entrypoint.sh renovate
+
+      - name: Summary
+        if: always()
+        run: |
+          {
+            echo "## Renovate"
+            echo
+            echo "- trigger: \`${{ github.event_name }}\`"
+            echo "- dry run: \`${{ github.event_name == 'workflow_dispatch' && inputs.dry_run || 'false' }}\`"
+            echo "- outcome: \`${{ job.status }}\`"
+            echo
+            echo "Renovate owns BOTH Helm chart bumps and container image tags here; Flux"
+            echo "image-automation is deliberately disabled (platform/kustomization.yaml)."
+            echo "Every change it proposes arrives as a pull request and must pass the merge gate."
+          } >> "$GITHUB_STEP_SUMMARY"

.gitea/workflows/scripts/images.sh

@@ -0,0 +1,373 @@
+#!/usr/bin/env bash
+###############################################################################
+# .gitea/workflows/scripts/images.sh — the image pipeline's driver (images.yaml calls this).
+#
+# Builds, verifies and (on main) PUBLISHES the three images this repo ships:
+#
+#   toolbox             infra/devcontainer/    the pinned dev/gate toolchain
+#   agent-runner        infra/agent/           FROM toolbox; the PM dispatcher + worker
+#   conformance-runner  conformance/incluster/ the in-cluster L2/L3 Chainsaw runner
+#
+# THE BUILDER (operator decision on PR #81): a ROOTLESS BuildKit daemon, platform/buildkit/,
+# reached over mTLS. No CI job may reach a privileged daemon, so there is no `docker build` and no
+# `docker run` here: this script drives buildkitd with a pinned `buildx` (remote driver) and
+# nothing else. The client certificate arrives as Actions secrets (BUILDKIT_CA,
+# BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY; infra/gitea/buildkit-client-secret.sh puts them
+# there). If the daemon is unreachable this FAILS LOUDLY — it never skips.
+#
+# VERIFICATION IS A BUILD TARGET. Each Containerfile ends in a `verify` stage that runs every
+# in-image assertion at build time (as the non-root uid, with RUN --network=none); this script
+# builds `<image>-verify` and only then the image. Build args come from each image's own
+# `build-image.sh --print-build-args`, the same list the local podman path uses.
+#
+# MODES (env MODE):
+#   verify   build the verify targets, never push, hold no registry credential. Pull requests.
+#   publish  verify, then build + push to ${REGISTRY}/${NAMESPACE}/<image> with tags
+#            `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows
+#            it). The digest recorded is the one the REGISTRY returns, read back after the push.
+#
+# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's
+# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this
+# file, platform/buildkit/versions.env) builds everything in verify mode, and nothing extra in
+# publish mode: re-publishing unchanged inputs would only mint new digests and churn Renovate PRs.
+#
+# THE agent-runner BASE:
+#   * verify mode -> FROM the toolbox built in this run, linked with a bake `target:` context —
+#     a pull request holds no registry credential and that toolbox exists in no registry. If the
+#     toolbox's inputs did not change it is built anyway, as the base.
+#   * publish mode, toolbox published in this run -> FROM the digest the registry just returned.
+#   * publish mode otherwise -> AGENT_BASE_IMAGE from infra/agent/versions.env (the published
+#     toolbox digest Renovate keeps current). If that is still the all-zero placeholder the
+#     toolbox has never been published, so it is built and published first (the bootstrap).
+#
+# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in
+# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and
+# infra/agent/versions.env (agent-runner input -> ONE more agent-runner build, whose own bump
+# lands in platform/pm-agent/ and stops). The worker Job template is deliberately NOT tracked:
+# it is baked INTO agent-runner, so bumping it would rebuild agent-runner, mint a new digest,
+# bump it again, forever. pm-dispatch overwrites that field at runtime anyway.
+###############################################################################
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
+cd "${REPO_ROOT}"
+
+# shellcheck disable=SC1091
+BK_HOST_OVERRIDE="${BUILDKIT_HOST:-}"; BK_NAME_OVERRIDE="${BUILDKIT_SERVERNAME:-}"
+. ./platform/buildkit/versions.env
+BUILDKIT_HOST="${BK_HOST_OVERRIDE:-${BUILDKIT_HOST}}"
+BUILDKIT_SERVERNAME="${BK_NAME_OVERRIDE:-${BUILDKIT_SERVERNAME}}"
+
+MODE="${MODE:-verify}"
+IMAGES="${IMAGES:-auto}"
+REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}"
+NAMESPACE="${NAMESPACE:-home-network}"
+HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}"
+BASE_SHA="${BASE_SHA:-}"
+SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}"
+ZERO_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
+
+ALL_IMAGES=(toolbox agent-runner conformance-runner)
+SHA_TAG="sha-${HEAD_SHA:0:12}"
+MOVING_TAG="main"
+
+log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; }
+yn()   { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; }
+
+case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac
+for t in git jq curl sha256sum; do command -v "$t" >/dev/null || die "the job image lacks '$t'"; done
+
+# --- per-image facts ------------------------------------------------------------------------
+# Inputs: a change under any of these rebuilds that image. conformance-runner takes its pins from
+# infra/devcontainer/versions.env. conformance/versions.env is NOT an input (it only RECORDS the
+# published digest), and must not become one: Renovate writes it after every publish -> a loop.
+inputs() {
+  case "$1" in
+    toolbox)            echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;;
+    agent-runner)       echo "infra/agent/" ;;
+    conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;;
+  esac
+}
+context_of() { case "$1" in toolbox) echo infra/devcontainer ;; agent-runner) echo infra/agent ;; conformance-runner) echo conformance/incluster ;; esac; }
+PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh platform/buildkit/versions.env"
+
+declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=()
+
+changed_any() { # <paths...> -> 0 if BASE_SHA..HEAD_SHA touches any
+  [ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ]
+}
+
+select_images() {
+  local i
+  if [ "${IMAGES}" != "auto" ]; then
+    for i in ${IMAGES}; do
+      case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac
+    done
+    return
+  fi
+  if [ -z "${BASE_SHA}" ] || [[ "${BASE_SHA}" =~ ^0+$ ]] || ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then
+    info "no usable diff base ('${BASE_SHA}'): building everything"
+    for i in "${ALL_IMAGES[@]}"; do WANT[$i]=1; NOTE[$i]="no diff base"; done
+    return
+  fi
+  info "diff base ${BASE_SHA} .. ${HEAD_SHA}"
+  for i in "${ALL_IMAGES[@]}"; do
+    # shellcheck disable=SC2046
+    if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi
+  done
+  # shellcheck disable=SC2086
+  if [ "${MODE}" = verify ] && changed_any ${PIPELINE_FILES}; then
+    for i in "${ALL_IMAGES[@]}"; do [ -n "${WANT[$i]:-}" ] || { WANT[$i]=1; NOTE[$i]="pipeline changed"; }; done
+  fi
+}
+
+# --- the builder: pinned buildx + mTLS to rootless buildkitd ---------------------------------
+WORK="$(mktemp -d)"
+trap 'rm -rf "${WORK}"' EXIT          # client key + registry auth live only here, only this long
+export BUILDX_CONFIG="${WORK}/buildx" DOCKER_CONFIG="${WORK}/docker"
+# Single-manifest images: the admission policies, Renovate and registry-retention.sh all reason
+# about one manifest per tag. (Provenance attestations would turn every push into an index.)
+export BUILDX_NO_DEFAULT_ATTESTATIONS=1
+mkdir -p "${BUILDX_CONFIG}" "${DOCKER_CONFIG}" "${WORK}/certs"
+echo '{}' > "${DOCKER_CONFIG}/config.json"
+
+setup_builder() {
+  log "Builder: rootless buildkitd at ${BUILDKIT_HOST} (mTLS)"
+  BUILDX="${BUILDX_BIN:-}"
+  if [ -z "${BUILDX}" ]; then
+    BUILDX="${WORK}/buildx-bin"
+    curl -fsSL --retry 3 -o "${BUILDX}" \
+      "https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.linux-amd64" \
+      || die "could not download buildx v${BUILDX_VERSION}"
+    echo "${BUILDX_SHA256}  ${BUILDX}" | sha256sum -c --quiet - \
+      || die "buildx v${BUILDX_VERSION} does not match the sha256 pinned in platform/buildkit/versions.env — refusing to run it"
+    chmod +x "${BUILDX}"
+  fi
+  info "client: $("${BUILDX}" version)"
+
+  local missing=()
+  [ -n "${BUILDKIT_CA:-}" ]          || missing+=(BUILDKIT_CA)
+  [ -n "${BUILDKIT_CLIENT_CERT:-}" ] || missing+=(BUILDKIT_CLIENT_CERT)
+  [ -n "${BUILDKIT_CLIENT_KEY:-}" ]  || missing+=(BUILDKIT_CLIENT_KEY)
+  [ "${#missing[@]}" -eq 0 ] || die "the builder's client certificate is missing: ${missing[*]} not set.
+       These Actions secrets are copied from the cert-manager Secret buildkit/buildkit-client-ci by
+       infra/gitea/buildkit-client-secret.sh (an operator action, after platform/buildkit is
+       deployed). Without them nothing can be built — this fails rather than skipping ON PURPOSE."
+  umask 077
+  printf '%s\n' "${BUILDKIT_CA}"          > "${WORK}/certs/ca.pem"
+  printf '%s\n' "${BUILDKIT_CLIENT_CERT}" > "${WORK}/certs/cert.pem"
+  printf '%s\n' "${BUILDKIT_CLIENT_KEY}"  > "${WORK}/certs/key.pem"
+  umask 022
+
+  "${BUILDX}" create --name ci-remote --driver remote \
+    --driver-opt "cacert=${WORK}/certs/ca.pem,cert=${WORK}/certs/cert.pem,key=${WORK}/certs/key.pem,servername=${BUILDKIT_SERVERNAME}" \
+    "${BUILDKIT_HOST}" >/dev/null
+  if ! out="$(timeout 90 "${BUILDX}" inspect --bootstrap --builder ci-remote 2>&1)"; then
+    printf '%s\n' "${out}" >&2
+    die "buildkitd is UNREACHABLE at ${BUILDKIT_HOST} — nothing can be built, and this job fails
+       rather than skipping. In order of likelihood:
+         1. platform/buildkit is not deployed yet. Flux applies it only after it merges to main
+            (Kustomization 'buildkit'); a pull request that ADDS it cannot reach it.
+         2. the BUILDKIT_* secrets are stale (the client cert was renewed) or from another CA:
+            re-run infra/gitea/buildkit-client-secret.sh.
+         3. the NetworkPolicy (platform/buildkit/networkpolicy.yaml) does not admit this runner."
+  fi
+  printf '%s\n' "${out}" | grep -E '^(Name|Driver|Status|BuildKit version|Platforms):' | sed 's/^/    /' || true
+}
+
+# KEY=VALUE lines from an image's build-image.sh --print-build-args -> a JSON object.
+build_args_json() {
+  local ctx; ctx="$(context_of "$1")"
+  "./${ctx}/build-image.sh" --print-build-args | jq -Rn '[inputs | select(length>0) | capture("^(?<k>[^=]+)=(?<v>.*)$")] | map({(.k): .v}) | add'
+}
+
+# bake_run <bakefile> <targets...>
+bake_run() {
+  local f="$1"; shift
+  "${BUILDX}" bake --builder ci-remote --progress plain -f "$f" \
+    --metadata-file "${WORK}/meta.json" "$@"
+}
+
+# Write a bake file with a verify target and a ship target for one image.
+#   $1 image, $2 agent base mode: "" | "link" | <image ref>
+write_bake() {
+  local img="$1" base="${2:-}" ctx args f
+  ctx="$(context_of "$img")"
+  args="$(build_args_json "$img")"
+  f="${WORK}/bake-${img}.json"
+  local contexts='{}'
+  if [ "${img}" = agent-runner ]; then
+    if [ "${base}" = link ]; then
+      # FROM ${AGENT_BASE_IMAGE} resolves to the named context below, i.e. the toolbox target
+      # of this same bake file — built here, never pushed anywhere.
+      args="$(jq -c '. + {AGENT_BASE_IMAGE: "toolbox-in-this-run"}' <<<"${args}")"
+      contexts='{"toolbox-in-this-run": "target:toolbox"}'
+    else
+      args="$(jq -c --arg b "${base}" '. + {AGENT_BASE_IMAGE: $b}' <<<"${args}")"
+    fi
+  fi
+  jq -n --arg img "$img" --arg ctx "$ctx" --argjson args "$args" --argjson contexts "$contexts" \
+        --arg tbargs "$( [ "${base}" = link ] && build_args_json toolbox || echo '{}')" \
+        --arg rev "${HEAD_SHA}" --arg repo "${REGISTRY}/${NAMESPACE}/${img}" \
+        --arg sha "${SHA_TAG}" --arg moving "${MOVING_TAG}" --arg mode "${MODE}" '
+    {target: (
+      {
+        ($img + "-verify"): {context: $ctx, dockerfile: "Containerfile", target: "verify",
+                              args: $args, contexts: $contexts, output: ["type=cacheonly"]},
+        ($img): ({context: $ctx, dockerfile: "Containerfile", target: $img, args: $args,
+                  contexts: $contexts,
+                  labels: {"org.opencontainers.image.revision": $rev}}
+                 + (if $mode == "publish"
+                    then {tags: [($repo + ":" + $sha), ($repo + ":" + $moving)], output: ["type=image,push=true"]}
+                    else {output: ["type=cacheonly"]} end))
+      }
+      + (if ($contexts | has("toolbox-in-this-run"))
+         then {toolbox: {context: "infra/devcontainer", dockerfile: "Containerfile", target: "toolbox",
+                         args: ($tbargs | fromjson), output: ["type=cacheonly"]}}
+         else {} end)
+    )}' > "$f"
+  echo "$f"
+}
+
+# --- registry helpers (publish mode only) ----------------------------------------------------
+REG_TOKEN=""
+registry_token() {
+  [ -n "${REG_TOKEN}" ] && return 0
+  REG_TOKEN="$(curl -fsS --config <(printf 'user = "%s:%s"\n' "${REGISTRY_USER}" "${REGISTRY_TOKEN}") \
+    "https://${REGISTRY}/v2/token?service=container_registry&scope=*" | jq -r '.token // empty')" \
+    || die "could not obtain a registry bearer token from https://${REGISTRY}/v2/token"
+  [ -n "${REG_TOKEN}" ] || die "registry returned no bearer token"
+}
+# Prints the manifest digest the REGISTRY serves for <image>:<tag>; nothing if it is absent (404).
+# Any other answer is fatal: "cannot tell" must never be read as "not published", or the
+# immutable-tag guard would be bypassed by an auth hiccup.
+registry_digest() {
+  registry_token
+  local hdrs code digest
+  hdrs="$(curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "${REG_TOKEN}") \
+    -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
+    -H 'Accept: application/vnd.oci.image.index.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
+    "https://${REGISTRY}/v2/${NAMESPACE}/$1/manifests/$2" | tr -d '\r')" \
+    || die "registry unreachable while reading $1:$2"
+  code="$(awk 'NR==1{print $2}' <<<"${hdrs}")"
+  digest="$(awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' <<<"${hdrs}" | tail -1)"
+  case "${code}" in
+    200) [[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]] || die "registry answered 200 for $1:$2 with no usable digest"
+         echo "${digest}" ;;
+    404) ;;
+    *)   die "registry answered HTTP ${code} for $1:$2 — cannot tell whether it exists" ;;
+  esac
+}
+
+# build_one <image> [agent base]  — verify, then (publish) build + push + read back.
+build_one() {
+  local img="$1" base="${2:-}" t0 f pushed got existing
+  t0=$(date +%s)
+  f="$(write_bake "$img" "$base")"
+  log "VERIFY ${img} (Containerfile target: verify, on rootless BuildKit)"
+  bake_run "$f" "${img}-verify"
+  BUILT[$img]=1
+  if [ "${MODE}" = publish ]; then
+    existing="$(registry_digest "${img}" "${SHA_TAG}")"
+    # Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would
+    # silently change what `sha-<commit>` means after something may already have pinned it.
+    [ -z "${existing}" ] || die "${REGISTRY}/${NAMESPACE}/${img}:${SHA_TAG} already exists (${existing}); tags are immutable"
+    log "BUILD + PUSH ${img} -> ${REGISTRY}/${NAMESPACE}/${img}:{${SHA_TAG},${MOVING_TAG}}"
+    bake_run "$f" "${img}"
+    pushed="$(jq -r --arg t "$img" '.[$t]["containerimage.digest"] // empty' "${WORK}/meta.json")"
+    # The AUTHORITATIVE digest is what the registry serves, read back — not what the client
+    # reported. They should agree; if they do not, nothing downstream may trust either.
+    got="$(registry_digest "${img}" "${SHA_TAG}")"
+    [ -n "${got}" ] || die "pushed ${img}:${SHA_TAG} but the registry does not serve it"
+    [ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] || die "${img}: client reported ${pushed} but the registry serves ${got}"
+    [ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \
+      || die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})"
+    DIGEST[$img]="${got}"
+    info "${img}: registry digest ${got}"
+  fi
+  SECS[$img]=$(( $(date +%s) - t0 ))
+  info "${img}: done in ${SECS[$img]}s"
+}
+
+# --- main ---------------------------------------------------------------------------------
+log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
+select_images
+
+if [ "${MODE}" = publish ]; then
+  : "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}"
+  REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}"
+  # buildx (and through it buildkitd, which does the push) reads registry auth from this file.
+  jq -n --arg r "${REGISTRY}" --arg a "$(printf '%s:%s' "${REGISTRY_USER}" "${REGISTRY_TOKEN}" | base64 -w0)" \
+     '{auths: {($r): {auth: $a}}}' > "${DOCKER_CONFIG}/config.json"
+  chmod 600 "${DOCKER_CONFIG}/config.json"
+  registry_token
+  # Publish idempotence: an image already published at THIS commit is not rebuilt. That is what
+  # makes a re-run after a half-finished publish safe — the immutable tag is never pushed twice.
+  for i in "${ALL_IMAGES[@]}"; do
+    [ -n "${WANT[$i]:-}" ] || continue
+    d="$(registry_digest "${i}" "${SHA_TAG}")"
+    if [ -n "${d}" ]; then
+      info "${i}: already published at ${SHA_TAG} (${d}); not rebuilding"
+      unset "WANT[$i]"; DIGEST[$i]="${d}"; NOTE[$i]="already published at this commit"
+    fi
+  done
+fi
+
+# Decide the agent-runner base (see header).
+AGENT_BASE=""
+if [ -n "${WANT[agent-runner]:-}" ]; then
+  PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")"
+  if [ "${MODE}" = verify ]; then
+    AGENT_BASE=link
+    [ -n "${WANT[toolbox]:-}" ] || { WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"; }
+  elif [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then
+    case "${PINNED_BASE}" in
+      *@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;;
+      *@sha256:*)         AGENT_BASE="${PINNED_BASE}" ;;
+      *) die "infra/agent/versions.env AGENT_BASE_IMAGE='${PINNED_BASE}' is not digest-pinned; CI will not build on a floating base" ;;
+    esac
+  fi
+fi
+
+for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip)  (${NOTE[$i]:-inputs unchanged})"; done
+if [ -z "${WANT[toolbox]:-}${WANT[agent-runner]:-}${WANT[conformance-runner]:-}" ]; then
+  info "nothing to build"
+else
+  setup_builder
+fi
+
+if [ -n "${WANT[toolbox]:-}" ]; then build_one toolbox; fi
+if [ -n "${WANT[agent-runner]:-}" ]; then
+  if [ -z "${AGENT_BASE}" ]; then
+    AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}"
+  fi
+  info "agent-runner base: $( [ "${AGENT_BASE}" = link ] && echo "the toolbox built in this run (bake target: context)" || echo "${AGENT_BASE}")"
+  build_one agent-runner "${AGENT_BASE}"
+fi
+if [ -n "${WANT[conformance-runner]:-}" ]; then build_one conformance-runner; fi
+
+# --- summary ------------------------------------------------------------------------------
+{
+  echo "## Images (${MODE})"
+  echo
+  echo "commit \`${HEAD_SHA}\` — builder: rootless BuildKit \`${BUILDKIT_HOST}\` (mTLS)"
+  echo
+  echo "| image | verify target built | seconds | published digest (registry-returned) | note |"
+  echo "|---|---|---|---|---|"
+  for i in "${ALL_IMAGES[@]}"; do
+    d="${DIGEST[$i]:-}"
+    echo "| \`${i}\` | $(yn "${BUILT[$i]:-}" yes no) | ${SECS[$i]:--} | $(yn "${d}" "\`${REGISTRY}/${NAMESPACE}/${i}@${d}\`" -) | ${NOTE[$i]:-inputs unchanged} |"
+  done
+  echo
+  if [ "${MODE}" = publish ]; then
+    echo "Tags pushed: \`${SHA_TAG}\` (immutable) and \`${MOVING_TAG}\` (moving)."
+    echo "Nothing here writes to git. Renovate opens the digest-bump PR on its next run."
+  else
+    echo "Verify mode: nothing was pushed and no registry credential was present."
+  fi
+} >>"${SUMMARY}"
+log "done"

.gitignore

@@ -37,6 +37,14 @@ coverage/
 infra/tofu/backend.hcl
 infra/tofu/.r2.env
 
+# Operator scratch, writable from inside the dev container (the repo is its only persistent
+# read-write mount). Holds things that must never be committed: a staged copy of the SOPS age or
+# WireGuard key, rendered VPN client profiles (infra/vpn-ingress/new-peer.sh writes here by
+# default). `infra/ansible/.gitignore` already does this for its own .cache/; this is the same
+# convention at the root. Do NOT rely on `*.key` alone to protect a staged key — that only covers
+# one filename shape, which is why this rule exists.
+.cache/
+
 # Claude Code local scratch: agent worktrees + session state. Nothing here is tracked;
 # ignoring it keeps `git add -A` from staging embedded worktree repos and keeps the lint
 # stage (which scans untracked files) from tripping on .claude/worktrees/.

CLAUDE.md

@@ -4,6 +4,125 @@ Read this before doing anything durable. It is the delivery contract for AI codi
 sessions and agents. The newest entry in `decisions/` is the current PM handoff and
 takes precedence on *what* to work on; this file governs *how* it lands.
 
+**Start with §0.** It is first because it is the rule you can break before you have read
+anything else: the very first `git checkout -b` in the wrong directory has already done the
+damage.
+
+## 0. Work in your OWN workspace. Never in the shared checkout.
+
+Before you create a branch, edit a file, or run `git` anything: **make yourself a git
+worktree outside the repository root, and do all of your work there.** One workspace per
+agent run, no exceptions.
+
+    # from the shared checkout, ONCE, at the start of your run
+    git fetch origin main
+    git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
+    cd /path/outside/the/repo/wt-<slug>        # everything after this happens HERE
+
+`/path/outside/the/repo` means outside this repository's tree — your session scratchpad,
+or a sibling like `/var/home/keith/source/wt-<slug>`. Branch from **`origin/main`**, never
+from `main` and never from `HEAD`; see "stale base" below.
+
+**The shared checkout is `/var/home/keith/source/home-network-k3s-impl`** — the operator's
+working copy, the one you are probably started in. In it, do **not**:
+
+- **`git checkout` / `git switch` / `git checkout -b`.** Its HEAD is shared state; moving
+  it moves the tree out from under whoever else is using it.
+- **edit, create or delete a file.** Including "just one doc", including a file you intend
+  to move later.
+- **`git add` / `git commit` / `git stash` / `git restore`.** `git add -A` there stages
+  whatever another agent left lying around; `git stash` *removes* their uncommitted work
+  into a place they will not look.
+- **`git worktree remove` anything you did not create.**
+
+Read-only commands there are fine, and the one write you do need is creating your own
+worktree.
+
+### Why — measured in the shared checkout, 2026-10-05
+
+| What was measured | Value |
+|---|---|
+| `checkout: moving` entries in its reflog | **107** — the tree has been yanked between branches 107 times |
+| the branch it was sitting on | `feat/pm-agent-issue-to-pr` — **not `main`** |
+| uncommitted trees left in it by other runs | **2** (`infra/agent/`, `platform/pm-agent/`), mtimes five minutes apart the same evening |
+| its local `main` vs `origin/main` | **two commits behind** (`4386d14` vs `22c94ef`) |
+| registered worktrees | **42**, of which **31** are on branches already merged to `main` |
+| gate scripts carrying an explicit `.claude/` exclusion | **9** |
+
+Each row is a distinct way the shared checkout damages work:
+
+- **Dirty tree, foreign files.** Two other runs' uncommitted directories were sitting in
+  it. An agent that edits there and runs `git add -A` ships someone else's half-finished
+  work inside its own PR, and the PR's author has no idea what those files are. This is not
+  hypothetical and it is not historical: **eleven minutes after those two directories were
+  measured, while this very section was being written, another run committed them** —
+  `5334abe feat(pm-agent): ...`, 1,000+ lines across `infra/agent/`, `clusters/`, `catalog/`
+  and `apps/` — onto the shared checkout's checked-out branch, moving its HEAD under anyone
+  reading it. Had this rule's author been working in that tree rather than in a worktree,
+  that commit and this one would have collided.
+- **Stale base.** Its `main` was two commits behind `origin/main`, so
+  `git checkout -b x main` there branches off a tree that is already not what CI will merge
+  into — the gate you run locally proves a base nobody else has. Worse, it was not even *on*
+  `main`: a bare `git checkout -b x` would have branched off `feat/pm-agent-issue-to-pr` and
+  silently inherited that work into your PR. `git fetch` first, and name `origin/main`
+  explicitly as the start point.
+- **Shared HEAD.** 107 branch switches in one tree is 107 chances to switch it while
+  another agent is mid-edit. The loser's changes are either carried onto the wrong branch
+  or lost to a checkout conflict — and the symptom surfaces in *their* session, not yours.
+
+### Put the workspace OUTSIDE the repo root
+
+A worktree *inside* the tree (`.claude/worktrees/agent-*/` — which is where Claude Code's
+own default puts it) is a full second copy of the repo that every tree-walking check then
+finds. Measured 2026-09-12: `find`-based discovery reported **the same manifest under test
+once per worktree, up to 18 times**, and the L0 check comparing that listing against the
+tree failed on paths that are not authored surface at all
+(`platform/monitoring/tests/run.sh:115-123`). Nine gate scripts now carry an explicit
+`.claude/` prune for this reason — which means the hazard is *worked around*, not fixed,
+and the next check that walks the tree will hit it again. Keep the copy out of the tree and
+none of that applies.
+
+### Clean up — and expect not to be able to
+
+31 of the 42 registered worktrees are on branches already merged to `main`. A registered
+worktree **pins its branch**: while it exists, that branch cannot be deleted or checked out
+anywhere else. So a left-behind workspace is a lock on a branch name, and the cost lands on
+whoever next needs it.
+
+Remove yours when you are done:
+
+    git worktree remove /path/outside/the/repo/wt-<slug>
+
+Measured 2026-10-05: on a worktree **you created yourself, in the session scratchpad**, that
+command **succeeds** — rc 0, no prompt, with or without a branch checked out. The "worktree
+removal is sandbox-blocked" note in circulation is about the `.claude/worktrees/agent-*` ones
+belonging to *other* sessions, some of which are additionally `locked` on git's own terms. So:
+remove yours, and only yours. If removal is refused, say so in your handoff and leave the
+worktree registered — do **not** `rm -rf` the directory, which leaves a registered worktree
+pointing at nothing and still holding the branch.
+
+If someone *else's* stale worktree is holding the branch you need, do not try to reclaim the
+branch name. Push a `rework/<pr-number>` branch to the PR's refspec instead.
+
+### If you are already standing in the shared checkout with edits
+
+Stop. Do not `git stash` (that moves other agents' work too). Create your worktree as
+above, redo the edits there, and leave the shared checkout's HEAD and working tree exactly
+as you found them.
+
+### Gating from a worktree
+
+Three known breakages, all with workarounds, all required every run: see §3, "Running the
+gate from a worktree".
+
+### If you are writing a brief for another agent
+
+The brief must name **its own explicit branch** and tell it to create **its own worktree**
+from `origin/main`. "Work on the current branch" and "continue in the checkout" are how
+branch sprawl and cross-contamination got here: an agent given no workspace uses the one it
+is started in, which is the shared one. Also carry §3's three gate workarounds into the
+brief — an agent that cannot gate will report rc 3 as green.
+
 ## 1. Never push to `main`. Ever.
 
 Every durable change reaches `main` exactly one way:
@@ -147,15 +266,21 @@ Exit codes are a contract — do not flatten them:
 | 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
 
 `spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
-beside this one; that single skip is expected and tolerated. Any *other* skip is a
+beside this one; that single skip is expected and tolerated — and from a worktree it is
-real gap in your run.
+avoidable outright (§3's worktree list, item 3). `gitea-settings` is the other expected
+skip: it needs `SETTINGS_API_TOKEN`, which exists only in CI, so **a local run cannot
+return 0** — the honest best local result is rc 3 with `gitea-settings` as the only stage
+that did not run. Any *further* skip is a real gap in your run. Report the rc and name the
+skipped stages; never round rc 3 up.
 
 **Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
 so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
 
-### Running the gate from a git worktree
+### Running the gate from a worktree
 
-Two unrelated failures, both with known workarounds — put both in any agent brief:
+This is the **normal** path, not an edge case — §0 requires you to be in a worktree, so
+every gate run you do hits all of these. Three known breakages, all with
+workarounds, all of them yours to apply every run — put them in any agent brief:
 
 1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
    and exists only in the main checkout. Do **not** copy it (it is a credential).
@@ -166,10 +291,17 @@ Two unrelated failures, both with known workarounds — put both in any agent br
    the `ENGINE` hook — a shim that intercepts `run` and re-execs
    `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
    `tracked_files()` is this bug, not a broken tree.
+3. **`spec-immutability` skips** — the skip called tolerated above is also *avoidable*. `dev.sh`
+   derives the sibling spec path from its own `REPO_ROOT`, which in a worktree is your
+   workspace, so `../home-network-k3s` does not resolve and it adds no mount. Mount the spec
+   repo read-only in the same `ENGINE` shim and pass `-e SPEC_REPO=/home-network-k3s`: with
+   both mounts the stage reports `pass`, not SKIPPED, and R-TEST-06 is actually covered
+   (measured — `decisions/2026-10-01-render-selftest-runs-in-gate.md`).
 
 ## 4. Branch naming
 
-Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
+Use `feat/`, `fix/`, `docs/`, `chore/`. **One branch, one workspace, one PR** — if you
+need a second branch, make a second worktree (§0); do not re-point an existing one. Note that `.gitea/workflows/gate.yml`'s **push**
 trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
 The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
 

README.md

@@ -11,6 +11,9 @@ specification** at [`../home-network-k3s`](../home-network-k3s).
 - **Done = a green, reproducible, requirement-tagged conformance test** — never "merged," never a
   manual demo (spec `docs/14`, `agents/definition-of-done.md`).
 - **No work on `main`.** Every change is a feature branch merged into `main` (spec `CONTRIBUTING.md`).
+- **No work in a shared checkout.** Every agent run gets its own git worktree, outside the repo
+  root, branched from `origin/main`. This checkout is shared mutable state — see
+  [`CLAUDE.md` §0](CLAUDE.md).
 - **No plaintext secrets** — SOPS-encrypted only (spec `docs/07`).
 - **Phase discipline** — only build what the active roadmap phase authorizes (spec `roadmap/phases.md`).
 
@@ -21,7 +24,8 @@ infra/          node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible,
 bootstrap/      one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
 platform/       platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
                 longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
-                vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
+                vpn-egress, vpn-ingress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
+                (vpn-egress = pods OUT via PIA; vpn-ingress = remote clients IN via WireGuard)
 platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
                 (ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
                 Split out because CRs and the CRDs they use cannot apply in one pass.
@@ -94,6 +98,20 @@ The running narrative — decisions, root causes, incidents, follow-ups — is
 
 ## Working on this repo
 
+**First, get out of this checkout.** It is the operator's working copy and several agents
+share it; concurrent runs here have overwritten each other's branches and left each other's
+uncommitted files behind. One workspace per change:
+
+```bash
+git fetch origin main
+git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
+cd /path/outside/the/repo/wt-<slug>
+```
+
+Outside the repo root specifically — a worktree under `.claude/worktrees/` is a second full
+copy that tree-walking gate checks then count twice. The rule, the measurements behind it, and
+the three gate workarounds a worktree needs are in [`CLAUDE.md` §0](CLAUDE.md) and §3.
+
 Everything the gate needs is baked into the pinned dev container — **do not install the
 toolchain on the host**:
 

apps/wave-0/gitea.yaml

@@ -78,7 +78,7 @@ spec:
     spec:
       containers:
       - name: gitea
-        image: gitea/gitea:1.24.3
+        image: gitea/gitea:1.27.3
         ports:
         - containerPort: 3000
         env:
@@ -191,6 +191,27 @@ spec:
       - port: '3000'
         protocol: TCP
 ---
+# CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): gitea-allow-pm-agent-ingress
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: gitea-allow-pm-agent-ingress
+  namespace: gitea
+  labels:
+    app.kubernetes.io/name: gitea
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: gitea
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: pm-agent
+    toPorts:
+    - ports:
+      - port: '3000'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): gitea-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-0/jackett.yaml

@@ -78,7 +78,7 @@ spec:
     spec:
       containers:
       - name: jackett
-        image: lscr.io/linuxserver/jackett:latest
+        image: lscr.io/linuxserver/jackett:v0.24.3025-ls29
         ports:
         - containerPort: 9117
         volumeMounts:
@@ -244,6 +244,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): jackett-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: jackett-allow-monitoring-scrape
+  namespace: jackett
+  labels:
+    app.kubernetes.io/name: jackett
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: jackett
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-0/nextcloud-db.yaml

@@ -48,7 +48,7 @@ spec:
   # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
   # pg_basebackup from the running primary, which is neither restarted nor disconnected.
   instances: 3
-  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.15
   # Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered
   # nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
   inheritedMetadata:
@@ -300,7 +300,7 @@ spec:
               # is already in it and it is already present on every node, so this
               # adds no new image and no new supply-chain surface. Keep the tag in
               # step with spec.imageName above when Renovate bumps it.
-              image: ghcr.io/cloudnative-pg/postgresql:16.4
+              image: ghcr.io/cloudnative-pg/postgresql:16.15
               imagePullPolicy: IfNotPresent
               command:
                 - psql

apps/wave-0/pihole.yaml

@@ -62,7 +62,7 @@ spec:
     spec:
       containers:
       - name: pihole
-        image: pihole/pihole
+        image: pihole/pihole:2026.10.1
         ports:
         - containerPort: 80
         env:
@@ -73,7 +73,7 @@ spec:
         - name: FTLCONF_dns_rateLimit_count
           value: '0'
         - name: FTLCONF_misc_dnsmasq_lines
-          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
+          value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
         - name: FTLCONF_webserver_api_password
           valueFrom:
             secretKeyRef:
@@ -232,6 +232,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: pihole-allow-monitoring-scrape
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-0/plex.yaml

@@ -77,7 +77,7 @@ spec:
   storageClassName: longhorn
   resources:
     requests:
-      storage: 8Gi
+      storage: 16Gi
 ---
 # PVC plex-media (R-STATE)
 apiVersion: v1
@@ -120,7 +120,7 @@ spec:
     spec:
       containers:
       - name: plex
-        image: plexinc/pms-docker:latest
+        image: plexinc/pms-docker:1.43.6.11085-d1e52e169
         ports:
         - containerPort: 32400
         env:
@@ -138,7 +138,19 @@ spec:
               name: plex-claim
               key: PLEX_CLAIM
         - name: ADVERTISE_IP
-          value: ${PLEX_ADVERTISE_URL}
+          value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
+        - name: PLEX_TLS_CRT
+          valueFrom:
+            secretKeyRef:
+              name: plex-vip-tls
+              key: tls.crt
+              optional: true
+        - name: PLEX_TLS_KEY
+          valueFrom:
+            secretKeyRef:
+              name: plex-vip-tls
+              key: tls.key
+              optional: true
         resources:
           limits:
             gpu.intel.com/i915: 1
@@ -148,6 +160,14 @@ spec:
         - name: media
           mountPath: /media
           readOnly: true
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# NOTE \u2014 LOAD-BEARING: SHELL VARIABLES HERE ARE WRITTEN BARE, NEVER BRACE-WRAPPED.\n# This whole script is rewritten by Flux postBuild envsubst BEFORE it ever runs, so a\n# brace-wrapped name is read as a cluster-vars knob: an undeclared one is a HARD\n# Kustomization failure under StrictPostBuildSubstitutions, and a declared one would be\n# replaced with its value. Shell vars are therefore `$PX`, `$TOKEN`, `$P12` \u2014 bare. Only\n# BASE_DOMAIN below is brace-wrapped, because it IS a knob and IS meant to be\n# substituted. Command substitution `$(cmd)` is left alone and needs no escaping. A shell\n# variable written with braces would be blanked or would fail the whole Kustomization \u2014\n# this repo has already lost a gitea OIDC bootstrap to exactly that mistake, and\n# catalog/render.py now refuses to emit it (which is how this very comment got caught).\n# DELIBERATELY NO `set -u`. The TLS env vars are declared `optional: true`, so when the\n# certificate has not issued yet they are genuinely UNSET, not empty \u2014 and under `set -u`\n# the first `[ -n \"$PLEX_TLS_CRT\" ]` would abort the script, which for a postStart hook\n# means kubelet KILLS THE CONTAINER. That is the precise failure the `optional` flag exists\n# to prevent, so turning it back on here would defeat the whole design. (The alternative,\n# a brace-wrapped `:-` default, would need envsubst escaping \u2014 the trap the note above\n# describes.) Every value below is explicitly guarded instead.\nPX=\"/config/Library/Application Support/Plex Media Server/Preferences.xml\"\nCERTDIR=/config/certs\nP12=$CERTDIR/plex-vip.p12\n# The PKCS#12 passphrase protects nothing and is deliberately not a secret: Plex needs it\n# in cleartext in customCertificateKey, and it and the bundle sit on the same PVC. It is a\n# container-format requirement, not an access control.\nP12PASS=plexvip\nLANNETS='10.0.0.0/8,192.168.0.0/16'\nCERTDOMAIN=plex-vip.${BASE_DOMAIN}\nAPI=http://127.0.0.1:32400\n\n# (1) Build the PKCS#12 bundle from the PEM the Secret supplies. Rebuilt on every start so\n# a cert-manager RENEWAL is picked up with no extra moving part.\nif [ -n \"$PLEX_TLS_CRT\" ] && [ -n \"$PLEX_TLS_KEY\" ]; then\n  mkdir -p $CERTDIR && chmod 700 $CERTDIR\n  printf '%s' \"$PLEX_TLS_CRT\" > $CERTDIR/tls.crt\n  printf '%s' \"$PLEX_TLS_KEY\" > $CERTDIR/tls.key\n  chmod 600 $CERTDIR/tls.key\n  if openssl pkcs12 -export -out $P12.tmp -inkey $CERTDIR/tls.key \\\n       -in $CERTDIR/tls.crt -passout pass:$P12PASS 2>/dev/null; then\n    mv -f $P12.tmp $P12\n    chown 1000:1000 $P12 2>/dev/null || true\n    chmod 600 $P12\n    echo \"[plex-settings] PKCS#12 bundle written\"\n  else\n    rm -f $P12.tmp\n    echo \"[plex-settings] WARN: pkcs12 export failed; keeping any previous bundle\" >&2\n  fi\n  rm -f $CERTDIR/tls.key\nelse\n  echo \"[plex-settings] WARN: TLS env empty; certificate not installed this start\" >&2\nfi\n\n# (2) Wait, bounded (~90s), for the local API. /identity needs no auth.\ni=0\nwhile [ \"$i\" -lt 45 ]; do\n  if [ \"$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 $API/identity 2>/dev/null)\" = \"200\" ]; then\n    break\n  fi\n  i=$(expr $i + 1); sleep 2\ndone\n\n# (3) The prefs API needs the server token. It already lives in Preferences.xml on this\n# PVC; it is never echoed here, and Plex redacts tokens in its own request log. An\n# UNCLAIMED server has none \u2014 a normal state on a fresh /config \u2014 so skip, do not fail.\nTOKEN=$(grep -o 'PlexOnlineToken=\"[^\"]*\"' \"$PX\" 2>/dev/null | head -1 | cut -d'\"' -f2)\nif [ -z \"$TOKEN\" ]; then\n  echo \"[plex-settings] not claimed yet \u2014 skipping prefs, will retry next start\" >&2\n  exit 0\nfi\n\n# Percent-encode what our values actually contain. Plex takes prefs as QUERY parameters on\n# a PUT, so a raw '/' or ',' would be read as structure rather than data.\nenc() {\n  printf '%s' \"$1\" | sed -e 's|%|%25|g' -e 's|/|%2F|g' -e 's|,|%2C|g' \\\n                         -e 's|:|%3A|g' -e 's| |%20|g'\n}\nsetpref() {\n  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 -X PUT\
+                \ \\\n    -H \"X-Plex-Token: $TOKEN\" \"$API/:/prefs?$1=$(enc \"$2\")\" 2>/dev/null)\n  if [ \"$code\" = \"200\" ]; then\n    echo \"[plex-settings] set $1\"\n  else\n    echo \"[plex-settings] WARN: $1 -> HTTP $code\" >&2\n  fi\n}\n\n# (4) THE LAN DEFINITION \u2014 the reason this hook was written. Without it Plex has NO notion\n# of a local network and tags EVERY client remote. Measured 2026-10-04: 468 requests\n# logged \"Allowed Network (WAN)\" and ZERO \"(LAN)\", including Seerr from inside the\n# cluster, which is why the server showed as Remote to LAN clients. Note allowedNetworks\n# (the ALLOWED_NETWORKS env) only waives AUTH \u2014 it does NOT make a client local. Two\n# different settings; only one of them has an env hook.\nsetpref LanNetworksBandwidth \"$LANNETS\"\n\n# (5) THE CERTIFICATE, which is what makes the `https` in ADVERTISE_IP servable. Set only\n# if the bundle exists \u2014 pointing Plex at a missing file would break its TLS listener.\nif [ -s \"$P12\" ]; then\n  setpref customCertificatePath \"$P12\"\n  setpref customCertificateKey \"$P12PASS\"\n  setpref customCertificateDomain \"$CERTDOMAIN\"\nelse\n  echo \"[plex-settings] WARN: no bundle; leaving certificate prefs untouched\" >&2\nfi\nexit 0\n"
       volumes:
       - name: config
         persistentVolumeClaim:

apps/wave-0/qbittorrent.yaml

@@ -122,7 +122,7 @@ spec:
     spec:
       containers:
       - name: qbittorrent
-        image: lscr.io/linuxserver/qbittorrent:5.1.2
+        image: lscr.io/linuxserver/qbittorrent:5.3.0
         ports:
         - containerPort: 8080
         env:
@@ -319,6 +319,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): qbittorrent-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: qbittorrent-allow-monitoring-scrape
+  namespace: qbittorrent
+  labels:
+    app.kubernetes.io/name: qbittorrent
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: qbittorrent
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-0/semaphore-db.yaml

@@ -49,7 +49,7 @@ spec:
   # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
   # pg_basebackup from the running primary, which is neither restarted nor disconnected.
   instances: 3
-  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.15
   # Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
   # semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
   inheritedMetadata:

apps/wave-1/gitea-actions.yaml

@@ -133,7 +133,7 @@ spec:
       initContainers:
         - name: dind
           restartPolicy: Always   # <- this field is what makes it a SIDECAR
-          image: docker:27-dind
+          image: docker:29-dind
           securityContext:
             privileged: true
           env:
@@ -167,7 +167,7 @@ spec:
       containers:
         # ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
         - name: runner
-          image: gitea/act_runner:0.2.11
+          image: gitea/act_runner:0.6.1
           # No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
           # registers from env on first boot and runs `act_runner daemon`. The two wait-loops
           # that used to live here are gone: kubelet now holds this container until the dind

apps/wave-1/keycloak.yaml

@@ -75,7 +75,7 @@ spec:
     spec:
       containers:
       - name: keycloak
-        image: quay.io/keycloak/keycloak
+        image: quay.io/keycloak/keycloak:26.0.6
         ports:
         - containerPort: 8080
         volumeMounts:

apps/wave-1/radarr.yaml

@@ -122,7 +122,7 @@ spec:
     spec:
       containers:
       - name: radarr
-        image: lscr.io/linuxserver/radarr:5.21.1
+        image: lscr.io/linuxserver/radarr:5.28.0
         ports:
         - containerPort: 7878
         env:
@@ -347,6 +347,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): radarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: radarr-allow-monitoring-scrape
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: radarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-1/semaphore.yaml

@@ -78,7 +78,7 @@ spec:
     spec:
       containers:
       - name: semaphore
-        image: semaphoreui/semaphore:v2.19.10
+        image: semaphoreui/semaphore:v2.19.17
         ports:
         - containerPort: 3000
         env:

apps/wave-1/sonarr.yaml

@@ -122,7 +122,7 @@ spec:
     spec:
       containers:
       - name: sonarr
-        image: lscr.io/linuxserver/sonarr:4.0.15
+        image: lscr.io/linuxserver/sonarr:4.0.20
         ports:
         - containerPort: 8989
         env:
@@ -345,6 +345,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-2/seerr.yaml

@@ -75,7 +75,7 @@ spec:
     spec:
       containers:
       - name: seerr
-        image: ghcr.io/seerr-team/seerr:v3.4.1
+        image: ghcr.io/seerr-team/seerr:v3.5.1
         ports:
         - containerPort: 5055
         volumeMounts:

catalog/gitea.yaml

@@ -10,7 +10,7 @@
 name: gitea
 kind: container
 source:
-  image: gitea/gitea:1.24.3     # pinned recent stable tag (R-UPD-01); update.range governs advancement
+  image: gitea/gitea:1.27.3     # pinned recent stable tag (R-UPD-01); update.range governs advancement
 # Headless-install bootstrap (formerly apps/gitea-bootstrap-patch.yaml; now first-class via the
 # ServiceDefinition env/envFrom/lifecycle fields, schema >=1.2.0). Gitea must come up INSTALLED
 # without the interactive web wizard and with a FIXED, persistent SECRET_KEY, so all of this is
@@ -208,6 +208,19 @@ inClusterClients:
     namespace: flux-system      # matched by NAMESPACE alone — the flux controllers' pod labels belong
                                 # to upstream's manifests, not to this platform's contract
     port: 3000                  # ONLY the Gitea HTTP port — the same single port the Gateway allow opens
+  # The PM agent service (platform/pm-agent/). Its dispatcher reads issues and posts comments
+  # over this port, and every worker Job clones, pushes and opens pull requests over it. Like
+  # Flux's source-controller above, it dials the ClusterIP DIRECTLY — deliberately, so the
+  # agent's path to the repository depends on neither public DNS, nor Cloudflare, nor the
+  # shared Gateway, nor ${BASE_DOMAIN}. And exactly as for Flux, that means this namespace's
+  # own default-deny (R-NET-01) DROPS the connection unless it is declared here: without this
+  # entry the dispatcher's egress allow is satisfied and the packet is still discarded at the
+  # Gitea end, which presents as an authentication timeout rather than as a policy denial.
+  # That is the 2026-09-13 lesson documented at length above, applied before it is relearned.
+  - name: pm-agent
+    namespace: pm-agent         # matched by NAMESPACE alone, so the dispatcher Deployment and every
+                                # ephemeral worker Job are covered by the one declaration
+    port: 3000                  # ONLY the Gitea HTTP port
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
   groups:

catalog/jackett.yaml

@@ -16,7 +16,7 @@
 name: jackett
 kind: container
 source:
-  image: lscr.io/linuxserver/jackett:latest
+  image: lscr.io/linuxserver/jackett:v0.24.3025-ls29
 exposure: internal              # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
 egress: vpn                     # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -70,7 +70,8 @@ auth:
     - ^/api/
 update:
   channel: stable               # R-UPD-01
-  range: ">=0.22.0"             # stay on the Jackett 0.22.x stable line
+  range: ">=0.22.0"             # floor only, no ceiling (0.24.x runs as of 2026-10-30; Jackett has no
+                                # major line to hold back from, so renovate.json5 mirrors no cap)
   cadence: "0 4 * * *"
   autoRollbackOnFailedHealth: true   # R-UPD-03
 observability:

catalog/keycloak.yaml

@@ -5,8 +5,15 @@
 # Validates against schemas/servicedefinition.schema.json.
 name: keycloak
 kind: container
+# DOCUMENTATION-ONLY IMAGE. The running server is NOT chosen here: the Keycloak operator
+# (platform/keycloak/kustomization.yaml, three pinned release URLs) picks it via its
+# RELATED_IMAGE_KEYCLOAK, and the Keycloak CR has no spec.image. This SD's render,
+# apps/wave-1/keycloak.yaml, is not listed in apps/wave-1/kustomization.yaml and deploys nothing
+# (ci/render-golden.sh RENDER_GOLDEN_NOT_ONBOARDED). The tag records the operator's version for
+# readers; renovate.json5 disables Renovate for this image so it cannot raise PRs against a file
+# nothing deploys. The Keycloak version moves with the operator — update this tag by hand then.
 source:
-  image: quay.io/keycloak/keycloak
+  image: quay.io/keycloak/keycloak:26.0.6
 exposure: public              # login/redirect flows for internet-exposed services; hardened (docs/15; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:

catalog/pihole.yaml

@@ -5,7 +5,7 @@
 name: pihole
 kind: container
 source:
-  image: pihole/pihole
+  image: pihole/pihole:2026.10.1
 exposure: internal            # LAN-only admin UI (docs/13 row 6; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
@@ -134,8 +134,48 @@ env:
   # then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
   # characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
   # while resolution for the zone does not work. Confirmed both ways against the live image.
+  # plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the LAN HALF of Plex's single advertised
+  # hostname. Plex publishes ONE URL to plex.tv for both local and remote clients
+  # (catalog/plex.yaml ADVERTISE_IP): on the LAN this answer points it at the tcp/32400
+  # LoadBalancer, while Cloudflare answers the same name publicly with the WAN address, NATed to
+  # the same VIP. WITHOUT THIS LINE the entry above answers it with ${GATEWAY_VIP} via the zone
+  # wildcard, where NOTHING listens on 32400 — measured 2026-10-04, plex-vip.${BASE_DOMAIN}
+  # resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
+  # failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
+  # every client that takes the ADVERTISED connection cannot reach the server at all.
+  #
+  # ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
+  # MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
+  #
+  #   dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+  #   dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+  #   dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+  #
+  # dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
+  # bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
+  #   * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
+  #     single lookup, which is maximum exposure to any momentary gap; while
+  #   * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
+  #     upstream (standard forwarder behaviour when the forward target does not answer) it latches
+  #     the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
+  # The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
+  # live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
+  # flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
+  # That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
+  #
+  # 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
+  # re-election (both seconds), short enough that a genuine VIP change still propagates inside a
+  # minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
+  # propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
+  # LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
+  # the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
+  # cache expires, the router falls back, and the public answer latches again. That is a real
+  # outage, and the fix for it is PiHole availability, not DNS caching.
+  #
+  # Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
+  # configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
   - name: FTLCONF_misc_dnsmasq_lines
-    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
+    value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
   # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
   # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
   # carries the full rationale and the measurement that found it.

catalog/plex.yaml

@@ -12,7 +12,7 @@
 name: plex
 kind: container
 source:
-  image: plexinc/pms-docker:latest
+  image: plexinc/pms-docker:1.43.6.11085-d1e52e169
 # plexinc/pms-docker env. Rendered VERBATIM onto the app container by render.py (schema env list).
 env:
   # PLEX_UID/PLEX_GID: the image boots as root (s6 PID1), chowns /config to this uid/gid, then drops
@@ -72,54 +72,227 @@ env:
       secretKeyRef:
         name: plex-claim
         key: PLEX_CLAIM
-  # ADVERTISE_IP ("Custom server access URLs") — NOW SET. This replaces a DEFERRAL note that stood
+  # ADVERTISE_IP ("Custom server access URLs") — the connection Plex publishes to plex.tv.
-  # here until 2026-10-04, and the deferral's premise has changed: it weighed only ONE candidate,
-  # `https://plex.home.keithjackson.dev` through the L7 Gateway. The operator chose the NATIVE L4
-  # path instead on 2026-10-03 (apps/wave-0/plex-remote.yaml) so streams go straight to
-  # ${PLEX_L4_VIP}:32400 and bypass the Envoy hop. What was still missing was a URL whose TLS
-  # CERTIFICATE validates — and getting that wrong is what took the service down.
   #
-  # THE VALUE IS A `plex.direct` NAME, NOT THE RAW VIP, AND THAT IS THE WHOLE POINT. Plex serves
+  # THE VALUE MUST BE RESOLVABLE BY EVERY CLIENT, NOT JUST BY CLIENTS THAT USE OUR DNS. That single
-  # 32400 with its own certificate; measured live 2026-10-04 it carries exactly one name:
+  # property is what this field keeps getting wrong, so it is the first thing to check before
-  #     CN = *.<serverhash>.plex.direct      (one SAN, no IP SANs at all)
+  # changing it. A Plex client resolves the advertised host ITSELF, with whatever resolver it
-  # so `https://<ip>:32400` can NEVER validate. `<dashed-ip>.<serverhash>.plex.direct` is Plex's own
+  # happens to use — and on a real LAN that is frequently NOT PiHole: browsers ship DoH on by
-  # mechanism for this: it is PUBLIC DNS that resolves to the embedded address, so the name points at
+  # default, phones have "private DNS", guest and IoT devices often take the router's DNS, and some
-  # the LAN VIP while the wildcard cert above matches it. Measured 2026-10-04 with FULL verification
+  # clients hard-code a public resolver.
-  # (no `curl -k`):
-  #     https://192-168-2-24.<hash>.plex.direct:32400/identity -> 200, TLS verify OK
-  #     https://192.168.2.24:32400/identity                    -> 000, cert validation FAILED
   #
-  # THIS FIELD EXISTS BECAUSE AUTO-DISCOVERY CANNOT WORK HERE. Left empty, Plex publishes the
+  # THAT IS NOT THEORETICAL — IT TOOK PLEX DOWN FOR THE WHOLE LAN ON 2026-10-05. This field briefly
-  # addresses it can see itself — which from a pod is the overlay IP (`10.0.2.136`), useless to a LAN
+  # advertised `plex-vip.${BASE_DOMAIN}`, a name that exists ONLY in PiHole's split-horizon answers
-  # client. The VIP has to be advertised explicitly; there is no "just clear it" fix.
+  # (catalog/pihole.yaml) and in no public zone. Clients that did not use PiHole could not resolve
+  # it; the SOHO router answered the failed lookup with ITSELF, and every one of them landed on the
+  # router's own web UI and reported a certificate error for `CN = %D`. Measured 2026-10-05: the
+  # router serves that self-signed cert (`C = ZZ, O = %D342e7e6b, CN = %D`) on BOTH `192.168.2.1:443`
+  # and the WAN address, while PiHole itself was answering perfectly — both replicas returned
+  # `plex-vip -> ${PLEX_L4_VIP}` and `plex -> ${GATEWAY_VIP}`. The DNS was right; the DEPENDENCY on
+  # it was the bug. Nothing Plex advertises listens on 443, which is how we know the clients had left
+  # the Plex path entirely and were talking to the router.
   #
-  # WHAT WENT WRONG, recorded so it is not re-introduced by hand. customConnections OVERRIDES Plex's
+  # SO: `<dashed-ip>.<serverhash>.plex.direct`, WHICH IS RESOLVER-INDEPENDENT. This is Plex's own
-  # own publishing, and on 2026-10-04 it held two hand-typed URLs, both uncompletable:
+  # mechanism and the reason it exists: `plex.direct` is a PUBLIC zone that returns the private
-  #   - `https://192.168.2.24:32400`              -> cert cannot cover an IP. Measured: a storm of
+  # address embedded in the label, so EVERY resolver on earth — PiHole, the router, 1.1.1.1, a DoH
-  #     `CERT: incomplete TLS handshake ... sslv3 alert certificate unknown` in Plex's log every few
+  # browser — answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no
-  #     seconds, from REAL LAN clients arriving through the VIP.
+  # split-horizon entry, no Cloudflare record, and no certificate work, because the wildcard cert
-  #   - `https://plex.home.keithjackson.dev:32400` -> nothing listens. The Gateway serves this host on
+  # Plex already serves on 32400 covers exactly this name. Re-verified live 2026-10-05 with FULL
-  #     443; measured `http=000` against 192.168.2.20:32400.
+  # validation (no `curl -k`): resolves to 192.168.2.24 and `/identity` returns **200, TLS verified**.
-  # With every advertised connection dead, plex.tv had nothing publishable, sat at
-  # `mapping state 'Mapped - Not Published'`, fell back to Plex Relay, and THE SERVER DISAPPEARED
-  # FROM THE ACCOUNT — clients reported "not authorized".
   #
-  # DECLARING IT HERE rather than in the web UI is the durable half of the fix. The image's
+  # WHY NOT THE NICER HOSTNAME, since a readable name in our own zone is obviously more pleasant:
-  # /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections" "${ADVERTISE_IP}"` (line 127)
+  # it cannot be made to work for clients that bypass PiHole. A public record for it would resolve
-  # OUTSIDE the PLEX_CLAIM/PlexOnlineToken guard, and its `/.firstRunComplete` marker sits on the
+  # to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too —
-  # container's OVERLAY fs, not the /config PVC — so this is re-applied on EVERY pod start and git,
+  # so the hostname is correct ONLY for clients using PiHole, and wrong for the rest. There is no
-  # not a form field, owns the value. Editing this env rolls the Deployment, which is what applies it.
+  # configuration of that name that is right for both. `plex.direct` is right for both.
+  # (The PiHole `plex-vip` answer is KEPT — it is harmless, costs nothing, and is genuinely useful
+  # for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises.)
   #
-  # TWO LIMITS, stated rather than discovered later:
+  # THE TWO EMBEDDED VALUES, and the honest cost of this choice. The dashed IP must track
-  #   - THE HASH IS SERVER IDENTITY, which is why the whole URL is a cluster-var and not built from
+  # ${PLEX_L4_VIP} and `<serverhash>` is this server's certificate identity — read it from the live
-  #     ${PLEX_L4_VIP}: envsubst cannot dash an IP, and the hash is not derivable from anything in
+  # listener with
-  #     this tree. It is not a secret (it ships in the cert and in the published connection list), but
+  #     openssl s_client -connect <vip>:32400 </dev/null | openssl x509 -noout -subject
-  #     if Plex reissues under a new hash this URL goes stale and must be re-read from the live cert.
+  # Nothing in this tree can derive either, and nothing checks that they still agree: envsubst
-  #   - THIS RESTORES LAN CLIENTS AND THE ACCOUNT LISTING, NOT WAN. Remote clients still need the
+  # cannot dash an IP, so this is a literal. That fragility is real and was the reason the hostname
-  #     off-repo tcp/32400 port-forward (runbooks/plex-remote-access.md). Once that exists, the WAN
+  # was tried in the first place — it is accepted here because an unresolvable name is a WORSE
-  #     twin of this URL can be appended comma-separated — customConnections takes a list.
+  # failure (silent, LAN-wide, and it points users at the router) than a stale literal, which at
+  # least fails loudly and only after someone changes the VIP.
+  #
+  # The image's /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections"
+  # "${ADVERTISE_IP}"` (line 127, verified in the running image), OUTSIDE the
+  # PLEX_CLAIM/PlexOnlineToken guard, with its `/.firstRunComplete` marker on the container OVERLAY
+  # fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not a web
+  # form, owns the value.
   - name: ADVERTISE_IP
-    value: ${PLEX_ADVERTISE_URL}
+    value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
+  # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
+  # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
+  # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this
+  # workload). The postStart hook converts it to the PKCS#12 bundle Plex requires.
+  #
+  # WHY ENV AND NOT A VOLUME, which would be the conventional choice. MEASURED 2026-10-04:
+  # `schemas/servicedefinition.schema.json` `storage` accepts only
+  # [class, export, mountPath, name, readOnly, size, subPath] with `additionalProperties: false`,
+  # `catalog/render.py` emits NO secret volumes, and no rendered app mounts a Secret as a volume.
+  # Mounting one therefore needs a schema + renderer change; `env.valueFrom.secretKeyRef` is
+  # already in the schema and already used by PLEX_CLAIM above. PEM is text, so it survives an env
+  # var intact (a raw PKCS#12 would not — it is binary and carries NUL bytes).
+  #
+  # COST, STATED PLAINLY: the private key is readable from this container's environment
+  # (/proc/1/environ, `kubectl exec env`) by anything that can already exec into the pod. That is
+  # the same audience that can read the converted bundle under /config, so the marginal exposure is
+  # small — but a secret VOLUME would still be the better shape, and adding secret-volume support
+  # to the schema is the clean follow-up that removes this note.
+  #
+  # `optional: true` IS LOAD-BEARING, NOT DEFENSIVE CLUTTER (schema 1.11.0, added for this). The
+  # Certificate is issued INTO THIS NAMESPACE, and the namespace is created by the rendered
+  # apps/wave-0/plex.yaml itself — so the cert CANNOT be issued in an earlier tier, and the
+  # Certificate and this Deployment land in the SAME reconcile. On a cold bootstrap the Secret is
+  # therefore briefly absent while cert-manager completes a DNS01 challenge. With a HARD ref, Plex
+  # would sit in CreateContainerConfigError until a certificate issued — and because `apps-wave-0`
+  # is `wait: true` with `apps-wave-1` dependsOn it, a failed or slow issuance would STALL THE WHOLE
+  # APP TIER (the same hazard the `hardware` block below documents for the GPU resource). Optional
+  # instead means: no cert -> Plex still starts and serves, the postStart hook logs that the
+  # certificate was not installed, and the next start after issuance converges. Plex falls back to
+  # its own `*.plex.direct` certificate in the meantime, so the degraded state is "advertised URL
+  # not yet servable over TLS", not "media server down".
+  - name: PLEX_TLS_CRT
+    valueFrom:
+      secretKeyRef:
+        name: plex-vip-tls
+        key: tls.crt
+        optional: true
+  - name: PLEX_TLS_KEY
+    valueFrom:
+      secretKeyRef:
+        name: plex-vip-tls
+        key: tls.key
+        optional: true
+lifecycle:
+  # LEVEL-TRIGGERED PLEX SETTINGS, re-asserted on EVERY container start — the same pattern and the
+  # same reasoning as catalog/pihole.yaml's and catalog/qbittorrent.yaml's postStart: desired state
+  # lives in git and is re-asserted unconditionally rather than typed into a web form once. Typing
+  # settings into the web form is precisely what took this service down on 2026-10-04.
+  #
+  # IT EXISTS BECAUSE THE IMAGE HAS NO ENV FOR THESE SETTINGS. Measured 2026-10-04 across the whole
+  # init surface (/etc/cont-init.d, /etc/services.d, /plex-common.sh, /usr/local/bin): the image
+  # calls setPref for exactly SIX prefs — MachineIdentifier, ProcessedMachineIdentifier,
+  # PlexOnlineToken, customConnections, allowedNetworks, TranscoderTempDirectory — and supports NO
+  # generic PLEX_PREFERENCE* hook of any kind. The two things we need (a LAN definition and a custom
+  # certificate) are reachable ONLY through Plex's own API, so this hook is the only GitOps path.
+  #
+  # IT MUST NEVER FAIL. A postStart hook that exits non-zero makes kubelet KILL the container, so
+  # every step is bounded and the script ends with an unconditional `exit 0`. A missing token, an
+  # unissued certificate or an unreachable API degrades to "not applied this start" and is retried
+  # on the next one; it never crashloops Plex.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # NOTE — LOAD-BEARING: SHELL VARIABLES HERE ARE WRITTEN BARE, NEVER BRACE-WRAPPED.
+          # This whole script is rewritten by Flux postBuild envsubst BEFORE it ever runs, so a
+          # brace-wrapped name is read as a cluster-vars knob: an undeclared one is a HARD
+          # Kustomization failure under StrictPostBuildSubstitutions, and a declared one would be
+          # replaced with its value. Shell vars are therefore `$PX`, `$TOKEN`, `$P12` — bare. Only
+          # BASE_DOMAIN below is brace-wrapped, because it IS a knob and IS meant to be
+          # substituted. Command substitution `$(cmd)` is left alone and needs no escaping. A shell
+          # variable written with braces would be blanked or would fail the whole Kustomization —
+          # this repo has already lost a gitea OIDC bootstrap to exactly that mistake, and
+          # catalog/render.py now refuses to emit it (which is how this very comment got caught).
+          # DELIBERATELY NO `set -u`. The TLS env vars are declared `optional: true`, so when the
+          # certificate has not issued yet they are genuinely UNSET, not empty — and under `set -u`
+          # the first `[ -n "$PLEX_TLS_CRT" ]` would abort the script, which for a postStart hook
+          # means kubelet KILLS THE CONTAINER. That is the precise failure the `optional` flag exists
+          # to prevent, so turning it back on here would defeat the whole design. (The alternative,
+          # a brace-wrapped `:-` default, would need envsubst escaping — the trap the note above
+          # describes.) Every value below is explicitly guarded instead.
+          PX="/config/Library/Application Support/Plex Media Server/Preferences.xml"
+          CERTDIR=/config/certs
+          P12=$CERTDIR/plex-vip.p12
+          # The PKCS#12 passphrase protects nothing and is deliberately not a secret: Plex needs it
+          # in cleartext in customCertificateKey, and it and the bundle sit on the same PVC. It is a
+          # container-format requirement, not an access control.
+          P12PASS=plexvip
+          LANNETS='10.0.0.0/8,192.168.0.0/16'
+          CERTDOMAIN=plex-vip.${BASE_DOMAIN}
+          API=http://127.0.0.1:32400
+
+          # (1) Build the PKCS#12 bundle from the PEM the Secret supplies. Rebuilt on every start so
+          # a cert-manager RENEWAL is picked up with no extra moving part.
+          if [ -n "$PLEX_TLS_CRT" ] && [ -n "$PLEX_TLS_KEY" ]; then
+            mkdir -p $CERTDIR && chmod 700 $CERTDIR
+            printf '%s' "$PLEX_TLS_CRT" > $CERTDIR/tls.crt
+            printf '%s' "$PLEX_TLS_KEY" > $CERTDIR/tls.key
+            chmod 600 $CERTDIR/tls.key
+            if openssl pkcs12 -export -out $P12.tmp -inkey $CERTDIR/tls.key \
+                 -in $CERTDIR/tls.crt -passout pass:$P12PASS 2>/dev/null; then
+              mv -f $P12.tmp $P12
+              chown 1000:1000 $P12 2>/dev/null || true
+              chmod 600 $P12
+              echo "[plex-settings] PKCS#12 bundle written"
+            else
+              rm -f $P12.tmp
+              echo "[plex-settings] WARN: pkcs12 export failed; keeping any previous bundle" >&2
+            fi
+            rm -f $CERTDIR/tls.key
+          else
+            echo "[plex-settings] WARN: TLS env empty; certificate not installed this start" >&2
+          fi
+
+          # (2) Wait, bounded (~90s), for the local API. /identity needs no auth.
+          i=0
+          while [ "$i" -lt 45 ]; do
+            if [ "$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 $API/identity 2>/dev/null)" = "200" ]; then
+              break
+            fi
+            i=$(expr $i + 1); sleep 2
+          done
+
+          # (3) The prefs API needs the server token. It already lives in Preferences.xml on this
+          # PVC; it is never echoed here, and Plex redacts tokens in its own request log. An
+          # UNCLAIMED server has none — a normal state on a fresh /config — so skip, do not fail.
+          TOKEN=$(grep -o 'PlexOnlineToken="[^"]*"' "$PX" 2>/dev/null | head -1 | cut -d'"' -f2)
+          if [ -z "$TOKEN" ]; then
+            echo "[plex-settings] not claimed yet — skipping prefs, will retry next start" >&2
+            exit 0
+          fi
+
+          # Percent-encode what our values actually contain. Plex takes prefs as QUERY parameters on
+          # a PUT, so a raw '/' or ',' would be read as structure rather than data.
+          enc() {
+            printf '%s' "$1" | sed -e 's|%|%25|g' -e 's|/|%2F|g' -e 's|,|%2C|g' \
+                                   -e 's|:|%3A|g' -e 's| |%20|g'
+          }
+          setpref() {
+            code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 -X PUT \
+              -H "X-Plex-Token: $TOKEN" "$API/:/prefs?$1=$(enc "$2")" 2>/dev/null)
+            if [ "$code" = "200" ]; then
+              echo "[plex-settings] set $1"
+            else
+              echo "[plex-settings] WARN: $1 -> HTTP $code" >&2
+            fi
+          }
+
+          # (4) THE LAN DEFINITION — the reason this hook was written. Without it Plex has NO notion
+          # of a local network and tags EVERY client remote. Measured 2026-10-04: 468 requests
+          # logged "Allowed Network (WAN)" and ZERO "(LAN)", including Seerr from inside the
+          # cluster, which is why the server showed as Remote to LAN clients. Note allowedNetworks
+          # (the ALLOWED_NETWORKS env) only waives AUTH — it does NOT make a client local. Two
+          # different settings; only one of them has an env hook.
+          setpref LanNetworksBandwidth "$LANNETS"
+
+          # (5) THE CERTIFICATE, which is what makes the `https` in ADVERTISE_IP servable. Set only
+          # if the bundle exists — pointing Plex at a missing file would break its TLS listener.
+          if [ -s "$P12" ]; then
+            setpref customCertificatePath "$P12"
+            setpref customCertificateKey "$P12PASS"
+            setpref customCertificateDomain "$CERTDOMAIN"
+          else
+            echo "[plex-settings] WARN: no bundle; leaving certificate prefs untouched" >&2
+          fi
+          exit 0
 # HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
 # INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
 # Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.
@@ -178,7 +351,7 @@ exposure: public              # edge port-forward + public DNS; clients + Seerr
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: config
-    size: 8Gi
+    size: 16Gi
     class: longhorn           # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
     mountPath: /config        # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
                               # renderer default /data/config. Without this the PVC sat unused and Plex wrote to the

catalog/qbittorrent.yaml

@@ -30,7 +30,7 @@
 name: qbittorrent
 kind: container
 source:
-  image: lscr.io/linuxserver/qbittorrent:5.1.2   # pinned stable tag (R-UPD-01); update.range governs advancement
+  image: lscr.io/linuxserver/qbittorrent:5.3.0   # pinned stable tag (R-UPD-01); update.range governs advancement
 # LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
 # LOAD-BEARING FOR THE BULK MOUNT, not cosmetic — exactly as on sonarr/radarr. Measured on prod
 # 2026-09-12 BEFORE this change: `id` inside the container reported the qbittorrent-nox process as
@@ -359,7 +359,7 @@ storage:
                                 # The live /config contents were copied onto each PVC before this landed,
                                 # so API keys and credentials other apps reference were preserved.
   # THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
-  # (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
+  # (192.168.2.30:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
   # /downloads, which is precisely where this app is ALREADY configured to write.
   #
   # WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
@@ -434,7 +434,7 @@ dependencies: []                # default-deny: reaches only DNS; arr reach it n
                                 # for weeks and neither has any CNP naming the NFS server or port 2049
                                 # (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
                                 # egress gateway off this traffic (its selector is a podSelector), and
-                                # 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
+                                # 192.168.2.30 is inside the 192.168.0.0/16 excludedCIDR besides — see
                                 # platform/vpn-egress/egress-steering.yaml, "BULK NFS".
 ingressRoutes:
   - host: qbittorrent.${BASE_DOMAIN}

catalog/radarr.yaml

@@ -8,7 +8,7 @@
 name: radarr
 kind: container
 source:
-  image: lscr.io/linuxserver/radarr:5.21.1   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
+  image: lscr.io/linuxserver/radarr:5.28.0   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
 # LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
 # LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
 # which cannot write a 1000:3000-owned 0755 tree.

catalog/render.py

@@ -156,11 +156,17 @@ OAUTH2_COOKIE_REFRESH = "2m30s"
 # ready endpoints. The flag is confirmed present in the PINNED v7.8.1 binary (`--help`),
 # which matters because the shared proxy is on v7.15.3 and could not have vouched for it.
 #
-# THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED. A container port is not a
+# THE SCRAPE WORKS AS OF 2026-10-05 — but it took THREE independent pieces, and this flag is
-# Service port: these sidecars' Services expose the app port and :4180, and each namespace's
+# only the first. A container port is not a Service port, and each namespace's default-deny
-# default-deny baseline has `ingress: []`, so Prometheus can neither resolve nor reach this
+# baseline denies all ingress, so the flag alone bought nothing for 63.7h:
-# port until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
+#   1. this flag + a named `metrics` container port   (PR #28)
-# are added. Do not read this flag as "metrics are working".
+#   2. a monitor that targets POD ports, not Service ports — platform/oauth2-proxy/podmonitor.yaml
+#      is a PodMonitor for exactly that reason (no Service exposes 44180)
+#   3. an ingress allowance from ns monitoring        (PR #49, r_monitoring_ingress_policy below)
+# Verified on prod 2026-10-05: all 6 targets up == 1, scraping in 1-6ms.
+# STILL do not read this flag alone as "metrics are working" — 1 without 3 is precisely the state
+# that produced 5 x TargetDown for 63.7h. If you add a sidecar metrics port anywhere else, check
+# that r_monitoring_ingress_policy covers it.
 OAUTH2_PROXY_METRICS_PORT = 44180
 
 # --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
@@ -1362,8 +1368,9 @@ def _oauth2_sidecar_container(s: dict) -> dict:
         f"--cookie-refresh={OAUTH2_COOKIE_REFRESH}",
         "--pass-access-token=true",
         "--set-xauthrequest=true",
-        # Start the /metrics listener. Prerequisite only — not yet reachable by Prometheus;
+        # Start the /metrics listener. Scraped as of 2026-10-05, but only because
-        # see OAUTH2_PROXY_METRICS_PORT above for exactly what is still missing.
+        # r_monitoring_ingress_policy also opens this port to ns monitoring — the flag alone was
+        # inert for 63.7h. See OAUTH2_PROXY_METRICS_PORT above for the three pieces required.
         f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}",
     ]
     secret = f"{s['name']}-oauth2-proxy"
@@ -1541,12 +1548,54 @@ def r_monitoring_ingress_policy(s: dict):
     trap Longhorn's own NetworkPolicies set (platform/longhorn/networkpolicy-metrics.yaml exists
     for exactly this reason), and it is a THIRD independent way for a metrics SLI to be inert.
 
-    Rendered only alongside a declared metricsEndpoint, and it opens ONLY that port to ONLY the
+    Rendered only alongside a port something actually serves, and it opens ONLY that port to ONLY
-    monitoring namespace: an allow for a port nothing serves would be a standing hole for no
+    the monitoring namespace: an allow for a port nothing serves would be a standing hole for no
     benefit.
+
+    TWO INDEPENDENT SOURCES OF A SERVED METRICS PORT, and this used to cover only the first:
+
+      1. `observability.metricsEndpoint` — the APP's own exporter. NOTE, measured 2026-10-04: no
+         ServiceDefinition in this catalog declares one, so before the sidecar clause below this
+         function had NEVER emitted a single object on any branch. It is correct but, so far,
+         entirely latent — do not read its presence as evidence that scrape ingress is solved.
+
+      2. THE forward-auth GATE SIDECAR's own `/metrics` on 44180 (added 2026-10-04). Every
+         `_needs_oauth2_sidecar` app gets `--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}`
+         and a named `metrics` container port from `_oauth2_sidecar_container`, so the port is
+         served whenever the sidecar exists — unconditionally, which is what makes it safe to
+         open without a per-app declaration.
+
+    WHY CLAUSE 2 KEYS ON `_needs_oauth2_sidecar` AND NOT ON `observability.metrics`: it must match
+    the condition the SCRAPE keys on, exactly. platform/oauth2-proxy/podmonitor.yaml selects pods
+    by the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL and nothing else, so a
+    target exists iff the sidecar exists. Keying the allowance on `observability.metrics` instead
+    would let `metrics: false` produce a discovered-but-blocked target — the same "two halves
+    authored apart" failure this function exists to prevent, just moved one field over.
+
+    THIS CLAUSE IS A BUG FIX, and the bug was live for 63.7h. Measured on prod 2026-10-04: all 6
+    sidecar targets across jackett/radarr/sonarr/qbittorrent/pihole read `health=down`,
+    `lastError: context deadline exceeded`, `lastScrapeDuration` exactly 10.0s — the scrapeTimeout,
+    i.e. packets DROPPED, not refused. Each app namespace ships a `default-deny` NetworkPolicy
+    with `policyTypes: [Ingress, Egress]` and NO `ingress` key at all (deny-all-ingress), and the
+    only ingress allowance is `<app>-allow-gateway-ingress`: `fromEntities: [ingress]` -> 4180.
+    Prometheus matched no clause. 5 x `TargetDown` + 6 x `OAuth2ProxyGateUnobserved` had been
+    re-notifying at ~22 emails/day since the PodMonitor landed on 2026-10-02.
+
+    WHY THE POLICY BELONGS HERE AND NOT IN platform/oauth2-proxy: podmonitor.yaml's header records
+    the objection — a policy in that tier would name app namespaces created by the SIBLING
+    apps-wave-0/1 tiers (measured Flux graph: `oauth2-proxy` and `apps-wave-0` both
+    `dependsOn=[platform-config]`, no ordering between them), so it would apply on a running
+    cluster and fail on a cold bootstrap. Rendered here it ships in the same tier that creates the
+    namespace, which dissolves that ordering problem entirely.
     """
-    ep = s["observability"].get("metricsEndpoint")
+    obs = s["observability"]
-    if not ep or not s["observability"]["metrics"]:
+    ep = obs.get("metricsEndpoint")
+    ports: list[str] = []
+    if ep and obs["metrics"]:
+        ports.append(str(ep["port"]))
+    if _needs_oauth2_sidecar(s):
+        ports.append(str(OAUTH2_PROXY_METRICS_PORT))
+    if not ports:
         return None
     return {
         "apiVersion": "cilium.io/v2",
@@ -1564,7 +1613,7 @@ def r_monitoring_ingress_policy(s: dict):
                         {"matchLabels": {"k8s:io.kubernetes.pod.namespace": MONITORING_NS}}
                     ],
                     "toPorts": [
-                        {"ports": [{"port": str(ep["port"]), "protocol": "TCP"}]}
+                        {"ports": [{"port": p, "protocol": "TCP"} for p in ports]}
                     ],
                 }
             ],

catalog/seerr.yaml

@@ -24,7 +24,7 @@
 name: seerr
 kind: container
 source:
-  image: ghcr.io/seerr-team/seerr:v3.4.1   # Seerr: maintained Overseerr/Jellyseerr successor, Plex-native request UI (docs/13 row 5); served on 5055
+  image: ghcr.io/seerr-team/seerr:v3.5.1   # Seerr: maintained Overseerr/Jellyseerr successor, Plex-native request UI (docs/13 row 5); served on 5055
 # The ghcr.io/seerr-team/seerr image runs as the non-root user `node:node` (UID 1000 / GID 1000 —
 # the standard node base-image user; confirmed from the image config's User field). On boot the app
 # creates /app/config/logs and writes its settings/DB under the mounted /app/config PVC. A fresh

catalog/semaphore.yaml

@@ -22,7 +22,7 @@
 name: semaphore
 kind: container
 source:
-  image: semaphoreui/semaphore:v2.19.10   # pinned recent stable tag (R-UPD-01); update.range governs advancement
+  image: semaphoreui/semaphore:v2.19.17   # pinned recent stable tag (R-UPD-01); update.range governs advancement
 env:
   # --- PostgreSQL connection (the CNPG-DB pattern). DIALECT/HOST/PORT/DB/USER are literals; PASS is
   #     sourced from the in-namespace SOPS secret (the CNPG -app secret lives in ns semaphore-db,

catalog/sonarr.yaml

@@ -8,7 +8,7 @@
 name: sonarr
 kind: container
 source:
-  image: lscr.io/linuxserver/sonarr:4.0.15   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
+  image: lscr.io/linuxserver/sonarr:4.0.20   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
 # LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
 # LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
 # which cannot write a 1000:3000-owned 0755 tree.

ci/lint.sh

@@ -27,3 +27,34 @@ fi
 
 yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}"
 ok "yaml/lint passed (${#files[@]} files)"
+
+# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
+# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
+# while the gate stayed green. Valid YAML, so yamllint cannot see it.
+missing=0
+while IFS=: read -r wf _ value; do
+  value="${value#*configurationFile:}"
+  value="$(echo "$value" | sed -E 's/#.*//; s/^[[:space:]"'\'']+//; s/[[:space:]"'\'']+$//')"
+  if [ -n "$value" ] && [ ! -e "$value" ]; then
+    err "${wf}: configurationFile '${value}' does not exist in the repo"
+    missing=1
+  fi
+done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true)
+[ "$missing" -eq 0 ] || exit 1
+ok "workflow configurationFile references resolve"
+
+# A workflow must not use an action that needs a Docker daemon: jobs on this runner get NONE, by
+# decision (decisions/2026-10-29-image-pipeline.md; act_runner maps docker_host to `-`). Such an
+# action is valid YAML and passes every other stage, then dies at runtime with
+# "failed to connect to the docker API at unix:///var/run/docker.sock". renovatebot/github-action
+# did exactly that on every Renovate run until 2026-10-30 (run 343). Run the tool's image as the
+# job `container:` instead (see .gitea/workflows/renovate.yaml). Container (`docker://`) actions
+# and Docker's own daemon-bound actions are rejected for the same reason. Comments are ignored.
+daemon_uses='uses:[[:space:]]*["'\'']?(docker://|renovatebot/github-action|docker/(build-push|setup-buildx|setup-qemu|login)-action)'
+bad=0
+while IFS= read -r hit; do
+  err "${hit%%:*}: needs a Docker daemon, which jobs here do not get: $(echo "${hit#*:*:}" | sed -E 's/^[[:space:]]+//')"
+  bad=1
+done < <(grep -HnE "^[^#]*${daemon_uses}" .gitea/workflows/*.y*ml 2>/dev/null || true)
+[ "$bad" -eq 0 ] || exit 1
+ok "no workflow uses a Docker-daemon-bound action"

ci/render-keycloak-passkey-policy.py

@@ -0,0 +1,490 @@
+#!/usr/bin/env python3
+# =============================================================================
+# ci/render-keycloak-passkey-policy.py
+#
+# Render platform/keycloak/passkey-desired-state.yaml FROM
+# platform/keycloak/keycloak-realm-import.yaml.
+#
+# WHY THIS EXISTS
+# ---------------
+# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
+# Done=True / Started=False it never re-runs. Measured on prod 2026-10-18, hours
+# after the passkey declaration merged:
+#
+#     kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath=...status
+#     Done=True  Started=False  HasErrors=False
+#
+# So the WebAuthn passwordless policy and the restructured `browser-mfa` flow that
+# keycloak-realm-import.yaml declares are INERT. They are in git, Flux applied
+# them, and they reach the realm never. platform/keycloak/passkey-reconciler.yaml
+# is the level-triggered loop that closes that; this renderer supplies the DESIRED
+# STATE it converges towards.
+#
+# THE ONE VALUE THAT IS NOT RENDERED: `bind-browser-flow`, git's switch for whether the
+# reconciler may BIND its verified candidate browser flow (one atomic PUT of the
+# realm's browserFlow). It is an operator DECISION -- "I have tested the candidate
+# flow in a browser; make it the login path" -- not a fact derivable from the realm
+# import, so it cannot be generated. It is instead CARRIED THROUGH from the committed
+# file: --write preserves it, --check accepts any value of it that is exactly the
+# string "true" or "false" and rejects everything else (an unquoted YAML true is a
+# bool, which a ConfigMap rejects at apply time -- failing the whole platform-config
+# Kustomization -- so it is refused HERE, at merge time). An absent file or key
+# renders as "false". Going live is therefore a one-line PR flipping that line.
+#
+# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
+# automountServiceAccountToken: false, exactly like its client-secret sibling. It
+# has NO Kubernetes API access at all, by design (R-SEC-02) — so it cannot read the
+# KeycloakRealmImport CR to learn what the realm is supposed to look like. Its input
+# must be MOUNTED. Hence a ConfigMap.
+#
+# THAT IS ALSO WHY THIS IS A RENDERER AND NOT A HAND-WRITTEN ConfigMap. A
+# hand-authored copy of the policy would be a SECOND SOURCE OF TRUTH for the RP ID,
+# the user-verification requirement and the whole flow tree — free to disagree with
+# the realm import. "Two declarations that can disagree" is the same shape as the
+# bug this whole effort is closing. So the ConfigMap is GENERATED from the realm
+# import and the generation is GATED:
+# conformance/tests/l0-static/keycloak-passkey-reconciler/check.sh runs this script
+# with --check on every merge, and the merge fails if the committed file is not what
+# this renderer produces. Same render==golden idiom as
+# ci/render-keycloak-client-urls.py, catalog/render.py and ci/render-golden.sh.
+#
+# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. webAuthnPolicyPasswordlessRpId is
+# written in the realm import as the Flux substitution token ${BASE_DOMAIN}, and it
+# is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files are
+# rendered by the SAME Flux Kustomization (./platform-config, whose
+# postBuild.substituteFrom reads cluster-vars), so both are substituted in the same
+# pass with the same value and cannot disagree about what the base domain is. This
+# renderer never parses, splits or reconstructs a hostname and never reads
+# cluster-vars — there is no second place the domain is decided.
+#
+# THE ONE THING THIS RENDERER DERIVES RATHER THAN COPIES
+# -----------------------------------------------------
+# The enrolment required action, `webauthn-register-passwordless`. It is
+# deliberately NOT declared in the realm import, and the import's own header
+# explains at length why declaring it there would be actively DANGEROUS: Keycloak
+# 26.0.6 treats `requiredActions` in a realm representation as REPLACE, not merge,
+# so declaring one action imports exactly that action plus delete_account and skips
+# DefaultRequiredActions.addActions entirely — deleting CONFIGURE_TOTP,
+# UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing CONFIGURE_TOTP is a hard
+# lock-out of every platform-admin, because auth-otp-form in the browser flow still
+# demands a TOTP they could then never enrol.
+#
+# So the required action cannot be copied from the import, because it is correctly
+# absent from it. It is instead DERIVED, as a consequence of what the import DOES
+# declare:
+#
+#   IF the declared browser flow contains a `webauthn-authenticator-passwordless`
+#   execution, THEN passkey logins are offered, THEREFORE the enrolment action must
+#   be registered and enabled — and NOT a defaultAction, because the import's whole
+#   stated posture is "additive and optional" and a defaultAction forces enrolment
+#   on every user at next login.
+#
+# This is a derivation, not a second declaration: there is no value here that a
+# human could edit into disagreement with the import. If the import stops declaring
+# the passkey execution, this renderer stops emitting the required action (and
+# fails, because then the whole desired state is empty of passkey intent).
+#
+# WHAT IS DELIBERATELY NOT EXPORTED
+#   * the TWO-FACTOR webAuthnPolicy* family. Nothing in this realm uses the
+#     `webauthn-authenticator` two-factor authenticator; the passwordless
+#     authenticator reads getWebAuthnPolicyPasswordless() only, so a policy written
+#     into the two-factor family is a silent no-op. Exporting it would hand the
+#     reconciler a write surface with no purpose.
+#   * `clients`, `users`, `groups`, `roles`. Client secrets and client URLs are the
+#     SIBLING reconciler's surface (platform/keycloak/client-secret-reconciler.yaml),
+#     and users/groups/roles are nobody's — prod's real user and the media-users /
+#     platform-admins / viewers groups must survive every run of everything.
+#   * execution `priority`. It is consumed here to ORDER the exported executions
+#     (Keycloak returns them in priority order), but the numbers themselves are not
+#     exported: the reconciler compares the resulting SEQUENCE, which is what
+#     actually determines behaviour. Exporting the integers would make a
+#     priority renumbering that changes no ordering read as drift.
+#
+# USAGE
+#   python3 ci/render-keycloak-passkey-policy.py            # print the rendered file
+#   python3 ci/render-keycloak-passkey-policy.py --write     # write it (regenerate)
+#   python3 ci/render-keycloak-passkey-policy.py --check     # exit 1 if it has drifted
+# =============================================================================
+from __future__ import annotations
+
+import argparse
+import difflib
+import json
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:  # pragma: no cover
+    print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(__file__).resolve().parent.parent
+IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
+OUT_FILE = "platform/keycloak/passkey-desired-state.yaml"
+DATA_KEY = "passkey-desired-state.json"
+CONFIGMAP_NAME = "keycloak-passkey-desired-state"
+NAMESPACE = "keycloak"
+
+# The realm keys this renderer exports, and therefore the ONLY realm keys the
+# reconciler may write. Every one is a webAuthnPolicyPasswordless* field. Kept in
+# lockstep with POLICY_FIELDS in the reconciler script; the L0 check asserts the two
+# tuples are identical, so they cannot drift apart silently.
+POLICY_STR_FIELDS = (
+    "webAuthnPolicyPasswordlessRpId",
+    "webAuthnPolicyPasswordlessRpEntityName",
+    "webAuthnPolicyPasswordlessRequireResidentKey",
+    "webAuthnPolicyPasswordlessUserVerificationRequirement",
+    "webAuthnPolicyPasswordlessAuthenticatorAttachment",
+    "webAuthnPolicyPasswordlessAttestationConveyancePreference",
+)
+POLICY_LIST_FIELDS = (
+    "webAuthnPolicyPasswordlessSignatureAlgorithms",
+    "webAuthnPolicyPasswordlessAcceptableAaguids",
+)
+POLICY_INT_FIELDS = ("webAuthnPolicyPasswordlessCreateTimeout",)
+POLICY_BOOL_FIELDS = ("webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister",)
+POLICY_FIELDS = (POLICY_STR_FIELDS + POLICY_LIST_FIELDS + POLICY_INT_FIELDS
+                 + POLICY_BOOL_FIELDS)
+
+# The field Keycloak 26.0.6 gates the WHOLE passwordless policy on
+# (DefaultExportImportManager.updateRealm: the policy is applied only when
+# RpEntityName is non-null and non-empty). A desired state without it would produce
+# an accepted PUT that changes nothing, so refuse to render one.
+RP_ENTITY_NAME = "webAuthnPolicyPasswordlessRpEntityName"
+
+# The execution whose presence is what makes passkey enrolment necessary.
+PASSKEY_AUTHENTICATOR = "webauthn-authenticator-passwordless"
+ENROL_ACTION = "webauthn-register-passwordless"
+
+
+BIND_KEY = "bind-browser-flow"
+BIND_VALUES = ("true", "false")
+
+
+def committed_bind(root: Path) -> str:
+    """The operator's bind-browser-flow value from the COMMITTED file, or "false".
+
+    This is the only input read from the output file, and it is validated, not trusted:
+    anything other than the exact strings "true" / "false" is a hard failure."""
+    target = root / OUT_FILE
+    if not target.is_file():
+        return "false"
+    docs = [d for d in yaml.safe_load_all(target.read_text(encoding="utf-8")) if d]
+    cm = next((d for d in docs if d.get("kind") == "ConfigMap"), None) or {}
+    data = cm.get("data") or {}
+    if BIND_KEY not in data:
+        return "false"
+    v = data[BIND_KEY]
+    if not isinstance(v, str) or v not in BIND_VALUES:
+        raise SystemExit(
+            "FATAL: %s data.%s is %r. It must be exactly the QUOTED string \"true\" or "
+            "\"false\": an unquoted true/false is a YAML boolean, which a ConfigMap rejects "
+            "at apply time and which would fail the whole platform-config Kustomization"
+            % (OUT_FILE, BIND_KEY, v))
+    return v
+
+
+def render(root: Path) -> str:
+    """Return the full text of passkey-desired-state.yaml as it should be committed."""
+    docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
+    imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
+    if imp is None:
+        raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
+    realm = (imp.get("spec") or {}).get("realm") or {}
+    realm_name = realm.get("realm")
+    if not isinstance(realm_name, str) or not realm_name:
+        raise SystemExit("FATAL: %s declares no spec.realm.realm name" % IMPORT_FILE)
+
+    # ---- the passwordless policy, copied VERBATIM ----------------------------------
+    policy: dict[str, object] = {}
+    for f in POLICY_STR_FIELDS:
+        if f not in realm:
+            continue                      # not declared => NOT MANAGED
+        v = realm[f]
+        # A bare ${BASE_DOMAIN} parses as a plain string; an int or bool here would be
+        # a YAML quoting mistake that must not be silently stringified.
+        if not isinstance(v, str):
+            raise SystemExit("FATAL: %s realm field %s is not a string (got %r) — "
+                             "quote it in the realm import" % (IMPORT_FILE, f, v))
+        policy[f] = v
+    for f in POLICY_LIST_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
+            raise SystemExit("FATAL: %s realm field %s is not a list of strings"
+                             % (IMPORT_FILE, f))
+        policy[f] = list(v)               # verbatim, order as authored
+    for f in POLICY_INT_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, int) or isinstance(v, bool):
+            raise SystemExit("FATAL: %s realm field %s is not an integer"
+                             % (IMPORT_FILE, f))
+        policy[f] = v
+    for f in POLICY_BOOL_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, bool):
+            raise SystemExit("FATAL: %s realm field %s is not a boolean"
+                             % (IMPORT_FILE, f))
+        policy[f] = v
+
+    if not policy:
+        # Fail closed. An empty policy would make the reconciler believe every field is
+        # unmanaged and silently stop reconciling — the realm import's own fail-open.
+        raise SystemExit("FATAL: %s declares no webAuthnPolicyPasswordless* field — "
+                         "refusing to render an empty desired state" % IMPORT_FILE)
+    if not policy.get(RP_ENTITY_NAME):
+        raise SystemExit(
+            "FATAL: %s does not declare a non-empty %s. Keycloak 26.0.6 applies the "
+            "passwordless policy ONLY when that field is set, so every realm PUT would "
+            "be accepted and change nothing" % (IMPORT_FILE, RP_ENTITY_NAME))
+
+    # ---- the flow tree, flattened per flow and ORDERED BY PRIORITY -----------------
+    browser_flow = realm.get("browserFlow")
+    if not isinstance(browser_flow, str) or not browser_flow:
+        raise SystemExit("FATAL: %s declares no spec.realm.browserFlow — the flow could "
+                         "be perfect and still not be the one anyone logs in through"
+                         % IMPORT_FILE)
+    raw_flows = realm.get("authenticationFlows") or []
+    if not isinstance(raw_flows, list) or not raw_flows:
+        raise SystemExit("FATAL: %s declares no authenticationFlows" % IMPORT_FILE)
+
+    flows: dict[str, object] = {}
+    for fl in raw_flows:
+        alias = fl.get("alias")
+        if not isinstance(alias, str) or not alias:
+            raise SystemExit("FATAL: %s has an authentication flow with no alias"
+                             % IMPORT_FILE)
+        if alias in flows:
+            raise SystemExit("FATAL: %s declares the flow %r twice"
+                             % (IMPORT_FILE, alias))
+        execs = fl.get("authenticationExecutions") or []
+        if not isinstance(execs, list) or not execs:
+            raise SystemExit("FATAL: %s flow %r declares no authenticationExecutions"
+                             % (IMPORT_FILE, alias))
+        ordered = []
+        for ex in sorted(execs, key=lambda e: (e.get("priority") or 0)):
+            is_flow = bool(ex.get("authenticatorFlow"))
+            name = ex.get("flowAlias") if is_flow else ex.get("authenticator")
+            if not isinstance(name, str) or not name:
+                raise SystemExit(
+                    "FATAL: %s flow %r has an execution with neither an authenticator "
+                    "nor a flowAlias" % (IMPORT_FILE, alias))
+            requirement = ex.get("requirement")
+            if not isinstance(requirement, str) or not requirement:
+                raise SystemExit("FATAL: %s flow %r execution %r declares no requirement"
+                                 % (IMPORT_FILE, alias, name))
+            entry: dict[str, object] = {
+                "kind": "flow" if is_flow else "authenticator",
+                "name": name,
+                "requirement": requirement,
+            }
+            cfg = ex.get("authenticatorConfig")
+            if cfg is not None:
+                if not isinstance(cfg, str) or not cfg:
+                    raise SystemExit("FATAL: %s flow %r execution %r has a non-string "
+                                     "authenticatorConfig" % (IMPORT_FILE, alias, name))
+                entry["configAlias"] = cfg
+            ordered.append(entry)
+        flows[alias] = {"topLevel": bool(fl.get("topLevel")), "executions": ordered}
+
+    if browser_flow not in flows:
+        raise SystemExit("FATAL: %s binds browserFlow to %r but declares no such flow"
+                         % (IMPORT_FILE, browser_flow))
+    # Every referenced subflow must be declared, or the reconciler's flattening walks
+    # into a hole and the audit silently covers less than it claims.
+    for alias, spec in flows.items():
+        for ex in spec["executions"]:
+            if ex["kind"] == "flow" and ex["name"] not in flows:
+                raise SystemExit("FATAL: %s flow %r references subflow %r, which is not "
+                                 "declared" % (IMPORT_FILE, alias, ex["name"]))
+
+    # ---- authenticator configs, copied verbatim ------------------------------------
+    configs: dict[str, dict] = {}
+    for cfg in realm.get("authenticatorConfig") or []:
+        alias = cfg.get("alias")
+        if not isinstance(alias, str) or not alias:
+            raise SystemExit("FATAL: %s has an authenticatorConfig with no alias"
+                             % IMPORT_FILE)
+        body = cfg.get("config") or {}
+        if not isinstance(body, dict) or not body:
+            raise SystemExit("FATAL: %s authenticatorConfig %r has no config body"
+                             % (IMPORT_FILE, alias))
+        # Values are stringified because Keycloak stores authenticator config values as
+        # strings. `negate: "false"` in the import and a stored "false" are the same
+        # declaration, and a YAML-unquoted `false` must not become a Python bool that
+        # then never matches.
+        configs[alias] = {k: str(v) for k, v in sorted(body.items())}
+
+    # Every configAlias a flow references must be defined, else the reconciler reports
+    # "git declares config X but defines no key/value for it" on every run forever.
+    for alias, spec in flows.items():
+        for ex in spec["executions"]:
+            ca = ex.get("configAlias")
+            if ca and ca not in configs:
+                raise SystemExit(
+                    "FATAL: %s flow %r execution %r references authenticatorConfig %r, "
+                    "which is not declared" % (IMPORT_FILE, alias, ex["name"], ca))
+
+    # ---- the enrolment required action, DERIVED (see the header) -------------------
+    offers_passkey = any(
+        ex["kind"] == "authenticator" and ex["name"] == PASSKEY_AUTHENTICATOR
+        for spec in flows.values() for ex in spec["executions"])
+    if not offers_passkey:
+        raise SystemExit(
+            "FATAL: %s declares a webAuthnPolicyPasswordless* policy but no %r execution "
+            "in any flow. The policy decides what a passkey must BE; without that "
+            "execution nothing ever OFFERS one, so the policy is unreachable "
+            "configuration and this desired state would be meaningless"
+            % (IMPORT_FILE, PASSKEY_AUTHENTICATOR))
+
+    want = {
+        "realm": realm_name,
+        "webAuthnPolicyPasswordless": policy,
+        # enabled: passkey enrolment must be possible, since the flow offers passkeys.
+        # defaultAction FALSE: enrolment stays OPT-IN. True would force every user to
+        # enrol a passkey at next login — the platform-wide lock-out the realm import's
+        # "additive and optional" posture exists to avoid. Derived, not copied; see the
+        # header for why the import cannot declare this itself.
+        "requiredAction": {"alias": ENROL_ACTION, "enabled": True,
+                           "defaultAction": False},
+        "browserFlow": browser_flow,
+        "flows": flows,
+        "authenticatorConfig": configs,
+    }
+
+    body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
+    indented = "\n".join(("    " + ln) if ln else "" for ln in body.splitlines())
+    return (HEADER + BIND_BLOCK % (BIND_KEY, committed_bind(root))
+            + "  %s: |\n" % DATA_KEY + indented + "\n")
+
+
+HEADER = """---
+# =============================================================================
+# GENERATED FILE — DO NOT EDIT BY HAND, except the one bind-browser-flow line below.
+#
+#   source:      platform/keycloak/keycloak-realm-import.yaml (spec.realm)
+#   regenerate:  python3 ci/render-keycloak-passkey-policy.py --write
+#   gated by:    conformance/tests/l0-static/keycloak-passkey-reconciler
+#                (runs this renderer with --check on every merge)
+#
+# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
+# the WebAuthn passwordless policy, the enrolment required action, and the declared
+# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
+# it and (a) converges the policy and the required action onto it, (b) builds the
+# declared flow tree BLUE/GREEN under a new alias and verifies it by readback, never
+# writing the flow users are logging in through, and (c) BINDS that verified flow as
+# the realm's browserFlow ONLY when `bind-browser-flow` below is "true".
+#
+# EVERYTHING HERE IS GENERATED EXCEPT ONE LINE: `bind-browser-flow`. That is an
+# operator decision, carried through by the renderer from this committed file (see
+# ci/render-keycloak-passkey-policy.py). It defaults to "false".
+#
+# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
+# once Done=True, so a realm-level field or a flow edited in git never reaches the
+# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
+# Done=True / Started=False, with the whole passkey block inert. The reconciler is
+# the level-triggered loop that closes that; this file is what it reconciles
+# TOWARDS.
+#
+# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
+# user-verification requirement and the flow tree would be a second source of truth,
+# free to disagree with the realm import — the same "two declarations that can
+# disagree" shape as the bug being closed. It is rendered from the import and the
+# render is gated, so the two cannot drift.
+#
+# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
+# substitution token exactly as the realm import writes it. Both files are rendered
+# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
+# same value in both and the domain is decided in exactly one place.
+#
+# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
+# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
+# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
+# nine more (a hard lock-out of every platform-admin). It is derived instead from
+# the presence of a webauthn-authenticator-passwordless execution in the flow:
+# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
+# false, so enrolment stays opt-in.
+#
+# A field NOT declared on the realm in the import is simply ABSENT here, and the
+# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
+# =============================================================================
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: %s
+  namespace: %s
+  labels:
+    app.kubernetes.io/name: %s
+    app.kubernetes.io/part-of: keycloak
+data:
+""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME)
+
+BIND_BLOCK = """  # ---------------------------------------------------------------------------
+  # THE BIND SWITCH -- the ONLY hand-edited line in this file. Exactly "true" or
+  # "false", QUOTED.
+  #
+  #   "false"  the reconciler builds and verifies the candidate browser flow and
+  #            logs `WOULD BIND <alias>`, but NEVER changes the realm's browserFlow.
+  #            Whatever the realm is bound to -- including a rollback -- is left alone.
+  #   "true"   it binds the verified candidate in one atomic PUT, and keeps the
+  #            realm bound to it: a binding changed by hand is changed back.
+  #
+  # GO LIVE: test the candidate in a browser first (platform/keycloak/README.md,
+  # "Passkeys"), then a PR setting this to "true".
+  # ROLL BACK: a PR setting this to "false" FIRST (or the next run rebinds), then
+  # one write setting browserFlow back to the previous alias from the Job log.
+  # ---------------------------------------------------------------------------
+  %s: "%s"
+"""
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    g = ap.add_mutually_exclusive_group()
+    g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
+    g.add_argument("--check", action="store_true",
+                   help="exit 1 if the committed file has drifted from the source")
+    args = ap.parse_args()
+
+    want = render(ROOT)
+    target = ROOT / OUT_FILE
+
+    if args.write:
+        target.write_text(want, encoding="utf-8")
+        print("wrote %s" % OUT_FILE)
+        return 0
+
+    if args.check:
+        if not target.is_file():
+            print("CHECK FAIL: %s does not exist. Run: python3 "
+                  "ci/render-keycloak-passkey-policy.py --write" % OUT_FILE,
+                  file=sys.stderr)
+            return 1
+        have = target.read_text(encoding="utf-8")
+        if have == want:
+            print("passkey-desired-state.yaml is in sync with %s" % IMPORT_FILE)
+            return 0
+        print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
+              "realm on a passkey policy the realm import does not declare, and build "
+              "the login flow from a tree nobody authored.\n"
+              "Regenerate with: python3 ci/render-keycloak-passkey-policy.py --write\n"
+              % (OUT_FILE, IMPORT_FILE), file=sys.stderr)
+        for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
+                                         fromfile="committed " + OUT_FILE,
+                                         tofile="rendered from " + IMPORT_FILE):
+            sys.stderr.write(line)
+        return 1
+
+    sys.stdout.write(want)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

clusters/base/buildkit.yaml

@@ -0,0 +1,21 @@
+---
+# Flux Kustomization -> ./platform/buildkit — rootless BuildKit, the image builder CI uses
+# (.gitea/workflows/images.yaml). Its own Kustomization so a stuck Certificate or a failing
+# builder can never stall any other tier (Flux health-checks Certificates under wait:true).
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: buildkit
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: true
+  dependsOn:
+    - name: platform   # cert-manager CRDs (Issuer/Certificate) come from its HelmRelease
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/buildkit

clusters/base/cluster-vars.yaml

@@ -101,7 +101,8 @@ data:
   # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
   # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
   # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
-  # plex-remote: PLEX_L4_VIP; plex: PLEX_ADVERTISE_URL),
+  # plex-remote + pihole: PLEX_L4_VIP),
+  # vpn-ingress (WG_CLIENT_CIDR),
   # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
   # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
   # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -189,32 +190,13 @@ data:
   # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
   # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
   PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
+  # SECOND CONSUMER since 2026-10-04: catalog/pihole.yaml reads this knob to answer
+  # plex-vip.${BASE_DOMAIN} on the LAN, which is the local half of the single hostname Plex
+  # advertises to plex.tv. Without that per-host answer the zone wildcard sends the name to
+  # ${GATEWAY_VIP}, where nothing listens on 32400. So this value is now typed into THREE places
+  # that must agree: this knob, the WAN port-forward on the SOHO router (off-repo), and PiHole's
+  # answer -- and only the first two were true before.
 
-  # PLEX_ADVERTISE_URL — the connection URL Plex publishes to plex.tv ("Custom server access URLs",
-  # the ADVERTISE_IP env in catalog/plex.yaml, which the image writes into the `customConnections`
-  # preference on every pod start). Read catalog/plex.yaml's ADVERTISE_IP block for the full why;
-  # the short version is the only thing that matters when editing this line:
-  #
-  #   IT MUST BE A `<dashed-ip>.<serverhash>.plex.direct` NAME, NEVER A BARE IP. Plex serves 32400
-  #   with a `*.<serverhash>.plex.direct` wildcard certificate and NO IP SAN, so `https://<ip>:32400`
-  #   fails TLS validation for every client. That exact mistake is what broke this service on
-  #   2026-10-04: with no completable connection, plex.tv published nothing, the server vanished from
-  #   the account and clients reported "not authorized". The dashed name is public DNS that resolves
-  #   to the embedded address, so it points at the LAN VIP AND matches the cert.
-  #
-  # WHY IT IS A WHOLE URL AND NOT BUILT FROM ${PLEX_L4_VIP}: postBuild substitution is plain
-  # envsubst — it cannot turn 192.168.2.24 into 192-168-2-24 — and `<serverhash>` is this server's
-  # certificate identity, derivable from nothing in this tree. Re-read it from the live cert
-  # (`openssl s_client -connect <vip>:32400 | openssl x509 -noout -subject`) if Plex ever reissues.
-  #
-  # The dashed IP here MUST track ${PLEX_L4_VIP} above. They are two halves of one address and
-  # nothing checks that they agree.
-  #
-  # dev  -> PLACEHOLDER, non-functional. dev is decommissioned and has no claimed Plex server, so
-  #         there is no real hash to put here; the base default only has to be a defined, correctly
-  #         shaped string (StrictPostBuildSubstitutions makes an UNDEFINED var a hard failure).
-  # prod -> the real 192.168.2.24 + live server hash; see clusters/prod/kustomization.yaml.
-  PLEX_ADVERTISE_URL: "https://192-168-1-57.0000000000000000000000000000000f.plex.direct:32400"
 
   # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
   # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
@@ -226,27 +208,54 @@ data:
   # prod -> 192.168.2.0/24  (prod LAN; nodes .10/.11/.12)
   LAN_CIDR: "192.168.1.0/24"                 # ingress scope for the PiHole DNS listener
 
+  # --- Remote-access VPN client subnet (platform/vpn-ingress, WireGuard) --------------------------
+  # The tunnel subnet handed to REMOTE VPN clients. `.1` is the server (the hostNetwork DaemonSet,
+  # on every node); `.2`-`.254` are devices, allocated one /32 each in
+  # platform/vpn-ingress/peers.yaml. Consumed twice in the pod's config-assembly script: to derive
+  # the server's own address, and as the `-s` match of the MASQUERADE rule that makes LAN replies
+  # routable.
+  #
+  # IT IS A KNOB RATHER THAN A LITERAL FOR ONE REASON: it is the only address range in this cluster
+  # that must be checked against EVERY OTHER range at once, so it belongs where those are declared.
+  # 10.90.0.0/24 was chosen to collide with none of them:
+  #     10.0.0.0/16     Cilium cluster-pool pod CIDR     (${CLUSTER_POD_CIDR})
+  #     10.43.0.0/16    k3s service CIDR                 (measured live 2026-10-05: kubernetes svc
+  #                                                       clusterIP 10.43.0.1)
+  #     10.72.192.0/18  the PIA tunnel's pushed addresses (platform/vpn-egress)
+  #     192.168.1.0/24  dev LAN / pfSense LAN            (${LAN_CIDR} base default)
+  #     192.168.2.0/24  prod LAN                         (${LAN_CIDR} on prod)
+  #
+  # NO PER-ENVIRONMENT OVERRIDE, and that is deliberate rather than an omission: unlike every VIP
+  # above, this range is NOT drawn from the LAN. It is private to the tunnel, so the same value is
+  # correct in both environments and a prod patch would be a no-op that implies otherwise.
+  #
+  # IF YOU CHANGE IT, every already-issued client profile breaks (its `Address` is from this range)
+  # and must be re-rendered — so treat it as fixed once the first device is enrolled.
+  WG_CLIENT_CIDR: "10.90.0.0/24"             # remote VPN clients; .1 = server. NOT a LAN range
+
   # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
-  # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
+  # THE address of the NAS whose ZFS datasets back every `class: bulk` volume: the media library
-  # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
+  # (Storage/plex, 8.33 T — plex, sonarr, radarr, qbittorrent) and the legacy Nextcloud archive
   # (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
   # `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
   # declared in the ServiceDefinition, because the path is identical in every environment and only
   # the ADDRESS is not.
   #
-  # WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
+  # IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read "the two clusters are on
-  # The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
+  # OPPOSITE SIDES of pfSense": the datasets lived on a Proxmox host whose only interface was
-  #   dev  nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
+  # vmbr0 192.168.1.9/24 on pfSense's LAN, which prod (192.168.2.0/24, pfSense's WAN) could reach
-  #        NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
+  # only through a DNAT at 192.168.2.102. Both pools were physically moved onto a standalone
-  #        /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
+  # TrueNAS at 192.168.2.30 — prod's OWN /24 — so there is no crossing to arrange and no candidate
-  #        per the convention of the LAN block above.
+  # values to choose between. Measured 2026-10-07: 192.168.2.102:2049 no longer answers;
-  #   prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
+  # 192.168.2.30:2049 and :111 do. There is now exactly ONE storage host and one correct value;
-  #        22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
+  # this stays a knob only because the renderer reads it per-environment.
-  #        the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
+  #
-  #        DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
+  # *** CHANGING THIS VALUE IS NOT A PLAIN RECONCILE. *** A bound PersistentVolume's spec.nfs is
-  # So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
+  # immutable — measured by server dry-run 2026-10-07:
-  # blast radius of shipping it unset, which is NOT the usual "fails to apply".
+  #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
-  BULK_NFS_SERVER: "192.168.1.9"
+  # so Flux's apply FAILS rather than updating, and the pods keep their old mount. The PVs must be
+  # deleted and recreated; the operator sequence is docs/BULK-NFS-STORAGE.md §10.
+  BULK_NFS_SERVER: "192.168.2.30"
 
   # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
   # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,

clusters/base/kured.yaml

@@ -0,0 +1,98 @@
+---
+# Flux Kustomization: kured (Kubernetes Reboot Daemon) — reconciles ./platform/kured.
+# Performs the cordon / drain / reboot / uncordon of a node that has a pending OS reboot, strictly
+# one node at a time, inside a nightly window, gated on Longhorn volume health. Design rationale,
+# every tuned value, and the honest list of what is NOT mitigated: platform/kured/helmrelease.yaml.
+#
+# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
+#
+# The same argument the Cilium split-out recorded in platform/kustomization.yaml and intel-gpu-plugin
+# reused, except the consequence is worse here than for either of them:
+#
+#   * `platform` is `wait: true` over twelve-plus components, so its Ready condition cannot tell you
+#     "the reboot daemon is wedged" from "longhorn is slow".
+#   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
+#     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a DaemonSet.
+#   * That tier's 10m interval / 5m timeout are called out there as WRONG for a 3-node DaemonSet
+#     rollout. This is a 3-node DaemonSet rollout.
+#   * And `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates
+#     apps-wave-1. A reboot daemon that failed to go healthy inside that tier would stall EVERY
+#     APPLICATION in the cluster on "dependency not ready". A component added to make maintenance
+#     safer must not acquire the ability to cause an outage; that trade is the whole argument.
+#
+# dependsOn: [platform] — AND THIS ONE IS A REAL EDGE, NOT CONVENTION
+#
+# Not `cilium` (what intel-gpu-plugin uses) and not `platform-config` (what the leaf UIs use). It is
+# `platform`, for two independent reasons:
+#
+#   1. CRD ORDERING (F-MON1). platform/kured ships a PrometheusRule, and the chart generates a
+#      ServiceMonitor. Both are CRs whose CRDs (monitoring.coreos.com) are installed by
+#      kube-prometheus-stack INSIDE the `platform` tier. Flux dry-runs every object in a pass before
+#      applying any of them, so a pass containing an unknown kind ABORTS — the failure that made the
+#      CDI StorageProfiles a true deadlock on prod on 2026-09-05 (see
+#      clusters/base/platform-config-cdi.yaml). Depending on `platform` guarantees the CRDs exist
+#      before this pass is attempted.
+#   2. FUNCTIONAL PREREQUISITE. kured's reboot gate QUERIES Prometheus
+#      (`prometheusUrl` in the HelmRelease). Prometheus lives in platform/monitoring. If kured ran
+#      first it would get connection errors, treat the gate as unevaluable and skip every reboot —
+#      the correct fail-safe direction, but silent. Longhorn, whose volume health IS the gate, is
+#      also in that tier.
+#
+# It does NOT need platform-config: no Keycloak, no TLS, no split-horizon DNS, no Gateway, no
+# HTTPRoute. kured serves no HTTP to users.
+#
+# wait: true — A HEALTH SIGNAL, NOT A GATE. Following intel-gpu-plugin rather than the leaf-UI
+# template. A leaf UI is legitimately not-Ready for long stretches of a cold start while it waits on
+# Keycloak and DNS, so its health is uninformative; kured has no such dependency — it is a DaemonSet
+# that opens a socket and watches a file, so on healthy nodes it is Ready in seconds and not-Ready
+# genuinely means BROKEN. This turns `flux get kustomization kured` into a one-line answer to "is
+# reboot automation live on all three nodes", which is the question the operator must be able to
+# answer before trusting a kernel upgrade to it.
+#
+# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
+# kured is this Kustomization's own Ready condition. Do not add `dependsOn: [kured]` anywhere: that
+# would hand a node-rebooting DaemonSet the power to stall whatever depends on it, which is precisely
+# what the placement above exists to prevent.
+#
+# prune: true is safe and deliberate. If this Kustomization is deleted, kured's DaemonSet goes with
+# it and reboots simply stop happening — nodes are left uncordoned and running. The one caveat is
+# that pruning mid-reboot would leave a node CORDONED with no daemon to uncordon it; that is a
+# `kubectl uncordon` away and is covered in runbooks/node-reboot-automation.md.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: kured
+  namespace: flux-system
+spec:
+  # A HelmRelease plus five static manifests, nothing to re-render; kustomize-controller is
+  # source-triggered, so this is only the drift-correction floor.
+  interval: 10m
+  retryInterval: 2m
+  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
+  # exceed 5m on a cold start (image pull on three nodes), which is one of the reasons the Cilium
+  # split-out gave for leaving that tier.
+  timeout: 10m
+  prune: true
+  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
+  wait: true
+  dependsOn:
+    - name: platform   # monitoring CRDs + Prometheus + Longhorn; see the header, this is load-bearing
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/kured
+  # REQUIRED, not boilerplate. platform/kured/alerts.yaml renders `${BASE_DOMAIN}` into the
+  # `runbook_url` annotation of all three rules. Without this block that token becomes the EMPTY
+  # STRING and the Kustomization still reports Ready — Flux's substitution fails OPEN — so every
+  # alert would ship a dead link to `https:///runbooks/node-reboot-automation`. The gate stage
+  # `flux-postbuild-substitution` asserts the converse (a rendered token implies this block), and
+  # BASE_DOMAIN is declared in clusters/base/cluster-vars.yaml.
+  #
+  # Nothing else in this path contains `${...}` or a bare `$`: the HelmRelease's alertFilterRegexp is
+  # a plain anchored alternation, and there are no shell snippets anywhere in the component — which
+  # is deliberate, because envsubst would eat a braced shell variable and `$$` would trip the
+  # dollar-collapse assertion in the same check.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars

clusters/base/kustomization.yaml

@@ -71,4 +71,32 @@ resources:
   # controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
   # Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
   - backup.yaml          # Flux Kustomization -> ./platform/velero (dependsOn platform-config)
+  # WireGuard remote-access VPN (north-south INGRESS, udp/51820) — POST-platform tier
+  # (dependsOn platform-config, wait:false). NOT the same thing as platform/vpn-egress, which sends
+  # pods OUT through PIA; this lets remote CLIENTS IN. Non-gating is load-bearing rather than
+  # conventional here: its DaemonSet cannot start until the operator creates the out-of-band
+  # `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have
+  # stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch.
+  - vpn-ingress.yaml     # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config)
+  # PM agent — reads Gitea issues and runs one worker agent per assignment that authors a
+  # change, gates it and opens a pull request. POST-APP tier (dependsOn apps-wave-0,
+  # wait:false): what it needs is GITEA, not Keycloak or the Gateway, and it must never gate
+  # anything — it CrashLoops by design while its credentials are still placeholders. Carries
+  # pm-agent-credentials.sops.yaml, so both overlays add a SOPS decryption patch for it (by
+  # name). See clusters/base/pm-agent.yaml and platform/pm-agent/README.md.
+  - pm-agent.yaml        # Flux Kustomization -> ./platform/pm-agent (dependsOn apps-wave-0)
+  # kured (Kubernetes Reboot Daemon) — its OWN Kustomization (dependsOn platform, wait:true), NOT a
+  # line in platform/kustomization.yaml. Same reasoning as the Cilium and intel-gpu-plugin
+  # split-outs, with a sharper consequence: `platform` gates platform-config and every apps wave, so
+  # a wedged reboot DaemonSet placed in that wait:true tier could stall every application in the
+  # cluster — a component added to make maintenance SAFER must not be able to cause an outage.
+  # wait:true here is a HEALTH SIGNAL, not a gate: nothing dependsOn it, and nothing should.
+  # The `platform` edge is load-bearing rather than conventional — it supplies the
+  # monitoring.coreos.com CRDs this component's PrometheusRule/ServiceMonitor need (F-MON1), and
+  # Prometheus itself, which kured QUERIES before every reboot to gate on Longhorn volume health.
+  # It reboots a node only when /var/run/reboot-required exists, which on these nodes only a
+  # deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
+  # See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
+  - kured.yaml           # Flux Kustomization -> ./platform/kured (dependsOn platform)
+  - buildkit.yaml        # Flux Kustomization -> ./platform/buildkit (dependsOn platform)
   - cluster-vars.yaml    # overlay knob surface (base defaults)

clusters/base/pm-agent.yaml

@@ -0,0 +1,57 @@
+---
+# Flux Kustomization: the PM agent service — reconciles ./platform/pm-agent.
+#
+# POST-APP TIER (dependsOn apps-wave-0). What this service actually depends on is GITEA: the
+# dispatcher authenticates to it at startup, reads issue threads from it, and every worker
+# clones and pushes to it. Gitea is in apps-wave-0, so that is the dependency — not
+# platform-config, which brings up the Keycloak realm, the wildcard certificate, the Gateway
+# and the L2 pool. This service has no UI, serves no traffic, takes no ingress and holds no
+# certificate, so none of those are prerequisites and claiming them would be a lie that
+# delays its reconcile behind four things it does not use.
+#
+# NON-GATING (wait:false), and here that is not merely prudent — the pod is EXPECTED to be
+# unhealthy for a while:
+#   * the committed credentials are PLACEHOLDERS, so until the operator replaces them the
+#     dispatcher fails its Gitea authentication at startup and CrashLoops, by design
+#     (platform/pm-agent/pm-agent-credentials.sops.yaml.example);
+#   * the image digest is likewise a placeholder until the operator pushes the image, so
+#     before that the pod cannot even pull.
+# Either of those inside a wait:true tier would stall everything downstream of it. wait:false
+# guarantees this service's health can NEVER gate anything, even if some future resource is
+# mistakenly pointed dependsOn it. Same non-gating leaf rationale as
+# headlamp/homepage/crafty/backup — with a stronger reason than any of them.
+#
+# PRUNE: true. The whole namespace is disposable — the dispatcher keeps no durable state (the
+# Gitea issue thread is its state) and worker Jobs are ephemeral by construction. So removing
+# this service from git really should remove it from the cluster, with nothing to orphan.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: pm-agent
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: apps-wave-0   # Gitea: the repository this service reads issues from and pushes to
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/pm-agent
+  # cluster-vars is substituted for parity with every other tier. NOTE for anyone editing
+  # platform/pm-agent/: envsubst runs over the WHOLE rendered manifest, comments included, so
+  # a `${...}` meant for a shell would be replaced with the empty string before the container
+  # starts (the hazard catalog/README.md documents at length — it silently killed gitea's and
+  # nextcloud's OIDC registration for days). Nothing under platform/pm-agent/ uses a `${}`
+  # token today, and the two scripts that DO contain shell live in the image, not in a
+  # manifest, which is one more reason they are baked in rather than mounted from a ConfigMap.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # SOPS decryption for pm-agent-credentials.sops.yaml is layered per-environment by the
+  # overlay (clusters/prod and clusters/dev patch THIS Kustomization by name), identical to
+  # crafty/headlamp/oauth2-proxy/backup.

clusters/base/vpn-ingress.yaml

@@ -0,0 +1,59 @@
+---
+# Flux Kustomization: WireGuard remote-access VPN — reconciles ./platform/vpn-ingress.
+#
+# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false). Not a line in
+# platform/kustomization.yaml, for the standard reason plus one specific to this component:
+#
+#   * STANDARD: a leaf must never sit in the gating `platform` tier. While `platform` is not-Ready
+#     NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves
+#     (DECISIONS-LOG: "Headlamp gating stall").
+#   * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the
+#     operator replaces the `REPLACE-ME` placeholder in platform/vpn-ingress/wg-server-key.sops.yaml
+#     with a real key (runbooks/vpn-remote-access.md step 2). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
+#     the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the
+#     cause (a missing manual step) invisible from any Flux condition.
+#
+# WHY dependsOn platform-config AND NOT cilium. This component needs no Gateway, no TLS cert, no
+# Keycloak and no LB-IPAM pool — it has no HTTPRoute and no LoadBalancer Service, and hostNetwork
+# means it does not even need a Cilium endpoint. It could legally reconcile much earlier. It is
+# pinned behind platform-config anyway so that the ONE thing remote clients actually consume
+# through it — split-horizon DNS and the shared Gateway — exists by the time the tunnel does.
+# A tunnel that comes up before the things on the far side of it is a support call, not a feature.
+#
+# wait:false: nothing in the graph dependsOn this, and its pods are legitimately not-Ready whenever
+# the server key is absent or the `wireguard` kernel module has not been loaded on a node. Neither
+# condition should be able to gate anything else, even if a future resource is mistakenly pointed
+# at it.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: vpn-ingress
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/vpn-ingress
+  # REQUIRED, not boilerplate: platform/vpn-ingress/daemonset.yaml renders ${WG_CLIENT_CIDR} into
+  # the config-assembly script. Without this block that token becomes the EMPTY STRING and the
+  # Kustomization still reports Ready (Flux's substitution fails OPEN — ci:
+  # flux-postbuild-substitution), leaving an initContainer that derives a server address of ".1/"
+  # and a MASQUERADE rule with no source. Both fail loudly at pod start, but the Kustomization
+  # would look healthy.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # SOPS decryption is layered per-environment by the overlay (clusters/dev and clusters/prod each
+  # patch THIS Kustomization by name), identical to headlamp/crafty/velero — this path carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml, the WireGuard server private key. The `sops-age`
+  # Secret is created out of band and referenced by NAME only. Without the overlay patch the Secret
+  # applies cleanly full of ENC[...] and only the tunnel is broken, so the patch is as load-bearing
+  # as the encrypted file itself.

clusters/dev/kustomization.yaml

@@ -122,6 +122,43 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
+  # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
+  # account and the Anthropic subscription token from `claude setup-token`. Without this
+  # stanza kustomize hands Flux the raw SOPS document and the Secret applies full of
+  # ciphertext: the APPLY SUCCEEDS, and the only symptom is a dispatcher that cannot
+  # authenticate to Gitea — so this stanza is as load-bearing as the file it decrypts (same
+  # silent-failure shape as the apps-vms note below/above). The `sops-age` Secret is created
+  # out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: pm-agent
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
   # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
   # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
@@ -242,15 +279,15 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.1.55"
-      # Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
+      # Bulk (external NFS) storage host. THIS VALUE IS INERT — dev is decommissioned, its nodes
-      # (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
+      # (192.168.1.40/.41/.42) no longer exist, and the Proxmox host this used to name (192.168.1.9)
-      # Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
+      # went with the pools it held. It is restated per this overlay's "supply every knob"
-      # Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
+      # convention, not because anything consumes it. It carries the real address so that no file
-      # end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
+      # in the tree renders a host that is gone; it has NOT been proven from a dev node, because
-      # change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
+      # there is no dev node to prove it from.
       - op: replace
         path: /data/BULK_NFS_SERVER
-        value: "192.168.1.9"
+        value: "192.168.2.30"
       # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
       # barmanObjectStore and Velero's backupStorageLocation write to.
       #

clusters/prod/kustomization.yaml

@@ -113,6 +113,43 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
+  # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
+  # account and the Anthropic subscription token from `claude setup-token`. Without this
+  # stanza kustomize hands Flux the raw SOPS document and the Secret applies full of
+  # ciphertext: the APPLY SUCCEEDS, and the only symptom is a dispatcher that cannot
+  # authenticate to Gitea — so this stanza is as load-bearing as the file it decrypts (same
+  # silent-failure shape as the apps-vms note below/above). The `sops-age` Secret is created
+  # out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: pm-agent
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
   # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
   # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
@@ -496,16 +533,6 @@ patches:
       - op: replace
         path: /data/PLEX_L4_VIP
         value: "192.168.2.24"
-      # PLEX_ADVERTISE_URL — the prod VIP above in Plex's `plex.direct` form, which is the ONLY form
-      # that validates: the server's cert is `*.130072f20c334d6c8f63d0dfef87309b.plex.direct` with no
-      # IP SAN (read off the live listener 2026-10-04), so a bare `https://192.168.2.24:32400` fails
-      # TLS for every client — the 2026-10-04 outage. Measured with full verification, no `curl -k`:
-      # this URL answers /identity with 200 and TLS verify OK; the bare-IP form fails to connect.
-      # THE DASHED IP MUST MATCH /data/PLEX_L4_VIP ABOVE (192.168.2.24 -> 192-168-2-24); nothing
-      # enforces that. The hash is server identity, not a secret, and changes only if Plex reissues.
-      - op: replace
-        path: /data/PLEX_ADVERTISE_URL
-        value: "https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400"
       # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
       # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
       # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
@@ -513,60 +540,48 @@ patches:
         path: /data/LAN_CIDR
         value: "192.168.2.0/24"
       # ------------------------------------------------------------------------------------------
-      # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
+      # BULK (EXTERNAL NFS) STORAGE HOST — the TrueNAS at 192.168.2.30.
       # ------------------------------------------------------------------------------------------
       # Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
-      # radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
+      # radarr, qbittorrent) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
       #
-      # THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
+      # THIS WAS THE REPO'S ONE LOAD-BEARING PLACEHOLDER, AND THE REASON IT COULD BE IS NOW GONE.
-      # vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
+      # The datasets lived on a Proxmox host with a single interface on pfSense's LAN
-      # 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
+      # (192.168.1.9), which prod could reach only through a DNAT at 192.168.2.102 — so the correct
-      # 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
+      # value here depended on an undecided operator choice, and guessing it would have authored a
-      # outcomes give DIFFERENT values here:
+      # cluster that mounts nothing. Both pools were physically moved onto a standalone TrueNAS at
-      #     port-forward 192.168.2.102:2049 -> 192.168.1.9:2049   =>  "192.168.2.102"
+      # 192.168.2.30, prod's OWN /24. No crossing to arrange, no candidates to choose between.
-      #     a WAN->LAN route to 192.168.1.0/24                    =>  "192.168.1.9"
+      # Measured 2026-10-07: 192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and :111 do.
-      # Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
-      # ever became reachable — mounts something unintended. So it stays a placeholder.
       #
-      # WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
+      # ONE PORT IS STILL ALL IT NEEDS: tcp/2049. The PVs mount nfsvers=4.1 and NFSv4 carries mount
-      # NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
+      # and lock in-protocol — no rpcbind/111, no mountd, no statd. The NAS advertises
-      # over v3 was in part this decision's cost: v3 would have needed four ports and three of them
+      # `+3 +4 +4.1 +4.2` (/proc/fs/nfsd/versions, measured), so 4.1 negotiates.
-      # on ephemeral numbers.
-      #
-      # WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
-      # Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
-      # the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
-      # address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
-      # 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
-      # addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
-      #
-      # *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
-      # A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
-      # placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
-      # failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
-      # TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
-      # ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
-      # apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
-      # a revision containing the bulk volumes; the dev overlay renders a working value, which is
-      # where these manifests are meant to be proven first.
       - op: replace
         path: /data/BULK_NFS_SERVER
-        # 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
+        # VERIFIED ON THE NAS 2026-10-07, before any cluster touched it (`exportfs -v`):
-        # host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
+        #   /mnt/Storage/plex       192.168.2.10/.11/.12  rw,sec=sys,secure,root_squash,no_all_squash
-        # reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
+        #   /mnt/Storage/nextcloud  192.168.2.10/.11/.12  ro,sec=sys,secure,root_squash,no_all_squash
-        # keeps the base value 192.168.1.9.
         #
-        # PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
+        # no_all_squash is the flag the entire permission model rests on: client uids pass through
-        #   - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
+        # untouched, so the arrs' PUID=1000 arrives as the owner of a 0755 1000:3000 tree. sec=sys
-        #   - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
+        # keeps that numeric rather than principal-mapped. The export deliberately does NOT carry
-        #     carries no_root_squash, so a root write would have passed and proved nothing, while
+        # no_root_squash, which the Proxmox one did: nothing here needs it, because every write to
-        #     LSIO images default to PUID 911 and would have gone Ready then failed every import
+        # the tree happens after the LSIO drop to uid 1000 and the images only chown their Longhorn
-        #     with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
+        # /config. Both subPath directories (plex/downloads, nextcloud/nextcloud) already exist at
-        #   - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
+        # 0755 1000:3000, so a squashed root can still traverse them for subPath resolution.
-        #     real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
+        #
-        # pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
+        # The ro= posture on the archive is now SERVER-enforced, which the old export also managed.
-        # so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
+        # The media tree's read-only half (plex) stays a CLIENT-side guarantee — all four media pods
-        value: "192.168.2.102"
+        # can land on any node, so the server sees the same three addresses for reader and writers
+        # alike and cannot tell them apart. Unchanged by the move; see docs/BULK-NFS-STORAGE.md §5.
+        #
+        # *** CHANGING THIS VALUE REQUIRES DELETING AND RECREATING THE PVs. *** A bound PV's
+        # spec.nfs is immutable — measured by server dry-run 2026-10-07:
+        #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+        # so Flux's apply FAILS rather than updating and the pods keep their old, dead mount. The
+        # operator sequence is docs/BULK-NFS-STORAGE.md §10. Retain + the renderer's pre-set
+        # claimRef make it safe: no data is touched and each recreated PV rebinds its own PVC.
+        value: "192.168.2.30"
       # ==========================================================================================
       # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
       # ==========================================================================================

conformance/incluster/Containerfile

@@ -98,7 +98,10 @@ RUN set -eux; \
 
 # --- runtime -----------------------------------------------------------------
 ARG FEDORA_VERSION=44
-FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
+# Stage `conformance-runner` is the image that ships; stage `verify` (end of file) is a TEST stage
+# built FROM it that only runs assertions and is never pushed. build-image.sh and the CI
+# pipeline build `--target verify` to prove it and `--target conformance-runner` for the image.
+FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION} AS conformance-runner
 
 ARG CHAINSAW_VERSION=
 ARG KUBECTL_VERSION=
@@ -198,3 +201,86 @@ WORKDIR /workspace
 # prepend something the policy cannot see. `bash` is the interactive fallback for a human
 # debugging the image locally.
 CMD ["/bin/bash"]
+
+# =============================================================================
+# verify — a TEST stage, never shipped. Every assertion conformance/incluster/build-image.sh used
+# to make with `run --user 65532 --read-only --tmpfs /workspace` now runs here, at build time,
+# because CI builds on rootless BuildKit where no container runtime exists.
+#
+# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE:
+#   * uid 65532 ............ exact (USER inherited from the conformance-runner stage).
+#   * no network ........... exact: RUN --network=none, and the first check proves it.
+#   * --read-only rootfs ... NOT reproducible (a RUN step always has a writable layer).
+#                            Approximated by permission: everything uid 65532 can write must lie
+#                            under /workspace, /tmp or /var/tmp. In the Job only /workspace (and
+#                            /results) are writable — the image points HOME and TMPDIR there — and
+#                            readOnlyRootFilesystem is enforced by
+#                            platform/kyverno/admission/conformance-trigger-job-shape.yaml.
+# =============================================================================
+FROM conformance-runner AS verify
+RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
+echo "--- no network (the build-time stand-in for the Job's isolation) ---"
+if getent hosts gitea.home.keithjackson.dev >/dev/null 2>&1 || timeout 3 bash -c '</dev/tcp/1.1.1.1/443' 2>/dev/null; then
+  echo "UNEXPECTED: network reachable in the verify stage"; exit 1
+fi
+echo "confirmed: no network"
+echo "--- id ---";             id
+echo "--- HOME ---";           echo "HOME=${HOME}"
+echo "--- bash ---";           bash --version | head -1
+echo "--- python3 ---";        python3 --version
+echo "--- git ---";            git --version
+echo "--- kubectl ---";        kubectl version --client
+echo "--- chainsaw ---";       chainsaw version
+echo "--- baked versions ---"; cat /etc/conformance-runner-versions
+echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
+stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
+if [ -w /usr/local/bin/conformance-clone ]; then
+  echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
+fi
+echo "confirmed: not writable by uid $(id -u)"
+# Each of these MUST fail. Assert the non-zero exit as well as the message: a check that only
+# greps stderr would pass a build where the guard had stopped rejecting.
+expect_fail() {
+  local label="$1"; shift
+  local out rc=0
+  out="$(env "$@" 2>&1)" || rc=$?
+  echo "--- ${label} ---"
+  printf "%s\n" "${out}" | head -2
+  if [ "${rc}" -eq 0 ]; then
+    echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
+  fi
+  echo "(rejected, exit ${rc})"
+}
+expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
+  conformance-clone --layer l2
+expect_fail "conformance-clone rejects a missing SHA" \
+  conformance-clone
+expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
+  CONFORMANCE_GIT_SHA=main conformance-clone
+expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
+  CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
+expect_fail "conformance-clone rejects a HOME off the workspace volume" \
+  CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+  HOME=/etc conformance-clone
+expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
+  CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+  CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
+expect_fail "conformance-clone refuses a clone target off the workspace volume" \
+  CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+  CONFORMANCE_SRC=/etc/src conformance-clone
+echo "--- the root filesystem is not writable by the runner (expected: a failure) ---"
+if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
+echo "confirmed: / is not writable"
+echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
+stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
+         | grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
+if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf '      %s\n' ${stray} | head -20; exit 1; fi
+echo "confirmed: no writable path outside /workspace, /tmp, /var/tmp"
+echo "--- HOME is writable (the workspace emptyDir) ---"
+touch "${HOME}/.probe" || { echo "UNEXPECTED: ${HOME} not writable"; exit 1; }
+echo "confirmed: ${HOME} is writable"
+echo "--- uid is non-zero ---"
+[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
+echo "confirmed: running as uid $(id -u), not root"
+echo "CONFORMANCE-RUNNER VERIFIED"
+VERIFY

conformance/incluster/README.md

@@ -61,10 +61,15 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
   > digest-pinned to the conformance-runner image, `command` exactly
   > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
   > `env`/`envFrom` were the same shape of hole and are closed in the same policy.
-- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
+- `build-image.sh` — builds the above and **verifies the toolchain by running it**: the
-  image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
+  Containerfile's `verify` stage executes every tool and every `conformance-clone` refusal as
-  prints the exact `podman push --digestfile …` command for the operator instead. See
+  uid 65532 with no network, at build time. It **never pushes**, and neither does anyone from a
-  `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
+  workstation: CI (`.gitea/workflows/images.yaml`) builds the same Containerfile and `verify`
+  stage on rootless BuildKit (`platform/buildkit/`), and on a push to `main` publishes the
+  verified image under `sha-<commit>` and `main`. Renovate then moves
+  `CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned
+  digest in a pull request (that file is a sensitive path, so the bump needs an
+  administrator's `/approve-override`).
   `--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
   and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
   default verify stays offline so an air-gapped build is fully checked, not half-checked).

conformance/incluster/build-image.sh

@@ -1,43 +1,49 @@
 #!/usr/bin/env bash
 ###############################################################################
 # conformance/incluster/build-image.sh — build the pinned `conformance-runner`
-# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+# image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
 #
-#   ./build-image.sh                 # build + verify + print the digest and push command
+#   ./build-image.sh                 # build the verify target, then the image
+#   ./build-image.sh --print-build-args   # print KEY=VALUE build args (what CI consumes)
 #   ./build-image.sh --no-verify     # build only (skip the in-image tool proof)
 #   ./build-image.sh --verify-clone  # ALSO really clone this repo with conformance-clone.
 #                                    #   Opt-in because it needs the Gitea host reachable; the
 #                                    #   default verify is offline so an air-gapped build is
 #                                    #   still fully checked rather than half-checked.
 #   ENGINE=docker ./build-image.sh   # build with docker instead of podman
+#   IMAGE_TAG=foo ./build-image.sh   # override the local tag (CI uses this)
 #
-# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
+# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
-# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
+# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
-# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
+# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
-# non-root uid the policy demands, and prints the exact `podman push` line for a human.
+# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
+# in the job summary; Renovate then opens the pull request that moves conformance/versions.env
+# onto that digest. The proof that the image works is the Containerfile's `verify` stage, which
+# this script (podman locally) and CI both build.
 #
 # VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
 # single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
 # pins the toolbox gate runs with. Nothing in this file invents a version.
 #
-# THE DIGEST CAVEAT — read it before pasting a digest into anything.
+# NO DIGEST IS PRINTED. A manifest digest does not exist until a registry holds the manifest,
-# The digest printed here is the LOCAL manifest digest. It is normally identical to the
+# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
-# registry digest (podman pushes the manifest it already has rather than recompressing),
+# push time (.gitea/workflows/scripts/images.sh) and Renovate carries it into
-# but that is a property of the push, not a guarantee. So the push command below uses
+# conformance/versions.env.
-# `--digestfile`: the digest the registry actually returns is the authoritative one, and
-# that is the one that goes into conformance/versions.env.
 ###############################################################################
 set -euo pipefail
 
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
 ENGINE="${ENGINE:-podman}"
+TAG_OVERRIDE="${IMAGE_TAG:-}"
 
 VERIFY=1
 VERIFY_CLONE=0
+PRINT_ARGS=0
 for arg in "$@"; do
   case "$arg" in
     --no-verify) VERIFY=0 ;;
+    --print-build-args) PRINT_ARGS=1 ;;
     --verify-clone) VERIFY_CLONE=1 ;;
     -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
     *) echo "unknown argument: $arg" >&2; exit 2 ;;
@@ -48,8 +54,6 @@ log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
 info() { printf '    %s\n' "$*"; }
 die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
 
-command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
-
 # --- pins: sourced, never invented -------------------------------------------
 TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
 [ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
@@ -59,14 +63,21 @@ source "${TOOLCHAIN_ENV}"
 : "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
 : "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
 
+# THE build args, in ONE place; `--print-build-args` hands them to the CI pipeline, which builds
+# this Containerfile on rootless BuildKit.
+BUILD_ARGS=("CHAINSAW_VERSION=${CHAINSAW_VERSION}" "KUBECTL_VERSION=${KUBECTL_VERSION}" "FEDORA_VERSION=${FEDORA_VERSION}")
+if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
+
+command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
+
 # --- image name: fixed by the admission policy -------------------------------
 # platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
 # gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
 # is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
 IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
-# The tag is build convenience only (a human-readable handle for the push). Admission
+# The tag is a local handle only. CI re-tags what it publishes (git SHA + `main`), and
-# ignores it: only the digest is ever referenced from a Job.
+# admission ignores tags entirely: only the digest is ever referenced from a Job.
-IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
+IMAGE_TAG="${TAG_OVERRIDE:-chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}}"
 IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
 
 log "Building ${IMAGE_REF}"
@@ -76,86 +87,31 @@ info "  CHAINSAW        ${CHAINSAW_VERSION}"
 info "  KUBECTL         ${KUBECTL_VERSION}"
 info "  FEDORA          ${FEDORA_VERSION}"
 
-"${ENGINE}" build \
+# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
-  --file "${HERE}/Containerfile" \
+LABEL_ARGS=()
-  --tag "${IMAGE_REF}" \
+[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
-  --build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
+
-  --build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
+ARG_FLAGS=()
-  --build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
+for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
-  "${HERE}"
 
 # --- verify: the point of the whole exercise ---------------------------------
-# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
+# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So the proof is that
-# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
+# the tools EXECUTE, as the NON-ROOT uid, with no network, that conformance-clone is not writable
-# filesystem — the shape the admission policy forces. A pass here is the only evidence
+# by that uid and refuses every hostile input, and that nothing outside /workspace is writable.
-# this image is usable; a build that merely succeeded is not.
+# That proof is the Containerfile's `verify` stage, built here and in CI alike (CI has no
+# container runtime — rootless BuildKit only). See the stage's header for what a build step can
+# and cannot reproduce of the Job's shape. A build that merely succeeded is not evidence.
 if [ "${VERIFY}" -eq 1 ]; then
-  log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
+  log "Verifying the toolchain (Containerfile target: verify)"
-  "${ENGINE}" run --rm \
+  "${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
-    --user 65532:65532 \
-    --read-only \
-    --tmpfs /workspace:rw,mode=1777 \
-    --entrypoint "" \
-    "${IMAGE_REF}" \
-    /bin/bash -euo pipefail -c '
-      echo "--- id ---";             id
-      echo "--- HOME ---";           echo "HOME=${HOME}"
-      echo "--- bash ---";           bash --version | head -1
-      echo "--- python3 ---";        python3 --version
-      echo "--- git ---";            git --version
-      echo "--- kubectl ---";        kubectl version --client
-      echo "--- chainsaw ---";       chainsaw version
-      echo "--- baked versions ---"; cat /etc/conformance-runner-versions
-      echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
-      stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
-      if [ -w /usr/local/bin/conformance-clone ]; then
-        echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
-      fi
-      echo "confirmed: not writable by uid $(id -u)"
-      # Each of these MUST fail. Assert the non-zero exit as well as the message: a check
-      # that only greps stderr would pass a build where the guard had stopped rejecting.
-      # (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
-      # expected failure — without it the verify dies at the first negative case.)
-      expect_fail() {
-        local label="$1"; shift
-        local out rc=0
-        out="$(env "$@" 2>&1)" || rc=$?
-        echo "--- ${label} ---"
-        printf "%s\n" "${out}" | head -2
-        if [ "${rc}" -eq 0 ]; then
-          echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
-        fi
-        echo "(rejected, exit ${rc})"
-      }
-      expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
-        conformance-clone --layer l2
-      expect_fail "conformance-clone rejects a missing SHA" \
-        conformance-clone
-      expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
-        CONFORMANCE_GIT_SHA=main conformance-clone
-      expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
-        CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
-      expect_fail "conformance-clone rejects a HOME off the workspace volume" \
-        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
-        HOME=/etc conformance-clone
-      expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
-        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
-        CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
-      expect_fail "conformance-clone refuses a clone target off the workspace volume" \
-        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
-        CONFORMANCE_SRC=/etc/src conformance-clone
-      echo "--- rootfs is read-only (expected: a failure) ---"
-      if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
-      echo "confirmed: / is read-only"
-      echo "--- HOME is writable (the workspace emptyDir) ---"
-      touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
-      echo "--- uid is non-zero ---"
-      [ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
-    '
   log "Toolchain verified"
 fi
 
+"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target conformance-runner \
+  --file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
+
 # --- verify the clone helper against the REAL repository (opt-in: needs network) ----
+# LOCAL ONLY: this one `run`s the image, which needs a container engine, so CI does not do it.
 # The negative cases above prove conformance-clone refuses bad input. This proves the
 # positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
 # rev-parse assertion is only an unexercised line of shell.
@@ -184,46 +140,12 @@ if [ "${VERIFY_CLONE}" -eq 1 ]; then
   log "conformance-clone verified against the real repository"
 fi
 
-# --- digest ------------------------------------------------------------------
+# --- result ------------------------------------------------------------------
-DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
 IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
 SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
 
 log "Built ${IMAGE_REF}"
-info "local manifest digest  ${DIGEST}"
 info "image id  ${IMAGE_ID}"
 info "size      $(( SIZE / 1024 / 1024 )) MiB"
-
+info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml) and"
-cat <<EOF
+info "Renovate moves conformance/versions.env onto the registry-returned digest."
-
-================================================================================
-NOT PUSHED. Publishing is an operator action — this script holds no registry
-credential and will never acquire one.
-
-OPERATOR: run these two commands, then paste the digest from the digestfile into
-conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
-
-    ${ENGINE} login gitea.home.keithjackson.dev
-
-    ${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
-      ${IMAGE_REF}
-
-Then:
-
-    cat /tmp/conformance-runner.digest
-    # -> sha256:....  THIS is the authoritative digest, not the local one above.
-
-    # conformance/versions.env
-    CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
-
-WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
-is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
-${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
-and so is any other repository path.
-
-TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
-\`${IMAGE_REPO}@sha256:*\` to be replaced with the
-LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
-That is a separate, human-reviewed PR against the security boundary.
-================================================================================
-EOF

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -363,6 +363,54 @@ SCRAPE_OWNERS: list[dict] = [
         "why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules "
                "that failed to notice prod running with zero volume backups",
     },
+    # ── ADDED 2026-10-17 (feat/automated-upgrades). A NEW OWNER, NOT A RELAXED ASSERTION. ──────
+    # This entry is purely additive: it registers a new metric family and its enablement, and it
+    # changes nothing about how any existing owner is evaluated. It makes the check STRICTER for the
+    # component it covers, not looser — it adds two `require` patterns plus a `companion` obligation
+    # that platform/kured must keep satisfying, so a future edit that disables kured's Service,
+    # ServiceMonitor or NetworkPolicy FAILS the gate instead of silently re-inerting three alerts.
+    #
+    # WHY THE CHECK HAD TO BE TOUCHED AT ALL, rather than the manifest being wrong: assertion (3)
+    # fails any alert over a metric no owner claims, and the escape hatch (KNOWN_UNWIRED_METRICS) is
+    # SHRINK-ONLY, so declaring `kured_*` as accepted debt is not permitted and would be the wrong
+    # answer anyway — the scrape genuinely IS wired here. The only two honest options were to
+    # register the owner (this) or to ship the reboot automation with no alerting at all, which for a
+    # component whose every failure mode is silent would have been the worse engineering choice.
+    # `conformance/**` is a sensitive approval path precisely so that this reasoning gets read.
+    {
+        "owner": "kured: service.create + metrics.create in the HelmRelease",
+        "prefixes": ("kured_",),
+        "file": "platform/kured/helmrelease.yaml",
+        # BOTH keys are required and the chart defaults BOTH to false. `metrics.create` makes the
+        # ServiceMonitor; `service.create` makes the Service it selects. Enable only the monitor and
+        # it selects nothing, so the single kured metric is never scraped — and a PrometheusRule over
+        # a series no target produces is INERT, which for this component means its own liveness
+        # alarm is the thing that breaks. Matched as two separate anchored blocks rather than one
+        # pattern because they are separate top-level values keys, not nested.
+        "require": [
+            r"service:\s*\n\s*create:\s*true",
+            r"metrics:\s*\n\s*create:\s*true",
+        ],
+        "forbid": [],
+        "why": "kured_reboot_required is the ONLY metric kured exposes (one GaugeVec, "
+               "cmd/kured/main.go:92-96) and it powers all three platform-health.node-reboots "
+               "rules. Those rules are the only way to learn that OS reboot automation has stalled: "
+               "kured stays 1/1 Running while rebooting nothing, so a missed scrape here is "
+               "indistinguishable from a cluster that needed no reboots",
+        # The ServiceMonitor ALONE is not sufficient — the same NetworkPolicy trap Longhorn hit.
+        "companion": {
+            "file": "platform/kured/kustomization.yaml",
+            "require": [r"^\s*-\s*networkpolicy\.yaml\s*$"],
+            "why": "platform/kured/networkpolicy.yaml ships a default-deny baseline (the kured "
+                   "namespace is a platform namespace, so Kyverno's app-namespace-scoped generator "
+                   "does not cover it) AND the ingress allow that lets Prometheus reach :8080 "
+                   "through it. Drop the networkpolicy from this kustomization and the deny-all "
+                   "goes too, so this currently fails SAFE — but reinstating the baseline without "
+                   "the allow reproduces Longhorn's measured failure exactly: targets DISCOVERED, "
+                   "every scrape 'context deadline exceeded', TSDB empty, all three rules quiet. "
+                   "Keeping the two in one file makes that pair inseparable",
+        },
+    },
 ]
 
 # `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)

conformance/tests/l0-static/alert-rules-unit-tested/check.sh

@@ -29,7 +29,8 @@
 # THEN  (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
 #           infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
 #           lockstep rule), the Containerfile takes it as a build ARG, installs it from the
-#           pinned prometheus release and records it for provenance, dev.sh passes it, and the
+#           pinned prometheus release and records it for provenance, build-image.sh (the one
+#           recipe `dev.sh build` and CI both use) passes it, and the
 #           CI runner installs it for the l0 stage;
 #       (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
 #           evidence only if the engine matches the one prod evaluates rules with;
@@ -246,10 +247,12 @@ check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
       "infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
       "that provenance line is how an operator tells a stale image from a current one")
 
-DEVSH = ROOT / "infra/devcontainer/dev.sh"
+# The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build`
-check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
+# and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe.
-      "infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
+BUILDSH = ROOT / "infra/devcontainer/build-image.sh"
-      "empty and the build would fetch a bogus URL")
+check(BUILDSH.is_file() and '"PROMTOOL_VERSION=$PROMTOOL_VERSION"' in BUILDSH.read_text(),
+      "infra/devcontainer/build-image.sh does not pass PROMTOOL_VERSION in its BUILD_ARGS — the ARG "
+      "would be empty and the build would fetch a bogus URL")
 
 INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
 if INSTALL.is_file():

conformance/tests/l0-static/ha-control-plane-as-code/check.sh

@@ -445,8 +445,21 @@ join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
 check(join is not None,
       "roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
       "— without a join target the servers are separate single-node clusters")
-join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
+# LINE-BOUNDED ON PURPOSE -- DO NOT "SIMPLIFY" BACK TO `.*` + re.S.
-                      K3S_TEXT, re.S)
+# This was `--server https://.*?\n(?:.*\n){0,10}?\s*when:...` with re.S, which made every `.`
+# match newlines too. That is nested unbounded repetition (a `{0,10}` of a quantifier that can each
+# swallow the whole document), so a FAILED or late match backtracks exponentially. It cost
+# **19m54s of CPU in CI** (measured 2026-10-18 via `docker top` on the live job container) and made
+# the whole merge gate take ~28 minutes instead of ~2 on any change touching `infra/**` -- because
+# `ci/touched-requirements.sh` maps `infra/*` to R-ENV-01, which pulls in this whole-repo check.
+# It was latent for as long as the regex existed and only crossed the threshold when
+# roles/k3s_server grew by ~100 lines (the kubelet graceful-shutdown config, PR #71): K3S_TEXT is
+# every *.yml in that role concatenated, so the role growing is the input growing.
+# `[^\n]*` cannot cross a line, so the `{0,10}` is a genuine ten-line window with nothing to
+# backtrack into. Measured on the same input: 0.2 ms, same verdict (match found).
+join_when = re.search(
+    r"--server https://[^\n]*\n(?:[^\n]*\n){0,10}?[ \t]*when:[ \t]*k3s_server_mode[ \t]*==[ \t]*'secondary'",
+    K3S_TEXT)
 check(join_when is not None,
       "the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
       "would also try to join itself")

conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh

@@ -26,7 +26,10 @@
 #       KNOWN_NON_CATALOG_PUBLISHERS — shrink-only, stale entries FAIL;
 #   (7) the check is LOAD-BEARING: whole-repo scope declared in chainsaw-test.yaml, honoured by
 #       ci/l0.sh's --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written
-#       ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears).
+#       ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears);
+#   (8) the external-dns HelmRelease declares `annotationPrefix` explicitly and it matches the
+#       prefix of EVERY external-dns annotation in the tree and in catalog/render.py — added after
+#       the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime.
 #
 # WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
 #   - That the record is gone from Cloudflare. Live read; belongs beside
@@ -90,9 +93,18 @@ CHECKS = 0
 #   ControllerValue = "dns-controller"                  -> the value that CLAIMS an object
 # source/annotations/processors.go IsControllerMismatch skips a source object when the key is
 # PRESENT and the value is NOT ControllerValue. So "disowned" == key present, value != claimed.
-CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
+#
+# THE PREFIX IS NOT A CONSTANT OF NATURE. Every key external-dns reads is `<annotationPrefix><name>`,
+# and the prefix is a runtime flag whose DEFAULT moved in v0.22.0 (#6424: alpha -> GA, "with no
+# fallback"). On 2026-11-07 the chart bump to v0.23.0 made every key below invisible to the running
+# controller while this check — which only asserted the keys were PRESENT — stayed green. (8) now
+# ties ANNOTATION_PREFIX to the HelmRelease's explicit `annotationPrefix`.
+ANNOTATION_PREFIX = "external-dns.alpha.kubernetes.io/"
+CONTROLLER_KEY = ANNOTATION_PREFIX + "controller"
 CONTROLLER_CLAIMED = "dns-controller"
-HOSTNAME_KEY = "external-dns.alpha.kubernetes.io/hostname"
+HOSTNAME_KEY = ANNOTATION_PREFIX + "hostname"
+# Any key that looks like an external-dns annotation, under ANY prefix variant (alpha, GA, typos).
+EXTERNAL_DNS_KEY_RE = re.compile(r"^external-dns(\.[a-z0-9-]+)*\.kubernetes\.io/")
 
 
 def check(cond, msg):
@@ -231,9 +243,7 @@ for sub in SCAN_DIRS:
             ann = md.get("annotations") or {}
             if kind == "HTTPRoute":
                 routes.append((rel, d))
-            elif kind == "Service" and any(
+            elif kind == "Service" and any(EXTERNAL_DNS_KEY_RE.match(a) for a in ann):
-                a.startswith("external-dns.alpha.kubernetes.io/") for a in ann
-            ):
                 dns_services.append((rel, d))
 
 check(len(routes) > 0, "inventory: no HTTPRoute found under apps/ platform/ runbooks/ — parse "
@@ -444,6 +454,67 @@ check(
     "and the whole-repo-invariants-wired meta-test fails the gate on a hand-written stage.",
 )
 
+# =============================================================================================
+# (8) THE RUNNING CONTROLLER READS THE PREFIX THIS TREE WRITES.
+#     (a) the external-dns HelmRelease sets `values.annotationPrefix` EXPLICITLY (an upstream
+#         default is exactly what moved under us) and it equals ANNOTATION_PREFIX;
+#     (b) it does not ALSO enable `--enable-legacy-annotation-prefix` (dual-read hides a
+#         half-migration; upstream says the flag will be removed);
+#     (c) every external-dns-looking annotation key on any object under apps/ platform/ runbooks/
+#         uses ANNOTATION_PREFIX — a key under another prefix is silently ignored at runtime;
+#     (d) catalog/render.py emits no external-dns key under another prefix.
+#     Migrating to the GA prefix = change ANNOTATION_PREFIX, the HelmRelease value, the renderer
+#     and every manifest in ONE commit; this assertion fails on any partial state.
+# =============================================================================================
+HR_PATH = ROOT / "platform/externaldns/helmrelease.yaml"
+hr = next((d for d in docs_of(HR_PATH) if d.get("kind") == "HelmRelease"
+           and (d.get("metadata") or {}).get("name") == "external-dns"), None)
+check(hr is not None, f"(8) {HR_PATH.relative_to(ROOT)} holds no HelmRelease `external-dns` — "
+                      f"cannot prove which annotation prefix the controller reads")
+values = ((hr or {}).get("spec") or {}).get("values") or {}
+hr_prefix = values.get("annotationPrefix")
+check(
+    hr_prefix == ANNOTATION_PREFIX,
+    f"(8a) external-dns HelmRelease `values.annotationPrefix` is {hr_prefix!r}, but every "
+    f"external-dns annotation in this tree uses {ANNOTATION_PREFIX!r}. A mismatch (or relying on "
+    f"the upstream default, which changed in v0.22.0) makes the controller IGNORE `controller: "
+    f"none`, the Gateway `target` and Service `hostname` — internal names go public, public names "
+    f"answer RFC1918 (incident 2026-11-07).",
+)
+extra = values.get("extraArgs") or []
+extra_s = " ".join(f"{k}={v}" for k, v in extra.items()) if isinstance(extra, dict) \
+    else " ".join(map(str, extra))
+check("enable-legacy-annotation-prefix" not in extra_s and "annotation-prefix" not in extra_s,
+      "(8b) external-dns extraArgs must not set --annotation-prefix or "
+      "--enable-legacy-annotation-prefix; the prefix is declared ONLY via values.annotationPrefix "
+      "so this check can see it")
+
+foreign: list[str] = []
+seen_keys = 0
+for sub in SCAN_DIRS:
+    base = ROOT / sub
+    if not base.is_dir():
+        continue
+    for p in sorted(base.rglob("*.yaml")):
+        rel = p.relative_to(ROOT).as_posix()
+        for d in docs_of(p):
+            for k in annotations_of(d):
+                if isinstance(k, str) and EXTERNAL_DNS_KEY_RE.match(k):
+                    seen_keys += 1
+                    if not k.startswith(ANNOTATION_PREFIX):
+                        foreign.append(f"{rel}:{'/'.join(ident(d))} {k}")
+check(seen_keys > 0, "(8c) no external-dns annotation found anywhere — parse contract broken "
+                     "(the Gateway target alone guarantees at least one)")
+check(not foreign,
+      f"(8c) external-dns annotations under a prefix the controller does NOT read "
+      f"({ANNOTATION_PREFIX!r} is configured): {foreign}")
+render_src = read(CATALOG / "render.py")
+bad_render = sorted({m for m in re.findall(r"external-dns(?:\.[a-z0-9-]+)*\.kubernetes\.io/",
+                                           render_src) if m != ANNOTATION_PREFIX})
+check(not bad_render,
+      f"(8d) catalog/render.py emits external-dns keys under {bad_render}, but the controller "
+      f"reads {ANNOTATION_PREFIX!r}")
+
 # =============================================================================================
 if FAILURES:
     for f in FAILURES:

conformance/tests/l0-static/keycloak-passkey-reconciler/chainsaw-test.yaml

@@ -0,0 +1,108 @@
+# =============================================================================
+# L0 STATIC — the passkey configuration is derived from one declaration, continuously
+# reconciled, and the browser login flow converges BLUE/GREEN: the BOUND flow is
+# unwritable, and binding a new one is git's explicit decision (bind-browser-flow)
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# R-IDP-03 (identity config is declarative) and R-IDP-04 (MFA available) are both
+# implemented through a mechanism that FAILS OPEN: the KeycloakRealmImport is a ONE-SHOT.
+# Measured on prod 2026-10-18, hours after the passkey declaration merged:
+#
+#   kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
+#   Done=True  Started=False  HasErrors=False
+#
+# Once it rests at Done=True the operator never re-runs it, so the WebAuthn passwordless
+# policy and the restructured `browser-mfa` flow that keycloak-realm-import.yaml declares
+# are INERT — in git, applied by Flux, reported Ready, reaching the realm never. Merging
+# them changed nothing about how anyone logs in, and nothing anywhere said so.
+# Declarative config that silently does not apply is not declarative config.
+#
+# GIVEN the repository as authored,
+# THEN  the reconciler's desired state is GENERATED from the realm import (so the RP ID —
+#       which is baked into every passkey at enrolment and can NEVER be re-keyed — is
+#       declared exactly once); both files are rendered by the SAME Flux Kustomization, so
+#       their ${BASE_DOMAIN} token cannot resolve to two values; the renderer's exported
+#       policy keys and the reconciler's writable keys are the SAME tuple; the realm
+#       import declares NO requiredActions block (26.0.6 treats it as REPLACE, which would
+#       delete CONFIGURE_TOTP and lock out every platform-admin); the enrolment action is
+#       enabled but NOT a defaultAction, so enrolment stays opt-in; the reconciler carries
+#       no ServiceAccount token, mounts only what it needs, passes no mode flag, and
+#       contains no dollar sign; the endpoint allow-list contains NO delete, NO
+#       single-execution or config-edit write, and NO required-action write for any alias
+#       other than the one named literally; the protected-subtree guard exists, is wired
+#       into the single request choke point and is fail-closed; the SIBLING client-secret
+#       reconciler's allow-list is NOT widened; and the reconciler's own `--selftest`
+#       (extracted from the shipped ConfigMap and executed) proves compare-then-write,
+#       idempotence, strict read-modify-write, the RpEntityName gate, the bound flow and
+#       its sub-flows unwritable, the git bind switch and a rollback that sticks, the
+#       conditional-OTP regression and fail-closed — with mutations.py proving that
+#       selftest can FAIL.
+#
+# WHY THE FLOW IS BUILT BLUE/GREEN AND BOUND ONLY ON GIT'S SAY-SO, asserted here rather
+# than only argued: converging a five-flow, ten-execution restructure needs
+# create-subflow, create-execution, set-requirement and bind-config as SEPARATE admin API
+# calls with no transaction. Doing that to the BOUND flow means a run that dies between
+# them leaves a HALF-BUILT FLOW AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp,
+# Semaphore, PiHole, Crafty, the *arr stack AND kubectl — this repo has had a total SSO
+# outage from a realm change (2026-09-06), and the conditional-OTP sub-flow once 403'd
+# every platform-admin. So every multi-call write lands on a NEW, unused candidate flow,
+# which is verified by readback before anything is bound. Binding it is then ONE atomic
+# PUT of browserFlow, made only when passkey-desired-state.yaml sets
+# bind-browser-flow: "true" (default "false"): an untested login flow never goes live on
+# a schedule tick, and with the switch off an operator's rollback is never undone.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML +
+# stdlib); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
+# Test wraps the SAME script.
+#
+# NOT proven here: that the LIVE realm's policy, enrolment action or browser flow match
+# git (that is cluster state — the reconciler's `--audit` mode is the read-only probe for
+# it, and NO L2 test is added by this change, so the live realm is UNVERIFIED by CI);
+# that a passkey ceremony actually completes or that the ALTERNATIVE fork degrades to the
+# password form (needs a real browser against a real realm); that the five flow-building
+# admin-API calls behave against a live Keycloak the way the 26.0.6 source says they do
+# (every proof here is against an in-process stub); or that the CronJob's schedule fires.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: keycloak-passkey-reconciler
+  labels:
+    conformance.home.k3s/R-IDP-03: "true"
+    conformance.home.k3s/R-IDP-04: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE, for the same reason the sibling keycloak-client-secret-reconciler
+    # check declares it: this invariant spans platform/ (the realm import, the reconciler,
+    # the generated desired state), platform-config/ (which Flux Kustomization renders
+    # them, and therefore which substitution pass they share), ci/ (the renderer) and
+    # clusters/ (that both are reachable from both entrypoints). A change that breaks it
+    # need not attribute R-IDP-03/04 at all, and the requirement-scoped selection would
+    # then filter this check out of exactly the change that introduces the bug — which is
+    # how 12 unusable prod OIDC clients sat behind an entirely green cluster on
+    # 2026-09-06.
+    conformance.home.k3s/scope: "whole-repo"
+    # Not safety-critical in the harness's sense (it mutates nothing and needs no
+    # cluster), but it guards a component that holds WRITE access to the identity
+    # provider's realm settings.
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      The realm's passkey configuration — policy, enrolment action AND browser login
+      flow — is declared once in the realm import, derived from it by a gated renderer,
+      and continuously reconciled; the flow converges blue/green, so a one-shot import
+      that never re-runs can neither leave the configuration inert nor let a reconciler
+      half-rewrite the login path of the whole platform.
+spec:
+  steps:
+    - name: audit-passkey-declaration-renderer-and-reconciler
+      description: >-
+        Assert the desired state is derived from the realm import and shares its
+        substitution pass, that the reconciler cannot write the bound flow or delete
+        anything and binds only on git's bind-browser-flow, that the sibling reconciler
+        is unwidened, and run the reconciler's own selftest plus the mutation suite that
+        proves it can fail.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/keycloak-passkey-reconciler/check.sh

@@ -0,0 +1,754 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — keycloak-passkey-reconciler
+# Serves: R-IDP-03 ("Identity config (realms, clients, groups, roles) is declarative."
+#                   — spec docs/15-identity-and-access.md)
+#         R-IDP-04 ("IdP secrets SOPS-encrypted; MFA available."
+#                   — spec docs/15-identity-and-access.md)
+#
+# WHY THIS EXISTS — THE REALM IMPORT IS A ONE-SHOT THAT FAILS OPEN
+# ---------------------------------------------------------------
+# platform/keycloak/keycloak-realm-import.yaml declares the WebAuthn PASSWORDLESS
+# policy and a restructured `browser-mfa` browser flow. Measured on prod 2026-10-18,
+# hours after that declaration merged:
+#
+#   kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
+#   Done=True  Started=False  HasErrors=False
+#
+# A KeycloakRealmImport is a ONE-SHOT. Once it rests at Done=True the operator never
+# re-runs it, so every realm-level field and every flow declared after the FIRST
+# import is inert — in git, applied by Flux, reported Ready, reaching the realm
+# never. The same shape as the 2026-09-06 client-secret incident (12 prod clients
+# holding the literal `(env:...)` placeholder behind an entirely green cluster), and
+# the same fix: a level-triggered reconciler. This check is the static half.
+#
+# THE RECONCILER HOLDS WRITE ACCESS TO THE IDENTITY PROVIDER. That is the reason this
+# check is unusually thorough about its BLAST RADIUS rather than only its behaviour.
+#
+# GIVEN the repository as authored,
+# THEN  (1) THE DESIRED STATE IS DERIVED, NOT DECLARED TWICE: the reconciler's input
+#           (platform/keycloak/passkey-desired-state.yaml) is GENERATED from the realm
+#           import by ci/render-keycloak-passkey-policy.py, and this check RUNS the
+#           renderer with --check, so a hand edit that makes the two disagree is
+#           unmergeable. The RP ID is the sharpest case: it is baked into every passkey
+#           at enrolment and can NEVER be re-keyed, so two declarations that can skew
+#           about the domain is an unrecoverable-credential bug, not a cosmetic one.
+#       (2) BOTH FILES ARE RENDERED BY THE SAME FLUX KUSTOMIZATION.
+#           postBuild.substituteFrom is per-Kustomization, so a split would let the
+#           ${BASE_DOMAIN} token in the realm import and the one in the desired state
+#           resolve to two different values — and an RP ID mismatch invalidates every
+#           enrolled passkey with no error the user can act on.
+#       (3) THE DESIRED STATE USES NO SUBSTITUTION TOKEN THE REALM IMPORT DOES NOT, and
+#           declares no realm key outside the reconciler's write surface.
+#       (4) THE TWO WRITE SURFACES AGREE: the renderer's exported policy keys and the
+#           reconciler's POLICY_FIELDS are the SAME tuple. If they drifted, the renderer
+#           could emit a key the reconciler refuses (fail-closed forever) or the
+#           reconciler could accept a key the renderer never emits (dead surface).
+#       (5) THE RECONCILER MANIFEST IS SAFE AND WIRED: reconciled from BOTH cluster
+#           entrypoints, mounts exactly the ConfigMaps its code defaults to, carries NO
+#           ServiceAccount token, and contains no dollar sign (it ships inside a
+#           SUBSTITUTED Flux path, where an unescaped dollar-brace expansion is
+#           rewritten before any container runs).
+#       (6) THE CURRENTLY BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the
+#           SOURCE, not the comments. The flow converges BLUE/GREEN (build under a new
+#           alias, verify, flip browserFlow in one atomic PUT), so flow-create and
+#           execution-add endpoints ARE on the allow-list and their presence is not the
+#           hazard. What is asserted instead: nothing is DELETABLE (the superseded flow
+#           is the rollback target); the execution- and config-id-scoped write endpoints
+#           stay off the list because they carry no flow alias and so cannot be screened;
+#           no required-action write reaches any alias but the one named literally; and
+#           the guard the allow-list CANNOT express exists in the source and is wired
+#           into the single request choke point, FAIL-CLOSED until the live binding has
+#           been read. That last one is what the whole design rests on: a reconciler that
+#           half-rewrites the BOUND browser flow is a total SSO outage on a loop, and
+#           this repo has had one (2026-09-06).
+#      (6b) THE BIND IS GIT'S DECISION: the desired-state ConfigMap carries
+#           bind-browser-flow as exactly "true" or "false", the script reads it from the
+#           SAME mount, and the CronJob passes no flag that could widen it. With "false"
+#           (the default) the reconciler builds and verifies but never PUTs browserFlow,
+#           so an operator's rollback sticks; the selftest proves that end to end.
+#       (7) THE SIBLING RECONCILER IS UNTOUCHED: client-secret-reconciler.yaml's
+#           allow-list is NOT widened by this work. Its --selftest asserts it refuses
+#           /users, /groups, /roles, /authentication and the whole-realm endpoint, and
+#           its guarantee that prod's real user and the media-users / platform-admins /
+#           viewers groups survive every run must stay LITERALLY true.
+#       (8) THE RECONCILER'S BEHAVIOUR IS PROVEN, NOT ASSERTED: this check EXTRACTS the
+#           script from the ConfigMap and executes its `--selftest`, which drives the
+#           real code paths against an in-process stub Keycloak that MODELS the three
+#           Keycloak 26.0.6 behaviours that bite — the RpEntityName gate on the
+#           passwordless policy, the all-or-nothing policy rebuild that defaults every
+#           omitted key, and the required-action representation replace.
+#      (8b) AND THAT PROOF IS FALSIFIABLE: mutations.py injects plausible regressions
+#           one at a time and requires --selftest to CATCH every one. A passing suite
+#           proves nothing if it cannot fail, and "reported success while the realm was
+#           broken" is the exact failure this component exists to end.
+#   AND (9) THIS CHECK IS LOAD-BEARING: it DECLARES
+#           `conformance.home.k3s/scope: whole-repo` so ci/l0.sh runs it unconditionally
+#           via --include-scope, ci/gate.sh blocks the merge when the l0 stage fails, and
+#           ci/touched-requirements.sh maps platform/keycloak/ to these requirement IDs.
+#           An unreferenced test is precisely the "looks applied and isn't" failure this
+#           repo keeps hitting.
+#
+# NOT proven here (named honestly, in the style of the sibling L0 checks):
+#   - that the LIVE realm's policy, enrolment action or browser flow match git. That is
+#     cluster state. The reconciler's own `--audit` mode is the read-only probe for it
+#     and is what an L2 test should run; no such L2 test is added by this change, so the
+#     live realm is UNVERIFIED by CI.
+#   - that the declared browser flow actually WORKS as a login path. Nothing offline can
+#     prove that a passkey ceremony completes, that the ALTERNATIVE fork degrades to the
+#     password form, or that a platform-admin is still asked for TOTP. Those need a real
+#     browser against a real realm.
+#   - that the reconciler's first real run against prod will succeed. The flow is built
+#     through five admin-API endpoints whose request shapes were read from the 26.0.6
+#     source but never exercised against a live Keycloak.
+#   - that the admin credential decrypts to a usable password. Key PRESENCE is checkable
+#     offline; the plaintext is not, and must not be.
+#   - that the CronJob's image can be pulled or its schedule fires.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+IMPORT_FILE="platform/keycloak/keycloak-realm-import.yaml"
+RECON_FILE="platform/keycloak/passkey-reconciler.yaml"
+STATE_FILE="platform/keycloak/passkey-desired-state.yaml"
+RENDERER="ci/render-keycloak-passkey-policy.py"
+SIBLING_FILE="platform/keycloak/client-secret-reconciler.yaml"
+
+# Honest SKIP (never green) if the thing under test is absent entirely.
+for f in "$IMPORT_FILE" "$RECON_FILE" "$STATE_FILE" "$RENDERER"; do
+  if [ ! -f "${ROOT}/${f}" ]; then
+    echo "check SKIPPED: ${f} absent — nothing to audit" >&2
+    exit 3
+  fi
+done
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit of the Keycloak passkey declaration, the renderer that derives the
+reconciler's desired state from it, and the reconciler's blast radius. PyYAML parses the
+manifests; lib/status.py is IMPORTED for the Flux graph walk rather than forked, so there
+is one definition of "what a Flux path renders" in the harness."""
+from __future__ import annotations
+
+import ast
+import json
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+import yaml
+
+ROOT = Path(sys.argv[1])
+# This body is fed to `python3 -` on stdin, so there is no __file__ to derive from; the
+# test directory is named relative to the repo root the wrapper passes in.
+DIR_PATH = (ROOT / "conformance" / "tests" / "l0-static"
+            / "keycloak-passkey-reconciler")
+sys.path.insert(0, str(ROOT / "conformance" / "lib"))
+import status as st  # noqa: E402
+
+IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
+RECON_FILE = "platform/keycloak/passkey-reconciler.yaml"
+STATE_FILE = "platform/keycloak/passkey-desired-state.yaml"
+RENDERER = "ci/render-keycloak-passkey-policy.py"
+SIBLING_FILE = "platform/keycloak/client-secret-reconciler.yaml"
+STATE_CM_NAME = "keycloak-passkey-desired-state"
+STATE_DATA_KEY = "passkey-desired-state.json"
+BIND_KEY = "bind-browser-flow"
+SCRIPT_KEY = "reconcile-keycloak-passkeys.py"
+ENROL_ACTION = "webauthn-register-passwordless"
+DOLLAR = chr(36)
+
+failures: list[str] = []
+
+
+def check(cond, msg):
+    if not cond:
+        failures.append(msg)
+
+
+def load(rel):
+    return list(yaml.safe_load_all((ROOT / rel).read_text()))
+
+
+# ================================================== (1) the desired state is DERIVED
+proc = subprocess.run([sys.executable, str(ROOT / RENDERER), "--check"],
+                      capture_output=True, text=True, timeout=120)
+check(proc.returncode == 0,
+      f"{STATE_FILE} has DRIFTED from {IMPORT_FILE} (ci/render-keycloak-passkey-policy"
+      f".py --check failed). The reconciler would converge the realm on a passkey policy "
+      f"the realm import does not declare, and audit the login flow against a tree "
+      f"nobody authored. The RP ID is the sharpest case: it is baked into every passkey "
+      f"at enrolment and can NEVER be re-keyed, so two declarations that skew about the "
+      f"domain is an unrecoverable-credential bug.\n        "
+      + "\n        ".join((proc.stdout + proc.stderr).strip().splitlines()[-12:]))
+
+# The realm import must actually declare the things the renderer derives from. If it
+# stops, the renderer fails -- but say so HERE too, because a SKIP-shaped failure in the
+# renderer would otherwise be the only signal.
+imp_docs = [d for d in load(IMPORT_FILE) if d]
+imp = next((d for d in imp_docs if d.get("kind") == "KeycloakRealmImport"), None)
+check(imp is not None, f"{IMPORT_FILE}: no KeycloakRealmImport document")
+realm = ((imp or {}).get("spec") or {}).get("realm") or {}
+policy_keys_in_import = sorted(k for k in realm
+                               if k.startswith("webAuthnPolicyPasswordless"))
+check(policy_keys_in_import,
+      f"{IMPORT_FILE} declares no webAuthnPolicyPasswordless* field, so there is no "
+      f"passkey policy for the reconciler to converge")
+check(realm.get("browserFlow"),
+      f"{IMPORT_FILE} declares no spec.realm.browserFlow — a perfect flow that is not "
+      f"BOUND changes nothing about how anyone logs in")
+
+# THE requiredActions TRAP, enforced rather than only commented. The realm import's own
+# header explains it at length: Keycloak 26.0.6 treats requiredActions in a realm
+# representation as REPLACE, not merge, so declaring ONE action imports exactly that
+# action plus delete_account and skips DefaultRequiredActions.addActions entirely --
+# deleting CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing
+# CONFIGURE_TOTP is a hard lock-out of every platform-admin, because auth-otp-form in the
+# browser flow still demands a TOTP they could then never enrol. The comment saying "DO
+# NOT ADD ONE" is advice; this is the gate.
+check("requiredActions" not in realm,
+      f"{IMPORT_FILE} declares spec.realm.requiredActions. Keycloak 26.0.6 treats that "
+      f"as REPLACE, not merge: declaring any action imports exactly those plus "
+      f"delete_account and NEVER runs DefaultRequiredActions.addActions, so "
+      f"CONFIGURE_TOTP, UPDATE_PASSWORD, VERIFY_EMAIL and seven more are DELETED. That "
+      f"is a permanent lock-out of every platform-admin, because auth-otp-form in this "
+      f"realm's browser flow still demands a TOTP they could no longer enrol. The "
+      f"enrolment action is already registered by default (priority 80, enabled, "
+      f"defaultAction false) and the passkey reconciler converges it through the "
+      f"per-alias admin API, which has no such hazard")
+
+# ============================== (2)(3) the two files share one substitution pass
+state_docs = [d for d in load(STATE_FILE) if d]
+state_cm = next((d for d in state_docs if d.get("kind") == "ConfigMap"), None)
+check(state_cm is not None, f"{STATE_FILE}: no ConfigMap")
+desired = {}
+if state_cm is not None:
+    check(state_cm["metadata"]["name"] == STATE_CM_NAME,
+          f"{STATE_FILE}: the ConfigMap is named "
+          f"{state_cm['metadata']['name']!r}, but the CronJob mounts {STATE_CM_NAME!r}")
+    data = state_cm.get("data") or {}
+    check(sorted(data) == sorted([STATE_DATA_KEY, BIND_KEY]),
+          f"{STATE_FILE}: expected exactly the data keys {STATE_DATA_KEY!r} and "
+          f"{BIND_KEY!r}, got {sorted(data)} — the reconciler's PASSKEY_STATE_FILE / "
+          f"PASSKEY_BIND_FILE defaults would not resolve")
+    # THE BIND SWITCH. An unquoted true/false is a YAML BOOLEAN, which a ConfigMap rejects
+    # at apply time -- failing the whole platform-config Kustomization, realm import
+    # included. The renderer refuses it too; asserted here so the reason is on record.
+    check(data.get(BIND_KEY) in ("true", "false"),
+          f"{STATE_FILE}: data.{BIND_KEY} is {data.get(BIND_KEY)!r}; it must be the QUOTED "
+          f"string \"true\" or \"false\"")
+    raw = data.get(STATE_DATA_KEY, "")
+    try:
+        desired = json.loads(raw)
+    except ValueError as e:
+        check(False, f"{STATE_FILE}: the {STATE_DATA_KEY} payload is not valid JSON "
+                     f"({e}) — the reconciler would fail closed on every run")
+
+# Both files must be rendered by ONE Flux Kustomization. substituteFrom is
+# per-Kustomization, so a split lets the same ${BASE_DOMAIN} token resolve to two
+# different values -- and an RP ID mismatch invalidates every enrolled passkey silently.
+import_owners = set()
+state_owners = set()
+recon_owners = set()
+for env in ("dev", "prod"):
+    entry = f"clusters/{env}"
+    if not (ROOT / entry).is_dir():
+        continue
+    applied, _ = st.reconciled_files(ROOT, entry)
+    check(RECON_FILE in applied,
+          f"{RECON_FILE} is not reachable from the {entry} entrypoint — the passkey "
+          f"healer would exist on one cluster only")
+    check(STATE_FILE in applied,
+          f"{STATE_FILE} is not reachable from the {entry} entrypoint — the reconciler "
+          f"would fail closed on that cluster forever")
+    check(IMPORT_FILE in applied, f"{IMPORT_FILE} is not reachable from {entry}")
+
+# Which Kustomization DIRECTORY lists each file. Asserted structurally: both must be
+# listed as resources of the same kustomization.yaml.
+for kpath in ROOT.glob("*/kustomization.yaml"):
+    try:
+        k = yaml.safe_load(kpath.read_text()) or {}
+    except Exception:
+        continue
+    res = [str(r) for r in (k.get("resources") or [])]
+    base = kpath.parent.name
+    if any(r.endswith("keycloak-realm-import.yaml") for r in res):
+        import_owners.add(base)
+    if any(r.endswith("passkey-desired-state.yaml") for r in res):
+        state_owners.add(base)
+    if any(r.endswith("passkey-reconciler.yaml") for r in res):
+        recon_owners.add(base)
+check(import_owners and import_owners == state_owners,
+      f"{IMPORT_FILE} is rendered by {sorted(import_owners)} but {STATE_FILE} by "
+      f"{sorted(state_owners)}. They MUST be the same Flux Kustomization: "
+      f"postBuild.substituteFrom is per-Kustomization, so a split would let the "
+      f"{DOLLAR}{{BASE_DOMAIN}} token in each resolve to a different value. An RP ID is "
+      f"baked into every passkey at enrolment and can never be re-keyed, so a domain the "
+      f"two files disagree about is an unrecoverable credential, not a cosmetic drift")
+check(recon_owners == state_owners,
+      f"{RECON_FILE} is rendered by {sorted(recon_owners)} but its desired state by "
+      f"{sorted(state_owners)} — the reconciler could be applied without its input")
+
+# (3) no substitution token the realm definition does not also use.
+import_text = (ROOT / IMPORT_FILE).read_text()
+TOKEN = re.compile(re.escape(DOLLAR) + r"\{([A-Za-z_][A-Za-z0-9_]*)")
+state_tokens = set(TOKEN.findall((ROOT / STATE_FILE).read_text()))
+import_tokens = set(TOKEN.findall(import_text))
+# The renderer's own HEADER mentions ${BASE_DOMAIN} in prose; that is the same token the
+# policy uses, so a subset check is the right assertion rather than an exact match.
+check(state_tokens <= import_tokens,
+      f"{STATE_FILE} uses substitution token(s) {sorted(state_tokens - import_tokens)} "
+      f"that {IMPORT_FILE} does not. A token the realm definition never uses is either a "
+      f"hand edit or a renderer bug, and an UNDEFINED one renders as the EMPTY STRING "
+      f"silently — which for an RP ID means every passkey ceremony fails")
+
+# ======================================= (4) the two write surfaces are the SAME tuple
+recon_docs = [d for d in load(RECON_FILE) if d]
+cm = next((d for d in recon_docs if d.get("kind") == "ConfigMap"), None)
+cj = next((d for d in recon_docs if d.get("kind") == "CronJob"), None)
+check(cm is not None, f"{RECON_FILE}: no ConfigMap carrying the reconciler script")
+check(cj is not None, f"{RECON_FILE}: no CronJob running the reconciler")
+
+script = ""
+if cm is not None:
+    data = cm.get("data") or {}
+    check(len(data) == 1,
+          f"{RECON_FILE}: the reconciler ConfigMap must carry exactly one script key "
+          f"(found {sorted(data)}) — a second copy would be untested by this gate")
+    check(SCRIPT_KEY in data,
+          f"{RECON_FILE}: expected the script under {SCRIPT_KEY!r}, got {sorted(data)}")
+    script = next(iter(data.values()), "")
+    check(len(script.splitlines()) > 200,
+          f"{RECON_FILE}: the extracted script is implausibly short "
+          f"({len(script.splitlines())} lines) — the ConfigMap shape changed and the "
+          f"selftest below would prove nothing")
+    # THE nextcloud TRAP. This file is rendered by a Flux Kustomization with
+    # postBuild.substituteFrom, so envsubst rewrites it before any container runs.
+    check(DOLLAR not in script,
+          f"{RECON_FILE}: the reconciler script contains a dollar sign. It ships inside "
+          f"a SUBSTITUTED Flux path, where a dollar-brace expansion is rewritten at "
+          f"reconcile time (see apps/wave-1/nextcloud.yaml). Rewrite without it — Python "
+          f"never needs one; chr(36) spells it where a literal is genuinely required")
+
+
+def tuple_literal(src, name):
+    """The value of a module-level tuple assignment, via ast. Parsing the SOURCE rather
+    than importing it keeps this check offline and side-effect free, and means the thing
+    asserted is the text that ships."""
+    try:
+        tree = ast.parse(src)
+    except SyntaxError:
+        return None
+    for node in tree.body:
+        if isinstance(node, ast.Assign):
+            for t in node.targets:
+                if isinstance(t, ast.Name) and t.id == name:
+                    try:
+                        return ast.literal_eval(node.value)
+                    except ValueError:
+                        return None
+    return None
+
+
+renderer_src = (ROOT / RENDERER).read_text()
+
+
+# POLICY_FIELDS is a CONCATENATION of other tuples in both files, so literal_eval of the
+# assignment itself cannot evaluate it. Resolve it from its parts instead. Both files name
+# their parts from the same vocabulary; a part present in one and not the other simply
+# contributes nothing, which is why the comparison below is of the resolved SETS.
+def resolve_policy(src):
+    parts = ("POLICY_STR_FIELDS", "POLICY_ORDERED_LIST_FIELDS",
+             "POLICY_UNORDERED_LIST_FIELDS", "POLICY_LIST_FIELDS", "POLICY_INT_FIELDS",
+             "POLICY_BOOL_FIELDS")
+    out = []
+    for p in parts:
+        v = tuple_literal(src, p)
+        if v:
+            out.extend(v)
+    return sorted(set(out))
+
+
+recon_keys = resolve_policy(script)
+rend_keys = resolve_policy(renderer_src)
+check(recon_keys and recon_keys == rend_keys,
+      f"the reconciler's writable policy keys {recon_keys} and the renderer's exported "
+      f"keys {rend_keys} DISAGREE. If the renderer emits a key the reconciler refuses, "
+      f"every run fails closed forever; if the reconciler accepts a key the renderer "
+      f"never emits, that is dead write surface on the identity provider")
+# Every exported key must actually be a passwordless key. The two-factor
+# webAuthnPolicy* family is a silent no-op for passkeys (the passwordless authenticator
+# reads getWebAuthnPolicyPasswordless() only), so exporting one would be a write surface
+# with no effect.
+check(all(k.startswith("webAuthnPolicyPasswordless") for k in recon_keys),
+      f"the reconciler's writable keys include a non-passwordless field: "
+      f"{[k for k in recon_keys if not k.startswith('webAuthnPolicyPasswordless')]}. "
+      f"The two-factor webAuthnPolicy* family is never consulted by a passkey login, so "
+      f"writing it is a silent no-op")
+# And the desired state must declare only those.
+if isinstance(desired.get("webAuthnPolicyPasswordless"), dict):
+    extra = sorted(set(desired["webAuthnPolicyPasswordless"]) - set(recon_keys))
+    check(not extra,
+          f"{STATE_FILE} declares policy key(s) {extra} outside the reconciler's write "
+          f"surface — it would fail closed on every run")
+    # The 26.0.6 gate: the policy is applied ONLY when RpEntityName is non-empty.
+    check(desired["webAuthnPolicyPasswordless"].get(
+              "webAuthnPolicyPasswordlessRpEntityName"),
+          f"{STATE_FILE} carries no non-empty "
+          f"webAuthnPolicyPasswordlessRpEntityName. Keycloak 26.0.6 applies the "
+          f"passwordless policy only when that field is set "
+          f"(DefaultExportImportManager.updateRealm), so every realm PUT would be "
+          f"accepted with 204 and change nothing")
+
+# THE ENROLMENT POSTURE. defaultAction true would force every user to enrol a passkey at
+# next login -- a platform-wide lock-out risk, and the exact opposite of the realm
+# import's stated "additive and optional" decision. It is one boolean, so it is gated.
+ra = desired.get("requiredAction") or {}
+check(ra.get("alias") == ENROL_ACTION,
+      f"{STATE_FILE}: requiredAction.alias is {ra.get('alias')!r}, expected "
+      f"{ENROL_ACTION!r} — the reconciler's allow-list names that alias literally, so "
+      f"any other value is unreachable and the pass would fail closed")
+check(ra.get("enabled") is True,
+      f"{STATE_FILE}: requiredAction.enabled must be true, else passkey ENROLMENT is "
+      f"impossible while the login flow offers passkeys")
+check(ra.get("defaultAction") is False,
+      f"{STATE_FILE}: requiredAction.defaultAction is {ra.get('defaultAction')!r}, which "
+      f"must be false. True forces EVERY user to enrol a passkey at next login — this "
+      f"IdP gates Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the "
+      f"*arr stack AND kubectl, so a misfiring mandatory enrolment is a total platform "
+      f"lock-out. The realm import's whole stated posture is additive and OPTIONAL")
+
+# ========================================= (5) the reconciler manifest is safe + wired
+if cj is not None:
+    pod = cj["spec"]["jobTemplate"]["spec"]["template"]["spec"]
+    ctr = pod["containers"][0]
+    vols = {v["name"]: v for v in pod.get("volumes", [])}
+    mounts = {m["name"]: m for m in ctr.get("volumeMounts", [])}
+    env = {e["name"]: e for e in ctr.get("env", [])}
+
+    # NO KUBERNETES API ACCESS. This is what makes the mounted-ConfigMap design strictly
+    # narrower than reading the KeycloakRealmImport CR through the API: no Role, no
+    # RoleBinding, no token. It is also why this change adds no **/rbac.yaml.
+    check(pod.get("automountServiceAccountToken") is False,
+          f"{RECON_FILE}: the reconciler must set automountServiceAccountToken: false. "
+          f"It talks only to the Keycloak admin API and reads a mounted ConfigMap, so a "
+          f"ServiceAccount token is pure blast radius (R-SEC-02)")
+    check("serviceAccountName" not in pod,
+          f"{RECON_FILE}: the reconciler must not name a ServiceAccount")
+
+    cm_vols = {n: v for n, v in vols.items() if "configMap" in v}
+    names = {(v["configMap"] or {}).get("name") for v in cm_vols.values()}
+    check(STATE_CM_NAME in names,
+          f"{RECON_FILE}: the CronJob does not mount the {STATE_CM_NAME} ConfigMap, so "
+          f"the reconciler would fail closed on every run. Mounted: {sorted(names)}")
+    check(cm["metadata"]["name"] in names,
+          f"{RECON_FILE}: the CronJob does not mount its own script ConfigMap")
+    # NO SECRET IS MOUNTED AS A VOLUME. The admin credential is consumed via
+    # secretKeyRef only; projecting it as a file would widen what a readable mount leaks.
+    check(not [v for v in vols.values() if "secret" in v],
+          f"{RECON_FILE}: the reconciler mounts a Secret as a volume. The admin "
+          f"credential is referenced by secretKeyRef only; it must never be projected "
+          f"as a file here")
+
+    # The mount paths must be the ones the code DEFAULTS to, or the container starts and
+    # fails closed forever while every manifest looks correct.
+    state_vol = next((n for n, v in cm_vols.items()
+                      if (v["configMap"] or {}).get("name") == STATE_CM_NAME), None)
+    if state_vol:
+        mp = (mounts.get(state_vol) or {}).get("mountPath")
+        check(mp == "/etc/keycloak-passkey",
+              f"{RECON_FILE}: the desired-state ConfigMap is mounted at {mp!r}, but the "
+              f"script's PASSKEY_STATE_FILE default is "
+              f"/etc/keycloak-passkey/{STATE_DATA_KEY}")
+        check(f'"/etc/keycloak-passkey/{STATE_DATA_KEY}"' in script,
+              f"{RECON_FILE}: the script's PASSKEY_STATE_FILE default no longer matches "
+              f"the mount path — they are a pair and must move together")
+        check((mounts.get(state_vol) or {}).get("readOnly") is True,
+              f"{RECON_FILE}: the desired-state mount must be readOnly")
+        check(f'"/etc/keycloak-passkey/{BIND_KEY}"' in script,
+              f"{RECON_FILE}: the script's PASSKEY_BIND_FILE default is not "
+              f"/etc/keycloak-passkey/{BIND_KEY}. An unresolved bind file reads as \"false\" "
+              f"(safe, but git's \"true\" would then silently never take effect)")
+        check(not (state_vol and (cm_vols[state_vol]["configMap"] or {}).get("items")),
+              f"{RECON_FILE}: the desired-state volume selects `items`, so one of its two "
+              f"keys may not be projected")
+
+    # The bind decision belongs to git. A container arg could only NARROW it today
+    # (--no-bind), but nothing in the CronJob may carry a mode flag at all: the shipped
+    # behaviour must be readable from passkey-desired-state.yaml alone.
+    cmd = [str(x) for x in (ctr.get("command") or []) + (ctr.get("args") or [])]
+    check(not [x for x in cmd if x.startswith("--")],
+          f"{RECON_FILE}: the CronJob passes {[x for x in cmd if x.startswith('--')]} to "
+          f"the reconciler. Its mode is decided by git (bind-browser-flow), not by flags")
+    # Retries burn the bounded candidate-alias budget on a deterministic failure.
+    check(cj["spec"]["jobTemplate"]["spec"].get("backoffLimit") == 0,
+          f"{RECON_FILE}: backoffLimit must be 0 — every failure here is deterministic "
+          f"within a tick, and each retried build consumes another candidate alias")
+
+    # The admin credential, referenced BY NAME. Never decrypted, never a literal.
+    for key in ("KEYCLOAK_ADMIN_USER", "KEYCLOAK_ADMIN_PASSWORD"):
+        e = env.get(key) or {}
+        ref = ((e.get("valueFrom") or {}).get("secretKeyRef") or {})
+        check(ref.get("name") == "keycloak-admin-credentials",
+              f"{RECON_FILE}: {key} must come from the keycloak-admin-credentials "
+              f"Secret by secretKeyRef, got {e!r}")
+        check("value" not in e,
+              f"{RECON_FILE}: {key} carries a literal `value`. A credential in a "
+              f"manifest is a plaintext secret in git")
+    check((env.get("KEYCLOAK_REALM") or {}).get("value") == desired.get("realm"),
+          f"{RECON_FILE}: KEYCLOAK_REALM is "
+          f"{(env.get('KEYCLOAK_REALM') or {}).get('value')!r} but the desired state "
+          f"declares realm {desired.get('realm')!r}. The reconciler would converge the "
+          f"wrong realm, or none")
+
+    check(ctr["securityContext"]["readOnlyRootFilesystem"] is True,
+          f"{RECON_FILE}: readOnlyRootFilesystem must be true")
+    check(ctr["securityContext"]["allowPrivilegeEscalation"] is False,
+          f"{RECON_FILE}: allowPrivilegeEscalation must be false")
+    check(pod["securityContext"].get("runAsNonRoot") is True,
+          f"{RECON_FILE}: runAsNonRoot must be true")
+    check(cj["spec"].get("concurrencyPolicy") == "Forbid",
+          f"{RECON_FILE}: concurrencyPolicy must be Forbid — two concurrent reconcilers "
+          f"would double the write attempts against the IdP during an outage recovery")
+    check(re.fullmatch(r"\S+ \S+ \S+ \S+ \S+", cj["spec"].get("schedule", "")),
+          f"{RECON_FILE}: schedule {cj['spec'].get('schedule')!r} is not a 5-field cron")
+
+# ===== (6) THE BOUND FLOW IS STRUCTURALLY UNWRITABLE — asserted against the SOURCE
+#
+# This component converges the browser flow BLUE/GREEN: it builds the target structure
+# under a NEW, unused alias, verifies it, then flips the realm's browserFlow in ONE
+# atomic PUT. It therefore NEEDS flow-create and execution-add endpoints, so unlike the
+# first design of this check the mere presence of a flow write is no longer the thing to
+# forbid. What must hold instead is narrower and stronger:
+#
+#   (a) nothing may be DELETED, ever — the superseded flow is the rollback target;
+#   (b) the single-execution and config-edit endpoints stay off the list — requirements
+#       are set through the flow-scoped endpoint, which the guard in (d) screens;
+#   (c) no required-action write may reach any alias but the one named LITERALLY;
+#   (d) and the guard the allow-list CANNOT express must exist in the source: a
+#       method-aware screen that refuses any flow write targeting the CURRENTLY BOUND
+#       flow's subtree, FAIL-CLOSED until that binding has been read.
+#
+# (d) is the load-bearing one. Without it the reconciler could mutate the flow users are
+# logging in through, and a run dying mid-restructure would leave a HALF-BUILT FLOW BOUND
+# AS THE LOGIN PATH for Grafana, Gitea, Nextcloud, Headlamp, Semaphore, PiHole, Crafty,
+# the *arr stack AND kubectl. This repo has already had a total SSO outage from a realm
+# change (2026-09-06).
+allow = tuple_literal(script, "_ALLOWED") or ()
+check(bool(allow),
+      f"{RECON_FILE}: could not parse the _ALLOWED endpoint allow-list out of the "
+      f"script. This check's central assertions about the write surface cannot be made, "
+      f"so it must FAIL rather than pass silently")
+if allow:
+    check(all(isinstance(e, tuple) and len(e) == 2 for e in allow),
+          f"{RECON_FILE}: _ALLOWED is not a tuple of (method, path) pairs. A path-only "
+          f"allow-list cannot distinguish reading a flow from writing one, which this "
+          f"component's whole safety argument rests on")
+    writes = [(m, p) for m, p in allow if m in ("PUT", "POST", "PATCH", "DELETE")]
+    # (a) nothing is ever deleted.
+    check(not [(m, p) for m, p in allow if m == "DELETE"],
+          f"{RECON_FILE}: the allow-list contains a DELETE method. Nothing this program "
+          f"does requires deleting anything: a superseded flow is left unbound as the "
+          f"ROLLBACK TARGET, and DELETE /admin/realms/<realm> deletes the realm")
+    # (b) the endpoints that would let it edit live objects in place.
+    banned = [(m, p) for m, p in writes
+              if re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/executions/"
+                              r"\[\^/\]\+", p)
+              or re.fullmatch(r"/admin/realms/\[\^/\]\+/authentication/config/\[\^/\]\+",
+                              p)
+              or p.endswith("/authentication/register-required-action")]
+    check(not banned,
+          f"{RECON_FILE}: the allow-list permits {banned}. The single-execution update "
+          f"and config-edit endpoints must stay off it: requirements are set through the "
+          f"FLOW-SCOPED endpoint, which carries the flow alias in its path and is "
+          f"therefore screenable against the bound subtree. An execution- or "
+          f"config-id-scoped write is not, so it would be an unscreened hole")
+    # (c) no required-action write may reach any other alias.
+    ra_writes = [(m, p) for m, p in writes if "required-actions" in p]
+    check(all(p.endswith("required-actions/" + ENROL_ACTION) for _, p in ra_writes),
+          f"{RECON_FILE}: the allow-list permits a required-action write to an alias "
+          f"other than {ENROL_ACTION!r}: {ra_writes}. Disabling CONFIGURE_TOTP is a "
+          f"permanent lock-out of every platform-admin, because auth-otp-form in the "
+          f"browser flow still demands a TOTP that could no longer be enrolled")
+    check(not [p for _, p in ra_writes if "[^/]+" in p.split("required-actions/")[-1]],
+          f"{RECON_FILE}: a required-action write path uses a WILDCARD alias. The alias "
+          f"must be named LITERALLY so no other action is writable by construction")
+    # Nothing outside the realm + authentication surface at all.
+    stray = [(m, p) for m, p in allow
+             if "/users" in p or "/groups" in p or "/roles" in p or "/clients" in p]
+    check(not stray,
+          f"{RECON_FILE}: the allow-list reaches {stray}. Users, groups and roles must "
+          f"be unreachable (prod's real user and the media-users / platform-admins / "
+          f"viewers groups must survive every run); clients are the SIBLING "
+          f"reconciler's surface and the two must not overlap")
+
+# ---- (d) THE PROTECTED-SUBTREE GUARD EXISTS, IS WIRED, AND IS FAIL-CLOSED ----------
+# Asserted against the SOURCE, because this is the property the blue/green design rests
+# on and a comment claiming it is worth nothing. Three things must all be true.
+check(re.search(r"^def _screen_write\(", script, re.M) is not None,
+      f"{RECON_FILE}: the script has no _screen_write() function. That is the guard "
+      f"that refuses any flow write targeting the CURRENTLY BOUND flow's subtree — the "
+      f"one thing the allow-list cannot express, because its write entries are "
+      f"alias-parameterised. Without it this component can mutate the live login path")
+# It must be CALLED from the single choke point, not merely defined.
+check(re.search(r"_screen_write\(method, path", script) is not None,
+      f"{RECON_FILE}: _screen_write() is defined but never called from _http() with the "
+      f"request's method and path. A guard that is not on the request path is decoration")
+# FAIL-CLOSED: the protected sets must start undetermined, and the reset that runs at
+# the top of every run must put them back there.
+reset_fn = re.search(r"def _reset_protected\(\):(.*?)(?=\n\S)", script, re.S)
+check(reset_fn is not None
+      and "_PROTECTED_FLOW_ALIASES = None" in reset_fn.group(1)
+      and "_PROTECTED_EXECUTION_IDS = None" in reset_fn.group(1),
+      f"{RECON_FILE}: _reset_protected() does not set both protected sets to None. They "
+      f"must start each run UNDETERMINED, because while they are, every flow write is "
+      f"refused — that is the fail-closed half of the guard, and it is what stops a "
+      f"build running before the live binding has been read")
+check(re.search(r"_reset_protected\(\)", script.split("def _selftest")[0]) is not None,
+      f"{RECON_FILE}: _reset_protected() is never called outside the selftest, so the "
+      f"protected sets are not reset per run and a previous run's binding could be "
+      f"trusted for this one")
+# The create-flow endpoint carries NO alias in its path, so it must be screened on its
+# BODY or it escapes the guard entirely. (Found by the selftest; recorded here so a
+# future edit cannot silently undo it.)
+check("_FLOW_CREATE_PATH" in script,
+      f"{RECON_FILE}: there is no separate screen for POST .../authentication/flows. "
+      f"That endpoint carries no alias in its path — the alias is in the request BODY — "
+      f"so without a body-based screen it bypasses the protected-subtree guard "
+      f"altogether")
+# And the one function allowed to change the live binding must be distinct from the
+# policy write, which must refuse to.
+# FINDING 2026-10-29: sub-flows are named by displayName in 26.0.6 (recurseExecutions
+# never sets `alias` on a sub-flow row). Reading `alias` shrank the protected subtree to
+# the top-level flow alone while every selftest passed against a stub that returned it.
+check(re.search(r'^def subflow_alias\(ex\):\n(?:.*\n)*?    name = ex\.get\("displayName"\)',
+                script, re.M) is not None,
+      f"{RECON_FILE}: subflow_alias() no longer reads displayName. Keycloak 26.0.6's "
+      f"executions listing names a sub-flow ONLY by displayName; reading `alias` yields "
+      f"None and the protected subtree silently shrinks to the top-level flow")
+# FINDING 2026-10-29: the requirement PUT's real target is body["id"].
+check("_EXEC_UPDATE_PATH" in script and "_BUILD_EXECUTION_IDS" in script,
+      f"{RECON_FILE}: the requirement PUT (PUT /flows/<alias>/executions) is no longer "
+      f"screened on its body id. updateExecutions resolves the execution realm-wide by id "
+      f"and ignores the path alias, so a path-only screen lets it write the bound flow")
+check("def declared_bind(" in script and "BIND_FILE" in script,
+      f"{RECON_FILE}: the reconciler no longer reads git's bind-browser-flow switch")
+check("def bind_browser_flow(" in script,
+      f"{RECON_FILE}: there is no bind_browser_flow() function. The atomic flip must be "
+      f"one named, auditable operation, not an incidental field in some other write")
+check(re.search(r"never rebinds the browser flow", script) is not None,
+      f"{RECON_FILE}: the policy write no longer refuses to change browserFlow. "
+      f"PUT /admin/realms/<realm> DOES read rep.getBrowserFlow(), so the policy pass "
+      f"can rebind the login path by accident — to an UNVERIFIED flow")
+
+# ======================================== (7) the SIBLING reconciler is NOT widened
+if (ROOT / SIBLING_FILE).is_file():
+    sib_docs = [d for d in load(SIBLING_FILE) if d]
+    sib_cm = next((d for d in sib_docs if d.get("kind") == "ConfigMap"), None)
+    sib_script = next(iter((sib_cm.get("data") or {}).values()), "") if sib_cm else ""
+    sib_allow = tuple_literal(sib_script, "_ALLOWED") or ()
+    check(bool(sib_allow),
+          f"{SIBLING_FILE}: could not parse its _ALLOWED list, so this check cannot "
+          f"prove the sibling was not widened")
+    if sib_allow:
+        # The sibling's list is PATH-ONLY by design. Its selftest asserts it refuses the
+        # whole-realm endpoint and the /authentication tree; assert the same here so a
+        # future edit that "just adds the realm endpoint there instead" is caught.
+        bad = [p for p in sib_allow
+               if isinstance(p, str)
+               and (re.fullmatch(r"/admin/realms/\[\^/\]\+", p)
+                    or "authentication" in p or "/users" in p or "/groups" in p
+                    or "/roles" in p)]
+        check(not bad,
+              f"{SIBLING_FILE}: its allow-list was WIDENED to {bad}. That reconciler's "
+              f"stated guarantee — prod's real user and the media-users / "
+              f"platform-admins / viewers groups survive every run — is expressed as "
+              f"endpoints it CANNOT reach, and its own --selftest asserts the refusal of "
+              f"/users, /groups, /roles, /authentication and the whole-realm endpoint. "
+              f"The passkey work is a SIBLING program with its own narrow list precisely "
+              f"so that guarantee stays literally true. Revert the widening")
+
+# ============================== (8) the reconciler's BEHAVIOUR, proven by its selftest
+if script:
+    with tempfile.TemporaryDirectory() as td:
+        p = Path(td) / "reconcile.py"
+        p.write_text(script)
+        proc = subprocess.run([sys.executable, str(p), "--selftest"],
+                              capture_output=True, text=True, timeout=300)
+        tail = (proc.stdout + proc.stderr).strip().splitlines()
+        check(proc.returncode == 0,
+              "the passkey reconciler's --selftest FAILED (compare-then-write / "
+              "idempotence / strict read-modify-write / the RpEntityName gate / "
+              "flow-is-unwritable / the conditional-OTP regression / fail-closed):\n"
+              "        " + "\n        ".join(tail[-14:]))
+        if proc.returncode == 0:
+            print("  passkey reconciler --selftest: " + (tail[-1] if tail else "(none)"))
+
+        # ---- (8b) THE SELFTEST IS FALSIFIABLE, not decorative ----------------------
+        # A passing suite proves nothing unless it can be shown to FAIL. mutations.py
+        # injects plausible regressions one at a time -- a partial policy PUT, a
+        # rebind of browserFlow, a widened allow-list that makes the flow writable, a
+        # fail-open on an unreadable input -- and requires --selftest to catch every
+        # one. This matters more than usual here: the whole incident class is a
+        # component that REPORTED SUCCESS while the realm was broken, and a green test
+        # that cannot fail is the same lie one level up.
+        mut = DIR_PATH / "mutations.py"
+        if mut.is_file():
+            mproc = subprocess.run([sys.executable, str(mut), str(p)],
+                                   capture_output=True, text=True, timeout=900)
+            mtail = (mproc.stdout + mproc.stderr).strip().splitlines()
+            check(mproc.returncode == 0,
+                  "the passkey reconciler's --selftest is NOT FALSIFIABLE — at least "
+                  "one injected regression survived it, so the suite does not prove "
+                  "what it claims:\n        " + "\n        ".join(mtail[-16:]))
+            if mproc.returncode == 0:
+                print("  passkey reconciler mutations: "
+                      + (mtail[-1] if mtail else "(no output)"))
+        else:
+            check(False,
+                  "conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py "
+                  "is missing — nothing proves the --selftest can fail, and an "
+                  "unfalsifiable proof is the failure mode this component exists to end")
+
+# ======================================================= (9) this check is load-bearing
+L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
+GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
+TOUCHED = ((ROOT / "ci" / "touched-requirements.sh").read_text()
+           if (ROOT / "ci" / "touched-requirements.sh").is_file() else "")
+SELF_YAML = DIR_PATH / "chainsaw-test.yaml"
+SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
+check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M)
+      is not None,
+      "keycloak-passkey-reconciler/chainsaw-test.yaml does not declare "
+      'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check '
+      "only when a change happens to attribute one of the requirements it is labelled "
+      "with. This invariant spans platform/, platform-config/, ci/ and clusters/, so a "
+      "change that breaks it need not attribute R-IDP-03/04 — which is how 12 unusable "
+      "prod OIDC clients sat behind an entirely green cluster on 2026-09-06")
+check("--include-scope" in L0,
+      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 "
+      "stage is a purely requirement-scoped run again and the scope annotation is inert "
+      "decoration — this check would then run only by coincidence")
+check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)
+      and re.search(r"fail\).*BLOCK=1", GATE),
+      "ci/gate.sh does not set the merge block when the l0 stage fails")
+check(re.search(r"^\s*platform/keycloak/\*\)", TOUCHED, re.M),
+      "ci/touched-requirements.sh has no platform/keycloak/* arm, so editing the realm "
+      "import or the reconciler would attribute NO requirement and the keycloak tests "
+      "would be filtered out of the merge gate's touched subset")
+
+# =============================================================================
+if failures:
+    print("CHECK FAIL: %d assertion(s) failed" % len(failures), file=sys.stderr)
+    for f in failures:
+        print("  FAIL %s" % f, file=sys.stderr)
+    raise SystemExit(1)
+print("keycloak-passkey-reconciler: all assertions passed "
+      "(desired state derived from the realm import, one substitution pass, write "
+      "surfaces agree, the BOUND flow is unwritable and fail-closed, nothing is "
+      "deletable, sibling allow-list unwidened, selftest green and falsifiable)")
+PY

conformance/tests/l0-static/keycloak-passkey-reconciler/mutations.py

@@ -0,0 +1,456 @@
+#!/usr/bin/env python3
+# =============================================================================
+# mutations.py -- prove the passkey reconciler's --selftest IS FALSIFIABLE.
+#
+# WHY THIS EXISTS
+# ---------------
+# A selftest that passes proves nothing on its own: a selftest asserting `True ==
+# True` also passes. This reconciler exists because a one-shot realm import
+# REPORTED SUCCESS while the realm was never converged (Done=True / Started=False
+# with the whole passkey block inert, measured on prod 2026-10-18), and the same
+# trap applies one level up -- a green test suite that cannot fail is the same lie
+# in a different place.
+#
+# It matters more here than for the sibling client reconciler, because this program
+# CONSTRUCTS AUTHENTICATION FLOWS and then REBINDS THE REALM'S LOGIN PATH. An
+# untested safety property on that surface is not a documentation gap, it is an
+# outage waiting for a schedule tick.
+#
+# So this injects a series of PLAUSIBLE regressions into the shipped script, one at
+# a time, and requires that --selftest FAILS for every one. A mutation that
+# SURVIVES means the suite does not actually check that property, and the check
+# fails, naming it.
+#
+# THE FIRST SIX ARE THE BLUE/GREEN SAFETY ARGUMENT ITSELF. The design builds a new
+# flow under an unused alias, verifies it, and flips `browserFlow` in one atomic
+# write -- never mutating the flow users are logging in through. Each of those
+# properties is enforced in code; these mutations are the proof that the enforcement
+# is load-bearing rather than commentary. Several model the obvious "simplification"
+# someone will propose:
+#   * "just build it under the declared alias, the suffix is ugly"
+#   * "bind it first, then check -- the check is just belt and braces"
+#   * "the protected-alias screen is redundant, the allow-list covers it"
+# Every one of those would be a total SSO outage waiting to happen.
+#
+# ANCHORS ARE AUTHORED AGAINST THE EXTRACTED SCRIPT, not against the YAML manifest.
+# The sibling mutations.py dedents by four because its anchors are copied out of the
+# ConfigMap block scalar; these are copied out of the script as `python3 -` sees it,
+# which is one less transformation to get wrong. check.sh passes the extracted file,
+# so what is proven falsifiable is byte-for-byte what runs on the cluster either way.
+#
+#   python3 mutations.py <path-to-extracted-reconciler.py>
+#
+# Runtime is ~60s (one subprocess per mutation, each running the full selftest
+# against an in-process stub Keycloak; no network, no cluster).
+# =============================================================================
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+# (name, the mistake being modelled, anchor, replacement)
+MUTATIONS: list[tuple[str, str, str, str]] = [
+    # ---------------------------------------------------------------- blue/green safety
+    ("build INTO the currently bound flow",
+     "'the suffix is ugly, just build under the declared alias' -- every "
+     "create-execution and set-requirement call would then land on the flow users are "
+     "logging in through, and a run dying part-way leaves a HALF-BUILT LOGIN PATH for "
+     "every SSO app and kubectl. This is the single regression the whole design exists "
+     "to make impossible",
+     "    def mapped(alias):\n"
+     "        return \"%s-%s\" % (alias, suffix)",
+     "    def mapped(alias):\n        return alias"),
+
+    ("the protected-subtree screen is never called",
+     "'the allow-list already covers it' -- it does not: the allow-list's write entries "
+     "are alias-parameterised, so only this screen can say 'but not the flow that is "
+     "bound'. Without it the reconciler may write anywhere in the authentication tree",
+     "    _screen_write(method, path, body if isinstance(body, dict) else None)",
+     "    pass"),
+
+    ("the per-run reset makes the protected sets EMPTY, not UNDETERMINED",
+     "reset them to frozenset() so the code reads more simply -- it silently removes "
+     "the fail-closed property, so a build that ran before the live binding had been "
+     "read would be PERMITTED, and could write straight into the live flow. (The "
+     "module-level initial value is deliberately NOT the anchor here: run() resets "
+     "these at the start of every run, so the initialiser is unreachable and mutating "
+     "it would prove nothing.)",
+     "    global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
+     "    _PROTECTED_FLOW_ALIASES = None\n"
+     "    _PROTECTED_EXECUTION_IDS = None",
+     "    global _PROTECTED_FLOW_ALIASES, _PROTECTED_EXECUTION_IDS, _BUILD_EXECUTION_IDS\n"
+     "    _PROTECTED_FLOW_ALIASES = frozenset()\n"
+     "    _PROTECTED_EXECUTION_IDS = frozenset()"),
+
+    ("bind BEFORE verifying the build",
+     "'bind it first, the readback is just belt and braces' -- it is not: the readback "
+     "is what catches a build that was accepted but came out wrong, including the "
+     "conditional-OTP config landing under the wrong key. Binding first makes that the "
+     "live login path before anyone notices",
+     "    verify = compare_shape(final_shape, expected, base, realm, token, configs,\n"
+     "                           \"the newly built flow %r\" % target)",
+     "    verify = []"),
+
+    ("a failed build is reported as SUCCESS",
+     "swallow the build error and return a converged result -- the realm is then left "
+     "on the old flow (which blue/green makes safe) but the run says it converged, so "
+     "nobody ever learns the passkey flow was never built. That is the one-shot "
+     "import's fail-open, reintroduced. NOTE: simply removing the early return here "
+     "is NOT a regression -- verify-before-bind catches a half-built flow anyway, which "
+     "is defence in depth working as intended -- so the mutation targets the reporting",
+     "            problems.append(\n"
+     "                \"flow: building %r FAILED (%s). Nothing was bound, so login is \"\n"
+     "                \"UNAFFECTED -- the old flow is still live. The partial flow is left in \"\n"
+     "                \"place, unbound and harmless\" % (candidate, e))\n"
+     "            return 0, problems",
+     "            return 1, []"),
+
+    ("an existing candidate flow is reused WITHOUT verifying it",
+     "'if the alias exists we already built it' -- a half-built leftover from a crashed "
+     "run lives under exactly that alias, so this binds a partially constructed login "
+     "path",
+     "            if cand_shape is not None and not compare_shape(\n"
+     "                    cand_shape, expected, base, realm, token, configs,\n"
+     "                    \"the existing flow %r\" % candidate):",
+     "            if cand_shape is not None:"),
+
+    # ------------------------------------- 2026-10-29 review findings, each made falsifiable
+    ("sub-flows identified by `alias` again",
+     "read `alias` off a sub-flow row, as the first version did -- Keycloak 26.0.6's "
+     "recurseExecutions sets only displayName (and flowId) on a sub-flow row, so every "
+     "sub-flow resolves to nothing and the protected subtree is the top-level flow alone: "
+     "a write into browser-mfa-forms would NOT be refused",
+     "    name = ex.get(\"displayName\")\n"
+     "    return name if isinstance(name, str) and name else None",
+     "    name = ex.get(\"alias\")\n"
+     "    return name if isinstance(name, str) and name else None"),
+
+    ("an unnameable sub-flow is tolerated",
+     "carry on when a sub-flow row has no displayName -- the protected subtree is then "
+     "SMALLER than the live tree, which is the fail-open the guard exists to prevent",
+     "            if name is None:\n"
+     "                return None, None",
+     "            if name is None:\n"
+     "                name = \"\""),
+
+    ("the requirement PUT is screened on its path only",
+     "trust the alias in PUT /flows/<alias>/executions -- updateExecutions resolves "
+     "body.id realm-wide and never checks it belongs to <alias>, so a candidate alias in "
+     "the path can carry a BOUND execution's id in the body",
+     "        targets.append((\"execution\", (body or {}).get(\"id\")))",
+     "        pass"),
+
+    ("execution-addressed writes are not held to the build tree",
+     "'outside the bound subtree is enough' -- it is not: every OTHER flow in the realm is "
+     "outside it too, so a stray id would edit an unrelated flow",
+     "        if what == \"execution\" and (_BUILD_EXECUTION_IDS is None\n"
+     "                                    or target not in _BUILD_EXECUTION_IDS):",
+     "        if False:"),
+
+    ("the requirement PUT may carry any key",
+     "let the body carry flowId / displayName -- the same handler then RENAMES whatever "
+     "flow flowId names, bound or not",
+     "        if extra:\n"
+     "            raise Refused(",
+     "        if False:\n"
+     "            raise Refused("),
+
+    ("git's bind switch is ignored",
+     "bind whenever verification passes, as the first version did -- the first live use of "
+     "an untested login flow becomes a schedule tick, and an operator's rollback is "
+     "undone within 15 minutes, forever",
+     "    bind = git_bind and not no_bind",
+     "    bind = not no_bind"),
+
+    ("bind=false still binds inside the flow pass",
+     "keep the flag but forget to honour it where the bind happens",
+     "    if not bind:\n"
+     "        # THE BIND SWITCH IS OFF.",
+     "    if False:\n"
+     "        # THE BIND SWITCH IS OFF."),
+
+    ("an absent bind switch means bind",
+     "default to true when the key is missing -- a ConfigMap predating the switch, or a "
+     "deleted line, would silently put an untested flow live",
+     "    except FileNotFoundError:\n"
+     "        return False",
+     "    except FileNotFoundError:\n"
+     "        return True"),
+
+    ("an invalid bind switch is guessed at",
+     "read anything that is not \"true\" as false -- a typo then passes silently instead of "
+     "stopping the run, and the operator believes the flow is live",
+     "    if raw == \"false\":\n"
+     "        return False\n"
+     "    raise ValueError(",
+     "    return False\n"
+     "    raise ValueError("),
+
+    ("bind=false rebuilds beside a hand-edited bound flow",
+     "treat the hand-edited bound candidate as a 'half-built leftover' -- with the switch "
+     "off this builds an -r2 next to the operator's hotfix on every tick",
+     "        if candidate in have and candidate == bound and not bind:",
+     "        if False:"),
+
+    # ------------------------------------------------------------------- idempotence
+    ("sub-flow aliases are compared as names",
+     "compare the sub-flow alias like any other field -- the flows this program builds "
+     "carry a hash suffix on every alias, so a built flow could NEVER be recognised as "
+     "converged: it would be rebuilt and the login path reflipped every fifteen "
+     "minutes, forever",
+     "            out.append((level, ex[\"kind\"], \"\" if is_flow else ex[\"name\"],\n"
+     "                        ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))",
+     "            out.append((level, ex[\"kind\"], ex[\"name\"],\n"
+     "                        ex[\"requirement\"], ex.get(\"configAlias\") or \"\", ex[\"name\"]))"),
+
+    ("the converged short-circuit is removed",
+     "drop the 'bound flow already matches git' early return -- every run rebuilds and "
+     "reflips, which is a permanent write loop against the IdP and churns the realm's "
+     "login path on a schedule",
+     "        if not drift:\n"
+     "            log(\"flow     ok (the bound flow %r matches git: %d execution(s), no \"\n"
+     "                \"write)\" % (bound, len(expected)))\n"
+     "            return 0, problems",
+     "        if False:\n            return 0, problems"),
+
+    ("the execution priority is dropped from the requirement write",
+     "omit `priority` from the body -- Keycloak's updateExecutions deserialises a "
+     "missing int as 0 and compares it against the stored priority, so this RESHUFFLES "
+     "the execution order while only a requirement change was intended. The passkey "
+     "branch would end up in the wrong place in the tree",
+     "            body={\"id\": ex.get(\"id\"), \"requirement\": want_req,\n"
+     "                  \"priority\": ex.get(\"priority\") or 0})",
+     "            body={\"id\": ex.get(\"id\"), \"requirement\": want_req})"),
+
+    ("the alias budget is unbounded",
+     "loop forever looking for a free alias -- a persistently failing build then creates "
+     "unbounded flow sprawl in the realm, and nothing here can delete any of it",
+     "    for attempt in range(1, MAX_BUILD_ATTEMPTS + 1):",
+     "    for attempt in range(1, 10000):"),
+
+    ("fail-open when the bound flow cannot be read",
+     "carry on when the live flow's executions are unreadable -- the program then cannot "
+     "know which flows are live, so it cannot prove a build would miss them, and it "
+     "would build with the protected sets still undetermined",
+     "            return 0, [\n"
+     "                \"flow: the realm binds browserFlow to %r but its executions could not \"\n"
+     "                \"be read, so this program cannot establish which flows are live. Flow \"\n"
+     "                \"writes stay FAIL-CLOSED for this run -- nothing was built or bound\"\n"
+     "                % bound]",
+     "            _set_protected(set(), set())\n            bound_shape, bound_raw = None, None"),
+
+    # ------------------------------------------------------- the conditional-OTP regression
+    ("authenticator config contents no longer compared",
+     "check only that a config is BOUND, not what it says -- this is THE conditional-OTP "
+     "regression: the original bug wrote the role under "
+     "`condition.evaluated.user.role` instead of `condUserRole`, so the role resolved "
+     "to null, the REQUIRED conditional step threw 'Invalid role name submitted: null' "
+     "and every platform-admin got a 403 from every SSO-gated app",
+     "            if str(have_cfg.get(key)) != str(want_cfg[key]):",
+     "            if False:"),
+
+    ("undeclared config keys are ignored",
+     "stop reporting keys the realm holds and git does not -- the stale "
+     "`condition.evaluated.user.role` key is the signature of the original null-role "
+     "bug, and a flow carrying both keys would look correct",
+     "        extra = sorted(k for k in have_cfg if k not in want_cfg)",
+     "        extra = []"),
+
+    ("flow structure divergence detected but not reported",
+     "leave the per-step comparison out -- the realm's login flow could be the old "
+     "pre-passkey shape forever and every run would report the realm converged, which "
+     "is the exact fail-open of the one-shot import this program replaces",
+     "        if observed[idx][:4] != expected[idx][:4]:",
+     "        if False:"),
+
+    ("the browserFlow BINDING is no longer compared",
+     "stop noticing that a correct flow is not the one bound -- a perfect, fully-built "
+     "passkey flow that nothing logs in through changes nothing at all",
+     "    if bound_shape is not None:\n"
+     "        drift = compare_shape(bound_shape, expected, base, realm, token, configs,",
+     "    if False:\n"
+     "        drift = compare_shape(bound_shape, expected, base, realm, token, configs,"),
+
+    # ------------------------------------------------------------- realm / policy writes
+    ("partial policy PUT instead of read-modify-write",
+     "send only the drifted policy keys -- Keycloak 26.0.6 rebuilds the WHOLE "
+     "passwordless policy from the representation and substitutes its own DEFAULT for "
+     "every key the body omits, so a partial PUT silently resets everything it did not "
+     "mention while returning 204",
+     "    body = dict(realm_rep)\n"
+     "    for key in drift:\n"
+     "        body[key] = wanted[key]",
+     "    body = {RP_ENTITY_NAME: realm_rep.get(RP_ENTITY_NAME),\n"
+     "            BROWSER_FLOW: realm_rep.get(BROWSER_FLOW)}\n"
+     "    for key in drift:\n"
+     "        body[key] = wanted[key]"),
+
+    ("the policy pass rebinds browserFlow as a side effect",
+     "'while we are here, let the policy write fix the flow binding too' -- the realm "
+     "PUT does read rep.getBrowserFlow(), so this is genuinely in reach, and it would "
+     "rebind the login path to an UNVERIFIED flow from a pass whose stated surface is "
+     "the passkey policy",
+     "    if body.get(BROWSER_FLOW) != realm_rep.get(BROWSER_FLOW):\n"
+     "        raise RuntimeError(",
+     "    body[BROWSER_FLOW] = \"browser\"\n"
+     "    if False:\n"
+     "        raise RuntimeError("),
+
+    ("the RpEntityName gate removed from desired-state validation",
+     "accept a desired state with an empty RpEntityName -- Keycloak 26.0.6 applies the "
+     "passwordless policy ONLY when that field is non-empty, so every realm PUT would "
+     "be accepted with 204 and change nothing: reported success, zero convergence",
+     "    if not policy.get(RP_ENTITY_NAME):\n"
+     "        raise ValueError(",
+     "    if False:\n        raise ValueError("),
+
+    ("the RpEntityName gate removed from the write path",
+     "drop the second, defence-in-depth guard -- it is unreachable while the "
+     "desired-state validation holds, which is exactly why deleting it would otherwise "
+     "be an invisible regression",
+     "    if not body.get(RP_ENTITY_NAME):\n"
+     "        raise RuntimeError(",
+     "    if False:\n        raise RuntimeError("),
+
+    ("no readback verification on the policy write",
+     "trust the 204 -- a server that accepts the PUT and does not apply it is reported "
+     "as REPAIRED, which is 'reports success while the realm never converged' again, "
+     "and it is precisely what the RpEntityName gate does when it is missed",
+     "    still = policy_drift(after, wanted)",
+     "    still = {}"),
+
+    ("AAGUID comparison made order-sensitive",
+     "compare the AAGUID allow-list in the order Keycloak returns it -- an allow-list "
+     "is a set membership test, so an order-sensitive comparison rewrites a CONVERGED "
+     "realm on every run forever while every run still reports success",
+     "            if stored != sorted(declared_value):\n"
+     "                drift[key] = (stored, sorted(declared_value))",
+     "            if stored != declared_value:\n"
+     "                drift[key] = (stored, declared_value)"),
+
+    ("signature-algorithm order treated as insignificant",
+     "sort the signature algorithms before comparing -- it is a PREFERENCE LIST handed "
+     "to the authenticator, so ES256-then-RS256 is not the same declaration as "
+     "RS256-then-ES256, and a reorder in the realm would never be corrected",
+     "            if stored != list(declared_value):",
+     "            if sorted(stored) != sorted(declared_value):"),
+
+    ("unmanaged policy keys forced to a value",
+     "iterate every writable key instead of only the declared ones -- a "
+     "webAuthnPolicyPasswordless field the realm holds and git does not declare would "
+     "be overwritten, turning 'not declared' into 'must equal my default'",
+     "    for key in sorted(wanted):\n        declared_value = wanted[key]",
+     "    for key in POLICY_FIELDS:\n        declared_value = wanted.get(key, \"\")"),
+
+    # ------------------------------------------------------------------ required actions
+    ("required-action PUT sends only the two drifted fields",
+     "a minimal body looks safer -- but updateRequiredAction REPLACES the provider from "
+     "the representation, so name, priority and config are all blanked: the enrolment "
+     "action keeps working but loses its priority and display name",
+     "    body = dict(rep)\n    for key in drift:\n        body[key] = wanted[key]",
+     "    body = {}\n    for key in drift:\n        body[key] = wanted[key]"),
+
+    ("the guarded default required actions are no longer audited",
+     "stop checking that CONFIGURE_TOTP / UPDATE_PASSWORD / delete_credential still "
+     "exist -- their disappearance is the signature of a requiredActions block in the "
+     "realm import, which 26.0.6 treats as REPLACE, and it is a silent permanent "
+     "lock-out of every platform-admin",
+     "    missing = [a for a in GUARDED_ACTIONS if a not in aliases]",
+     "    missing = []"),
+
+    # ----------------------------------------------------------------- inputs / coupling
+    ("the three passes are coupled",
+     "abort the run at the first failing pass -- a policy write that fails for an "
+     "unrelated reason would then suppress the browser-flow pass, which is the most "
+     "safety-critical thing here",
+     "        problems.extend(found)\n        if found:\n            rc = 1",
+     "        problems.extend(found)\n        if found:\n            rc = 1\n"
+     "            break"),
+
+    ("policy-key validation removed from the desired state",
+     "accept any realm key from the ConfigMap -- an arbitrary realm attribute becomes "
+     "writable through a file edit, so a hand-edited ConfigMap could set sslRequired, "
+     "bruteForceProtected or any other top-level realm field on the live IdP",
+     "        if key not in POLICY_FIELDS:",
+     "        if False:"),
+
+    ("fail-open on an unreadable desired state",
+     "treat a missing ConfigMap as 'nothing declared' -- the run reports success while "
+     "reconciling nothing, which is precisely the fail-open of the realm import this "
+     "whole component exists to replace",
+     "    if not isinstance(raw, dict) or not raw:\n"
+     "        raise ValueError(\"%s does not hold a non-empty JSON object\" % path)",
+     "    if not isinstance(raw, dict) or not raw:\n        return {}"),
+
+    ("an undeclared authenticatorConfig reference is tolerated",
+     "stop validating that every configAlias a flow references is actually declared -- "
+     "the build then fails at the config-binding step, AFTER flows and executions have "
+     "already been created, leaving leftovers on every run",
+     "            if ca and ca not in configs:\n"
+     "                raise ValueError(",
+     "            if False:\n                raise ValueError("),
+
+    ("the flow-cycle guard is removed",
+     "drop the `seen` check -- a cyclic flow declaration then recurses until the "
+     "interpreter dies, in a CronJob that retries",
+     "        if alias in seen:\n"
+     "            raise ValueError(\"flow %r is cyclic in the desired state\" % alias)",
+     "        if False:\n"
+     "            raise ValueError(\"flow %r is cyclic in the desired state\" % alias)"),
+]
+
+
+def main() -> int:
+    if len(sys.argv) != 2:
+        print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
+        return 2
+    src = Path(sys.argv[1]).read_text()
+
+    problems: list[str] = []
+    killed = 0
+    for name, why, old, new in MUTATIONS:
+        # An anchor that no longer matches is a FAILURE, not a skip: it means the code
+        # moved and this mutation silently stopped proving anything.
+        n = src.count(old)
+        if n != 1:
+            problems.append("%s: anchor matches %d times (expected exactly 1) -- the "
+                            "code moved and this mutation no longer proves anything. "
+                            "Re-anchor it against the current source." % (name, n))
+            continue
+        with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
+            fh.write(src.replace(old, new, 1))
+            path = fh.name
+        try:
+            proc = subprocess.run([sys.executable, path, "--selftest"],
+                                  capture_output=True, text=True, timeout=300)
+        finally:
+            Path(path).unlink(missing_ok=True)
+        if proc.returncode == 0:
+            problems.append("%s SURVIVED -- --selftest still passed with this regression "
+                            "applied, so nothing in the suite actually checks it. The "
+                            "mistake it models: %s" % (name, why))
+            continue
+        killed += 1
+        out = proc.stdout + proc.stderr
+        m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
+        how = ("%s assertion(s) caught it" % m.group(1) if m
+               else "the selftest aborted (caught, less cleanly)")
+        print("  killed  %-54s %s" % (name, how))
+
+    print()
+    if problems:
+        print("MUTATION CHECK FAIL: %d of %d mutation(s) not proven"
+              % (len(problems), len(MUTATIONS)), file=sys.stderr)
+        for p in problems:
+            print("  FAIL %s" % p, file=sys.stderr)
+        return 1
+    print("MUTATION CHECK OK: all %d injected regressions were caught by --selftest "
+          "(the suite is falsifiable, not decorative)" % killed)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

conformance/tests/l0-static/tls-lifecycle-automated/check.sh

@@ -42,6 +42,19 @@
 #           whose every solver is an automated challenge type (dns01 / http01) — the
 #           same code path serves issuance AND renewal, so a solver a human must
 #           complete would defeat both;
+#       (5b) the ONE exception to (5) — WORKLOAD mTLS, added 2026-10-29 — is a Certificate
+#           annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`. The spec's TLS
+#           requirements are about EXTERNALLY REACHABLE services (R-TLS-01; docs/08), and
+#           ADR-0006 says outright that "mTLS between workloads is a separate concern … not
+#           covered here". A public ACME CA cannot issue for a `*.svc.cluster.local` name or a
+#           client-auth certificate at all. The annotation is NOT trusted: an exempt
+#           Certificate must PROVE it is workload-internal — its Secret is served by NO Gateway
+#           listener, every DNS name / commonName is cluster-internal (a bare label or
+#           `*.svc[.cluster.local]`), it carries no IP/URI/email SANs, and its issuer is a
+#           NAMESPACED Issuer in its own namespace whose spec is `selfSigned` or a `ca` backed
+#           by a Secret that a tracked, likewise-exempt Certificate produces. Issuance and
+#           renewal stay unattended (cert-manager renews CA-issued certs itself), and (9)
+#           still applies. Every exemption is printed. First user: platform/buildkit.
 #       (6) each dns01 solver's provider credential is referenced BY NAME and that
 #           Secret is delivered SOPS-encrypted from git via a kustomization — the
 #           operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step;
@@ -265,6 +278,7 @@ CERT_SECRETS: dict[tuple[str, str], str] = {           # (namespace, secretName)
     for rel, d in CERTS
 }
 listeners_checked = 0
+GATEWAY_SECRETS: set[tuple[str, str]] = set()          # every (namespace, Secret) a listener serves
 for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
     gw_ns = ns_of(gw)
     for lis in ((gw.get("spec") or {}).get("listeners") or []):
@@ -280,6 +294,7 @@ for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
         for r in refs:
             kind = str(r.get("kind", "Secret"))
             key = (str(r.get("namespace") or gw_ns), str(r.get("name")))
+            GATEWAY_SECRETS.add(key)
             check(kind == "Secret",
                   f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only "
                   "Secret refs can be traced to a cert-manager Certificate")
@@ -324,8 +339,61 @@ CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manag
 ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS}
 solver_creds: list[tuple[str, str, str, str]] = []     # (issuer, ns, secretName, ref field)
 issuers_verified: set[str] = set()
+
+# (5b) workload-mTLS exemption — see the header. Proven, never taken on trust.
+SCOPE_ANN = "homelab.keithjackson.dev/tls-scope"
+INTERNAL_NAME = re.compile(r"^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$")
+NS_ISSUERS = {(ns_of(d), name_of(d)): (rel, d) for rel, d in of_kind("Issuer", "cert-manager.io/")}
+EXEMPT_SECRETS = {(ns_of(d), str((d.get("spec") or {}).get("secretName", "")))
+                  for _, d in CERTS
+                  if ((d.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN) == "workload-mtls"}
+exempted: list[str] = []
+
+def workload_mtls_ok(rel: str, cert: dict) -> bool:
+    """True if `cert` claims the workload-mTLS scope AND proves it; records failures."""
+    ann = ((cert.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN)
+    if ann is None:
+        return False
+    cns, cname = ns_of(cert), name_of(cert)
+    spec = cert.get("spec") or {}
+    ok = True
+    def need(cond, msg):
+        nonlocal ok
+        check(cond, f"{rel}: Certificate {cns}/{cname} claims {SCOPE_ANN}={ann!r} but {msg}")
+        ok = ok and bool(cond)
+    need(ann == "workload-mtls", "the only accepted value is 'workload-mtls'")
+    need((cns, str(spec.get("secretName", ""))) not in GATEWAY_SECRETS,
+         "its Secret is served by a Gateway listener — that is external TLS, which must be ACME (R-TLS-01)")
+    names = [str(n) for n in (spec.get("dnsNames") or [])]
+    if spec.get("commonName"):
+        names.append(str(spec["commonName"]))
+    for n in names:
+        internal = INTERNAL_NAME.match(n) and ("." not in n or n.endswith(".svc") or n.endswith(".svc.cluster.local"))
+        need(bool(internal), f"name {n!r} is not cluster-internal (bare label or *.svc[.cluster.local])")
+    for f in ("ipAddresses", "uris", "emailAddresses"):
+        need(not spec.get(f), f"it carries {f}, which a workload-internal certificate has no use for")
+    ref = spec.get("issuerRef") or {}
+    need(str(ref.get("kind", "Issuer")) == "Issuer",
+         "its issuer is not a NAMESPACED Issuer — a ClusterIssuer could sign for any namespace")
+    hit = NS_ISSUERS.get((cns, str(ref.get("name", ""))))
+    need(hit is not None, f"its Issuer {ref.get('name')!r} is not tracked in namespace {cns!r}")
+    if hit is not None:
+        ispec = hit[1].get("spec") or {}
+        kinds = sorted(ispec)
+        need(kinds in (["selfSigned"], ["ca"]),
+             f"its Issuer is {kinds} — only selfSigned or ca are accepted for workload mTLS")
+        if kinds == ["ca"]:
+            need((cns, str((ispec.get("ca") or {}).get("secretName", ""))) in EXEMPT_SECRETS,
+                 "its CA Issuer's Secret is not produced by a tracked workload-mtls Certificate "
+                 "in the same namespace — the CA itself would have to be minted by hand")
+    if ok:
+        exempted.append(f"{cns}/{cname}")
+    return ok
+
 for rel, cert in CERTS:
     spec = cert.get("spec") or {}
+    if workload_mtls_ok(rel, cert):
+        continue          # (5b): proven workload-internal; (9) below still applies
     ref = spec.get("issuerRef") or {}
     kind = str(ref.get("kind", "Issuer"))
     per_env = resolve_per_env(str(ref.get("name", "")))
@@ -379,6 +447,8 @@ for rel, cert in CERTS:
                             solver_creds.append((iname, want_ns, n, f"{provider}.{field}"))
 
 check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing")
+if exempted:
+    print(f"  (5b) workload-mTLS certificates exempted from ACME after proving it: {', '.join(exempted)}")
 
 # (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand.
 check(len(solver_creds) > 0,

conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml

@@ -171,6 +171,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): paas-demo-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: paas-demo-allow-monitoring-scrape
+  namespace: paas-demo
+  labels:
+    app.kubernetes.io/name: paas-demo
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: paas-demo
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): paas-demo-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml

@@ -198,6 +198,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

@@ -198,6 +198,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

conformance/versions.env

@@ -40,32 +40,29 @@ PYTHON_MIN="3.9"
 #   gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
 # Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
 CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
-# Build handle only. Admission never sees the tag; a Job must reference the digest.
+# Local build handle only. Admission never sees the tag; a Job must reference the digest.
+# CI publishes under `sha-<commit>` and `main` instead (.gitea/workflows/images.yaml).
 CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
 #
 # ############################################################################
-# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET.        #
+# # THE DIGEST IS WRITTEN BY RENOVATE, NOT BY HAND.                          #
 # ############################################################################
-# The image has been BUILT and its toolchain verified locally (non-root uid 65532,
+# CI builds and verifies the image (non-root uid 65532, read-only rootfs, the
-# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
+# conformance-clone refusals) and, on a push to main, publishes it and prints the digest the
-# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
+# REGISTRY returned (.gitea/workflows/images.yaml). Renovate follows the image's `main` tag
-# pushing needs a registry credential, and publishing the artifact a fail-closed security
+# through the marker below and opens the pull request that moves this value. Nobody pushes
-# boundary then trusts is an OPERATOR action, not an agent's.
+# this image from a workstation, and nobody pastes a digest here.
 #
-# OPERATOR, to complete the pin:
+# ALL-ZERO UNTIL THE FIRST PUBLISH. Before 2026-10-29 this was EMPTY, so that a consumer
-#   1. ./conformance/incluster/build-image.sh          # rebuild, verify, print the commands
+# rendering the Job would fail visibly. It is now the same impossible all-zero digest
-#   2. podman login gitea.home.keithjackson.dev
+# platform/pm-agent/ uses, because Renovate can only replace a digest that is there to match.
-#   3. podman push --digestfile /tmp/conformance-runner.digest \
+# A consumer MUST treat the all-zero value as "not published" (no consumer exists yet: the
-#        "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
+# L2/L3 trigger is unwired, docs/plans/l2-l3-conformance-ci-wiring.md). Note also that
-#   4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
+# chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the
-#      — the digest the REGISTRY returned, not the local manifest digest the build printed.
+# `sha256:` prefix this value carries — the trigger must strip it.
-#   5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
+# renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main
-#
+CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:65cc7c1ab576fe8cdf45e7f68c7cf4eb5ce7e0db46e62191f6190c51610b20d9"
-# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
+CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}"
-# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
-# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
-CONFORMANCE_RUNNER_IMAGE_DIGEST=""
-CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
 #
 # AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
 # its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —

decisions/2026-10-04-hubble-flow-export-plex.md

@@ -0,0 +1,157 @@
+---
+date: 2026-10-04
+branch: feat/hubble-flow-export-plex
+title: Record the pre-SNAT client address for Plex via Hubble static flow export, instead of breaking externalTrafficPolicy to get it
+---
+
+## What changed
+
+Operator requirement, narrowed explicitly: **this is 100% about log visibility** — knowing which
+address connected to Plex. Not bandwidth metering, which does not matter here.
+
+**The problem, measured.** Plex's own log cannot show it. Every LoadBalancer Service in this cluster
+is `externalTrafficPolicy: Cluster`, so the node holding the L2 lease for the VIP SNATs the client
+before forwarding to the backend pod:
+
+| Measured 2026-10-04 | Value |
+|---|---|
+| Node announcing `192.168.2.24` | `k3s-prod-3` (also holds `.20` Gateway, `.23` pihole-dns) |
+| Node running the Plex pod | `k3s-prod-1` (`10.0.0.135`) |
+| Sources in Plex's log | **3,214 × `10.0.1.x`**, 105 × `10.0.2.x`, 120 × loopback |
+| `192.168.x` sources in Plex's log, ever | **0** |
+| `X-Forwarded-For` in Plex's log | **0** (it serves raw tcp/32400, not HTTP via the Gateway) |
+| `10.0.1.78` identity (`cilium-dbg bpf ipcache`) | `identity=1` → `reserved:host` = prod-3 itself |
+
+**Nothing on the router is involved** — the operator correctly pointed out there is no hairpin NAT
+here. The rewrite is entirely in-cluster, done by Cilium's BPF load-balancer on the lease holder,
+because the backend pod is on a different node.
+
+**The address is not lost, only unrecorded.** Cilium observes the flow at ingress, *before* its own
+translation, so Hubble already holds the true source:
+
+    192.168.2.133 -> 10.0.0.135    (real LAN client -> the Plex pod)
+    10.0.1.78     -> 10.0.0.135    (the same traffic post-SNAT — what Plex sees)
+
+What was missing is **retention**: the ring buffer was **4095/4095 flows at ~303 flows/s — about
+thirteen seconds of history** — with hubble metrics disabled and no exporter configured.
+
+So: enable the static flow exporter, scoped to Plex's port, writing to the agent's stdout.
+
+- **`platform/cilium/helmrelease.yaml`** (authoritative): `hubble.export.static.enabled: true`,
+  `filePath: stdout`, `allowList` of `{"destination_port":["32400"]}` and
+  `{"source_port":["32400"]}`.
+- **`platform/cilium/values.yaml`** (reference mirror) and
+  **`infra/ansible/roles/cilium/templates/cilium-values.yaml.j2`** (the `[parity]` source of truth):
+  the same block, so a bootstrap render does not differ from what Flux reconciles — a mismatch would
+  be a permanent Helm diff and an avoidable agent rollout.
+
+**`filePath: stdout`, not a file,** because Grafana Alloy already runs as a DaemonSet tailing every
+pod's logs into Loki (`platform/monitoring/alloy.yaml`). stdout therefore reaches Loki with **no new
+hostPath mount, no file-rotation handling, and no Alloy change at all**. Upstream documents the
+value: *"To get Hubble flows directly exported to the logs instead of written to a rotated file,
+`stdout` can be defined as `hubble-export-file-path`."* Query:
+
+    {namespace="kube-system", app="cilium"} |= "32400"
+
+**SCOPED BY PORT, NOT BY NAMESPACE — a measured decision, not a preference.** A
+`--to-namespace plex` query returned **only the post-SNAT flows**; the flow carrying the real client
+address did not match it, because at the point of observation the destination identity is not yet
+resolved to the plex pod. A namespace filter would have dropped exactly the record this change
+exists to capture.
+
+**WHY NOT FIX THE SNAT INSTEAD — both alternatives rejected on evidence.**
+- `externalTrafficPolicy: Local` would put the real address in Plex's own log, but cilium 1.20's L2
+  announcer is **explicitly incompatible** with it. Upstream docs for this exact version:
+  *"incompatible with the `externalTrafficPolicy: Local` on services as it may cause service IPs to
+  be announced on nodes without pods causing traffic drops"* and *"L2 announcements isn't currently
+  aware of this setting"*. Live proof it is not theoretical: the VIP is announced by prod-3 while the
+  pod runs on prod-1, so the flip would black-hole tcp/32400 **immediately**. Making it safe means
+  pinning both the pod and the announcement to one node and surrendering Plex's ability to
+  reschedule — a real availability loss to buy a log field. Same trap
+  `platform/crafty/service-l4.yaml` documents from an 8-day Minecraft outage.
+- **DSR** also preserves the client address, but `routing-mode` here is **vxlan**, not
+  native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO router's stateful
+  NAT on the remote path. Cluster-wide blast radius across all eight LB Services for one service's
+  logging.
+
+**A correction to the repo's own record.** `decisions/2026-10-03-plex-remote-access-l4.md` states the
+SNAT puts the client inside `192.168.0.0/16` (the ingress node's LAN address), and I repeated that in
+the previous entry. **It is wrong**: measured, the SNAT source is `10.0.1.x` — the node's
+`cilium_host` inside the pod CIDR. This matters, because it means `LanNetworksBandwidth` can never
+discriminate local from remote while eTP is `Cluster`: `10.0.0.0/8` must be in the LAN set for the
+in-cluster consumers (Seerr) to count as local, and every external client arrives in that same range.
+PR #48's LAN definition therefore fixes the Remote badge by making *everything* local — acceptable
+only because metering is explicitly not a goal here.
+
+## What I verified
+
+Live prod, 2026-10-04, read-only.
+
+- **Every measurement in the table above**, including the VIP-lease-to-node map
+  (`cilium-dbg shell -- db/show l2-announce` on all three agents) and the pod's node.
+- **Hubble already carries the pre-SNAT source** — the single finding this whole approach rests on:
+  `hubble observe --port 32400` on the lease holder returned `192.168.2.133 -> 10.0.0.135` **and**
+  its return flow, alongside the post-SNAT `10.0.1.78 -> 10.0.0.135`.
+- **The retention gap is real:** `cilium-dbg status` reported `Hubble: Ok  Current/Max Flows:
+  4095/4095 (100.00%), Flows/s: 303.62  Metrics: Disabled`, and the `cilium-config` ConfigMap
+  contains no `hubble-export-*` key at all.
+- **The namespace-filter trap**, directly: `--to-namespace plex` returned only `10.0.1.78` and
+  `10.0.1.71`; the `192.168.2.133` flow was absent. This is why the allowList is port-scoped.
+- **All eight LoadBalancer Services are `eTP=Cluster`** (crafty ×2, gateway, pihole-dns ×2, plex,
+  unifi ×2) — so this is a cluster-wide pattern, not a Plex-specific choice.
+- **The eTP:Local incompatibility is upstream-documented for v1.20 specifically**, quoted above from
+  the docs for the running version — not inferred from the crafty incident.
+- **`routing-mode: tunnel`** and **`bpf-lb-mode-annotation: false`** in the live `cilium-config`,
+  which is what rules DSR out.
+- **The chart's value paths and defaults**, read from cilium v1.20.1's own `values.yaml`:
+  `hubble.export.static.{enabled,filePath,fieldMask,fieldAggregate,aggregationInterval,allowList,denyList,fileMaxSizeMb,fileMaxBackups,fileCompress}`,
+  with `filePath` defaulting to `/var/run/cilium/hubble/events.log`. The keys used here exist.
+- **Alloy's pipeline shape**, read from `platform/monitoring/alloy.yaml`: a DaemonSet with
+  `loki.source.kubernetes "pods"` → `loki.process` → `loki.write` at
+  `http://loki.monitoring.svc.cluster.local:3100`. It already tails kube-system pods, which is why
+  stdout needs no Alloy change.
+- **All three value files agree** (one `destination_port` occurrence each) and both YAML files parse,
+  with `export.static` resolving to the intended object.
+- **Toolbox gate**: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two known environmental skips
+  (`gitea-settings` — no local `SETTINGS_API_TOKEN`; `conformance` L2/L3 — `CLUSTER_AVAILABLE`
+  unset). **Not green. Reported as rc 3.**
+- **No sensitive paths touched** — `platform/cilium/**` and `infra/ansible/**` are not in
+  `infra/gitea/approval-policy.json`, so this needs no approval override, unlike PR #48.
+
+## What I did NOT verify
+
+- **THE OUTCOME. Nothing has been reconciled and no flow has ever been exported.** `cilium-config`
+  still has no `hubble-export-*` key. Every claim is about inputs.
+- **Whether `filePath: stdout` is accepted as written.** This is the single largest risk. The value
+  comes from a verbatim upstream doc sentence (*"`stdout` can be defined as
+  `hubble-export-file-path`"*), but I did not confirm the exact notation in cilium 1.20.1's source,
+  and the chart default is a real filesystem path. If the literal string is not special-cased, the
+  agent may create a FILE named `stdout` in its working directory and nothing reaches Loki — a
+  silent no-op, not a crash. **Fallback if so:** leave `filePath` at the chart default and add a
+  `local.file_match` + `loki.source.file` target to Alloy with a hostPath mount for
+  `/var/run/cilium/hubble`.
+- **Whether the FlowFilter JSON is accepted.** `{"destination_port":["32400"]}` follows the proto
+  (`destination_port` is a repeated string) and the documented shape (`'{"verdict":["DROPPED"]}'`),
+  but upstream gives **no port example** and I did not validate it against the running agent. A
+  malformed filter could be rejected — or, worse, be silently treated as "no match" and export
+  nothing.
+- **THIS CHANGE ROLLS CILIUM.** Editing the HelmRelease restarts the agent DaemonSet across all
+  three nodes. BPF maps are preserved so established traffic should survive, but prod-3 currently
+  announces the Gateway VIP, **PiHole's DNS VIP** and Plex's VIP, so a lease change during the
+  rollout could briefly interrupt LAN DNS for the whole house. Not measured, and worth doing at a
+  quiet moment.
+- **Log volume in practice.** Estimated "a few hundred to a few thousand lines a day" from the
+  observed request counts; never measured as flows. Both directions are exported and no `fieldMask`
+  is set, so each record is a full flow object. If it is noisy, narrow with `fieldMask` rather than
+  by dropping a filter.
+- **Loki's retention for this data**, and whether the `{namespace="kube-system", app="cilium"}` label
+  set actually matches (it depends on the cilium pods carrying
+  `app.kubernetes.io/name`, which Alloy maps to `app` — I did not confirm that label exists on the
+  cilium DaemonSet pods). The query in the comment may need adjusting to `pod=~"cilium-.*"`.
+- **That the exported flow actually contains the pre-SNAT address.** Hubble's *live* stream does
+  (measured). I am assuming the exporter sees the same flow objects; I did not confirm the exporter
+  is not fed a post-translation view.
+- **The parity convention has no automated check.** I kept all three files in sync by hand because
+  the comments demand it; nothing in `ci/` or `conformance/` verifies cilium Helm↔Ansible parity, so
+  a future edit can still drift them silently.

decisions/2026-10-04-oauth2-sidecar-metrics-scrape-netpol.md

@@ -0,0 +1,155 @@
+---
+date: 2026-10-04
+branch: fix/oauth2-sidecar-metrics-scrape-netpol
+title: The forward-auth gate sidecars' /metrics was NetworkPolicy-blocked — the renderer now opens 44180 to ns monitoring
+---
+
+> **Added 2026-10-05, after merge:** the top item under *What I did NOT verify* below — "that the
+> scrape now succeeds" — **has since been verified on prod**. All 6 targets read `up == 1`. The
+> original text is left exactly as written; the measurements are in
+> [[2026-10-05-oauth2-sidecar-scrape-verified]].
+
+## What changed
+
+`catalog/render.py`'s `r_monitoring_ingress_policy` now emits its CiliumNetworkPolicy for a
+**second** reason: the forward-auth gate sidecar's own `/metrics` port (44180), keyed on
+`_needs_oauth2_sidecar(s)`. Previously it fired only from `observability.metricsEndpoint`.
+
+That matters more than it sounds, because **no ServiceDefinition in this catalog declares a
+`metricsEndpoint`** (measured: zero grep hits across `catalog/*.yaml`). So the function was wired
+in at the call site and had **never emitted a single object on any branch** — correct, and
+entirely latent. Its presence in the tree read as "scrape ingress is handled". It was not.
+
+Re-rendered `apps/` from that change: five new CNPs, one per forward-auth app
+(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`), each opening **only** 44180 to **only**
+`fromEndpoints: [k8s:io.kubernetes.pod.namespace: monitoring]`.
+
+The bug being fixed: `platform/oauth2-proxy/podmonitor.yaml` landed 2026-10-02T04:58:44Z and
+discovered 6 targets that have **never once scraped successfully**. Each app namespace ships a
+`default-deny` NetworkPolicy with `policyTypes: [Ingress, Egress]` and **no `ingress` key at all**
+— deny-all-ingress in NetworkPolicy semantics — and the only ingress allowance is
+`<app>-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180. Prometheus matched no
+clause, so its packets were dropped.
+
+Two deliberate choices, both recorded in the function's docstring:
+
+- **Clause 2 keys on `_needs_oauth2_sidecar`, not on `observability.metrics`.** It has to match
+  the condition the *scrape* keys on: `podmonitor.yaml` selects pods by the
+  `homelab.keithjackson.dev/auth-mode: forward-auth` pod label and nothing else, so a target
+  exists iff the sidecar exists. Keying on `observability.metrics` would let `metrics: false`
+  produce a discovered-but-blocked target — the same "two halves authored apart" failure this
+  function exists to prevent, moved one field over.
+- **The policy is rendered here, not added to `platform/oauth2-proxy/`.** podmonitor.yaml's own
+  header records why that tier cannot host it: it would name app namespaces created by the
+  *sibling* `apps-wave-0/1` tiers, so it would apply on a running cluster and fail on a cold
+  bootstrap. Rendered per-app it ships in the same tier that creates the namespace.
+
+## What I verified
+
+Measured on prod 2026-10-04, read-only, via the pinned toolbox:
+
+- **The cause, not a guess.** All 6 sidecar targets in pool
+  `podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0`: `health=down`,
+  `lastError: context deadline exceeded`, `lastScrapeDuration` exactly `10.0s` — the configured
+  `scrapeTimeout`. Hitting the timeout precisely means packets **dropped, not refused**.
+- **The listener is fine.** jackett's `oauth2-proxy` container carries
+  `--metrics-address=0.0.0.0:44180` and a named port `metrics: 44180`.
+- **The deny is universal.** All six namespaces (`jackett radarr sonarr qbittorrent pihole
+  crafty`): `default-deny` with `ingress` key absent, `nIngress=0`; the only ingress CNP allowance
+  is 4180 `fromEntities: [ingress]`. **Zero** namespaces had any `-allow-monitoring-scrape` object.
+- **Frequency.** Single continuous episode, no flapping: `ALERTS_FOR_STATE` = 229343s (63.7h),
+  firing since 2026-10-02 05:09 UTC — 11 minutes after the PodMonitor was created, i.e. TargetDown's
+  `for: 10m`. A 15-day `query_range` shows **1 episode** per namespace, not many. Alertmanager:
+  `group_by: [alertname, namespace]`, `repeat_interval: 12h` on `email-warning` → 5 TargetDown
+  groups = 10 emails/day, plus 6 `OAuth2ProxyGateUnobserved` groups = 12/day, ~22/day total.
+  0 silences.
+- **The re-render is deterministic and the diff is exactly the intended one.** `apps/` is
+  byte-identical to `render.py --all catalog -o apps --wave-layout --no-image-automation`
+  (12/12 SAME before my edit; the 11-file "drift" against a default render is solely the
+  ImageRepository/ImagePolicy pair, which `apps/` is generated without). After the edit:
+  5 files changed, +21 lines each, nothing else touched.
+- `render.py --selftest`: **PASS**.
+- **Three L1 goldens needed regenerating, and the diff is +63 lines / -0.** The first gate run
+  caught `container-workload-render` (R-PLAT-02) and `egress-class-render` (R-NET-05/06/07) as
+  FAIL with exactly the new CNP as the drift. Regenerated with the same invocations the checks
+  use (`render.py <fixture>`, no flags). Evidence the regen is faithful rather than a rubber
+  stamp: `default-deny-baseline-render`'s golden came back **byte-identical** (its fixture is not
+  forward-auth), and the other three gained only the one CNP, port 44180, ns monitoring.
+- Toolbox gate `./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`: **rc 3 — UNDETERMINED,
+  NOT rc 0, and I am not reporting it as green.** Every stage that ran is green
+  (spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1); `l1` went
+  FAIL -> pass once the goldens were regenerated. The gate's own words: "Stages that did not run:
+  gitea-settings" — skipped for want of `GITEA_SETTINGS_TOKEN`, which is an environment gap on
+  every local run and unrelated to this change. L2/L3 conformance is a `[warn]`, not a not-run
+  stage: UNVERIFIED for all 8 touched requirements (R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01
+  R-NET-05 R-NET-06 R-NET-07 R-PLAT-02), no cluster.
+- `--emit-waves` was **not** run (it re-onboards `vm-demo` + `keycloak`); only the per-service
+  render path was used, so no wave kustomization changed.
+
+## This PR needs a human approval (sensitive path) — MEASURED, not predicted
+
+CI run #230 on PR #49, head `f9446be3`: **`MERGE GATE: BLOCKED`, and `approval-policy` is the ONLY
+failing stage.** Read from the job log, not from the status tick:
+
+    [skip] stage spec-immutability: SKIPPED (not run)   <- sibling spec repo absent in CI; expected
+    [ ok ] stage decisions-log: pass
+    [ ok ] stage lint: pass
+    [ ok ] stage l0: pass
+    [ ok ] stage render-golden: pass
+    [ ok ] stage l1: pass
+    [fail] stage approval-policy: FAIL
+    [ ok ] stage gitea-settings: pass                   <- could NOT run locally; green in CI
+    [skip] stage conformance: SKIPPED (not run)
+
+    sensitive paths touched:
+      conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml
+      conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml
+      conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml
+
+Exactly the three regenerated goldens, nothing else. `l1` and `render-golden` both pass in CI,
+which is the independent confirmation that the regenerated goldens are right rather than merely
+self-consistent with my local run.
+
+One local-reproduction note worth recording, because it wasted a step: running
+`PR_NUMBER=49 SETTINGS_API_TOKEN=... ci/approval-policy.sh` with the credential from
+`git credential fill` reports `could not determine PR author — BLOCKING`. That is an **artifact**,
+not a finding — `~/.git-credentials` holds a *password*, and `Authorization: token <password>`
+returns 401 `user does not exist [uid: 0]`. The stage needs a real API token. I nearly reported
+that 401 as the CI failure; the job log is what settled it.
+
+Regenerating the L1 goldens means this branch touches `conformance/**`, which is in
+`sensitive_paths` in `infra/gitea/approval-policy.json` ("the tests and the harness; changing them
+changes what green MEANS"). So `ci/approval-policy.sh` will require an approving review from a
+non-author, and the local gate **cannot** see that: it reports `approval-policy: green` because
+offline it evaluates N/A.
+
+The golden update is not optional — a renderer change that adds an object necessarily drifts the
+goldens that pin the renderer's output — but it does mean the gate correctly refuses to
+self-approve this one. **An agent must not clear this with `/approve-override`**, even though the
+`gitea-admin` account holds `owner`: that is precisely the self-approval the stage exists to
+prevent. The operator posts the override as a PR *review* of type "Comment", or approves normally.
+
+## What I did NOT verify
+
+- **That the scrape now succeeds.** This is a git-only change and nothing is applied until Flux
+  reconciles `main`. The 5 CNPs have never existed on the cluster, so `up == 1` for those targets
+  is **UNVERIFIED** — it is a prediction from the measured drop signature, not an observation.
+  After merge, confirm `up{container="oauth2-proxy"}` goes to 1 for all 6 targets and that the 11
+  alerts resolve. `send_resolved: true` is set, so expect ~11 resolved emails.
+- **That the CNP `endpointSelector` matches pihole's pods.** It selects
+  `app.kubernetes.io/name: pihole`, identical to the shape of `pihole-allow-gateway-ingress`
+  which demonstrably works for 4180 — so this is inference from a working sibling, not a
+  measurement of the new object.
+- **`crafty` is still unobserved and will keep firing `OAuth2ProxyGateUnobserved`.** It is
+  hand-managed under `platform/crafty`, outside the renderer, and has **no `--metrics-address`
+  at all** — so there is no port to open and this change cannot help it. Separate work.
+- **Whether 44180 is reachable in the Cilium data path once allowed.** Only the policy was
+  reasoned about; no L2 probe was run (CI has `vars.CLUSTER_AVAILABLE` unset, so L2/L3 did not
+  run here either — the gate proves the L0/L1 offline subset only).
+- **Why `OAuth2ProxyGateUnobserved` started at 2026-10-04 06:44** rather than 10-02 with
+  TargetDown. Alertmanager did not restart (TargetDown's `startsAt` is still 10-02) and the rule's
+  `for` is 1h, so neither explains it. Something changed on the monitoring side that morning which
+  I did not identify. It does not affect this fix.
+- I did not re-run a port-forward to 44180 to prove the endpoint serves *today*; I confirmed the
+  flag and named port on the live pod spec. The 200/10208-byte measurement is podmonitor.yaml's,
+  from 2026-10-01.

decisions/2026-10-04-plex-unblock-app-tier.md

@@ -0,0 +1,126 @@
+---
+date: 2026-10-04
+branch: fix/plex-unblock-app-tier
+title: Unblock the app tier — my Certificate stalled apps-wave-0, and the advertised https URL had no servable certificate
+---
+
+## What changed
+
+**I broke the app tier with PR #48 and this reverts the part that did it.** Two faults, both mine,
+both measured after the merge:
+
+1. **Kyverno blocks cert-manager in app namespaces.** The temporary private-key Secrets
+   (`plex-vip-<random>`) were denied by `secret-provenance/secrets-must-be-flux-managed`:
+
+       resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies
+       secret-provenance:
+         secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by
+         Flux from SOPS-encrypted git ... or be exempt controller-generated Secrets.
+
+   The policy matches every Secret in a namespace labelled
+   `homelab.keithjackson.dev/app-namespace: true` and excludes only Secrets labelled
+   `homelab.keithjackson.dev/secret-managed-by: controller` or carrying a Flux label. **The `plex`
+   namespace has that label; the `gateway` namespace — where the working wildcard cert lives — does
+   NOT.** That asymmetry is why the existing Certificate was never affected and mine was, and I did
+   not check it. cert-manager's temp key Secrets carry neither exempting label, and `secretTemplate`
+   cannot label them (the keymanager controller creates them, not the Certificate template). Result:
+   **no CertificateRequest, Order or Challenge was ever created.**
+
+2. **Flux health-checks the Certificate — guarding the pod was not enough.** `optional: true` did
+   exactly what it was designed to do: Plex rolled and ran fine without the Secret (verified). But
+   `apps-wave-0` is `wait: true`, so Flux waits on every applied object with a known health type:
+
+       health check failed after 5m0.045564329s: timeout waiting for:
+       [Certificate/plex/plex-vip status: 'InProgress']
+
+   apps-wave-0 never went Ready, and apps-wave-1/-2 plus everything downstream of platform-config
+   sat in `DependencyNotReady` behind it. I reasoned about the pod and missed the object.
+
+A third, independent problem the same PR introduced: **the advertised URL was `https://` with no
+servable certificate.** Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, which cannot
+cover `plex-vip.${BASE_DOMAIN}` — the precise failure mode of the outage earlier that day.
+
+- **`apps/wave-0/plex-cert.yaml` DELETED**, and `plex-cert.yaml` removed from
+  `apps/wave-0/kustomization.yaml` `resources`, so Flux prunes the stuck Certificate and the health
+  gate clears. **Deleted rather than merely unregistered** — my first attempt kept the file as
+  "tracked but not onboarded" (the shape `keycloak.yaml` and `vm-demo.yaml` use) and **the repo's own
+  gate rejected it**: `tls-lifecycle-automated` (R-TLS-02) fails any Certificate in the tree that no
+  Flux path reconciles — *"nothing reconciles it, so a deleted Certificate/issuer/controller would
+  stay deleted until a human re-applied it."* That pattern is legal for a Deployment and not for a
+  TLS object, which is a distinction I had not known.
+- **`catalog/render.py`:** the now-dangling `plex-cert` EXTRA_UNITS entry removed.
+- **The re-enable recipe is recorded at the call site** — in `catalog/plex.yaml` beside the scheme
+  it is coupled to — rather than in the deleted file: the Kyverno exemption, the non-gating
+  Kustomization, and the `https` flip, as one change.
+- **`catalog/plex.yaml`:** `ADVERTISE_IP` → **`http://`**`plex-vip.${BASE_DOMAIN}:32400`, explicitly
+  temporary, with the restore condition written at the call site.
+
+**Everything else from PR #48 is kept and working** — the hostname design, the PiHole split-horizon
+answer, the `LanNetworksBandwidth` hook, the `optional` schema field, and the TLS env (inert without
+the Secret, so re-enabling is additive).
+
+## What I verified
+
+Live prod, 2026-10-04, read-only, after the #48/#50 merges.
+
+- **The stall and its exact cause**: the health-check timeout message above, and the full dependency
+  cascade (`cilium` → "dependency 'flux-system/flux-system' revision is not up to date", everything
+  else `DependencyNotReady`). The root `cilium` wait was a normal revision-gated cascade; the
+  *Certificate* was the real blocker.
+- **The Kyverno denial text, verbatim**, from the cert-manager controller log, repeating every ~30s
+  across at least 15 distinct Secret names.
+- **The policy's own structure**, read from `platform/kyverno/admission/secret-provenance.yaml`:
+  match = Secrets in `app-namespace: true` namespaces; exclude = `secret-managed-by: controller` OR
+  a `kustomize.toolkit.fluxcd.io/name` label; validate = unconditional `deny`.
+- **The namespace-label asymmetry**: `plex` carries
+  `homelab.keithjackson.dev/app-namespace: "true"`; `gateway` does not.
+- **cert-manager's controller ServiceAccount is `cert-manager`** in namespace `cert-manager` — the
+  identity a future exemption should name.
+- **`optional: true` worked**: the new pod (`plex-c4f87c84d-zgjq5`) is `1/1 Running` with no Secret
+  present, and `/config/certs` was never created.
+- **The postStart hook RAN and took effect** — the thing the whole hook was written for:
+  `LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is now set in the live `Preferences.xml`, and
+  `customConnections="https://plex-vip.home.keithjackson.dev:32400"` was applied from the env. The
+  certificate prefs are correctly **ABSENT**, i.e. the hook's "no bundle → leave them alone" guard
+  behaved as designed.
+- **PiHole's split-horizon answer is correct**, asked of `pihole-0` directly:
+  `plex-vip.home.keithjackson.dev -> 192.168.2.24`, alongside `plex -> .20` and `crafty-mc -> .22`,
+  and the live `/etc/pihole/dnsmasq.conf` carries
+  `address=/plex-vip.home.keithjackson.dev/192.168.2.24`. **This corrects my own earlier check**,
+  which read `.20` and which I nearly reported as a failure — that test resolved via CLUSTER DNS
+  (CoreDNS) from inside a pod and never consulted PiHole at all.
+- **The gate caught my first attempt and I fixed it rather than working around it.** Keeping the
+  file unregistered produced `MERGE GATE: BLOCKED` (rc 1) with
+  `CHECK FAIL (tls-lifecycle-automated): 1 of 612 assertions failed`. After deleting the file and
+  the dangling EXTRA_UNITS entry: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1` and `approval-policy` all **pass** — the rc 3 being only the two known
+  environmental skips (`gitea-settings`, `conformance` L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **Whether this actually unblocks the tier.** Unreconciled at time of writing. The reasoning is
+  that pruning the Certificate removes the only failing health check, but I have not seen
+  apps-wave-0 go Ready.
+- **Whether `http://` restores client connectivity.** PiHole resolves the name and Plex serves 32400
+  (both measured), but I have not watched a Plex client complete a connection over the advertised
+  URL, and I did not re-check whether plex.tv publishes it.
+- **THE REMOTE BADGE — the symptom that started this.** `LanNetworksBandwidth` is set, but the log
+  still tags traffic `Allowed Network (WAN)`: **82 WAN, 0 LAN**, with entries continuing after the
+  pref was written. So either Plex needs a restart for it to take effect, or that log tag is not
+  driven by this pref at all. **I do not know which, and the user-visible badge is the only real
+  test** — it is in the Plex client, which I cannot see.
+- **The Hubble flow export is NOT active.** The cilium agents are still the pre-merge pods, so the
+  `filePath: stdout` question is still open and no flow has been exported. The Cilium rollout — and
+  its LAN-DNS risk, since prod-3 holds the Gateway, PiHole-DNS and Plex VIPs — is still AHEAD.
+- **Whether a cert-manager exemption is the right shape**, or whether it widens
+  `secret-provenance` more than the operator wants. It is `platform/kyverno/**`, a sensitive path,
+  and deliberately not attempted here.
+- **Whether the pruned Certificate leaves residue** — I did not check for orphaned
+  `plex-vip-<random>` Secrets or CertificateRequests after the prune. Note Kyverno DENIED every one
+  of those Secrets, so there may be nothing to clean, but that is an inference, not a check.
+- **Whether `tls-lifecycle-automated` would also have objected to the Certificate living in a
+  `wait: false` Kustomization** (the recommended re-enable shape). It demands a reconciling Flux
+  path, which that provides, but I did not test it.
+- **The unencrypted advertised URL.** `http://` is tolerable only while remote access does not work
+  (no WAN forward). Nothing enforces that assumption, and nothing will complain if `http` is still
+  there the day the port-forward appears.

decisions/2026-10-04-plex-vip-hostname-and-lan-networks.md

@@ -0,0 +1,151 @@
+---
+date: 2026-10-04
+branch: feat/plex-vip-hostname-and-lan-networks
+title: Plex — one advertised hostname for local and remote, its own LE certificate, and a LAN definition so LAN clients stop being reported as remote
+---
+
+## What changed
+
+Two operator-reported symptoms, after the 2026-10-04 "server missing from the account" fix
+(decisions/2026-10-04-plex-advertise-plexdirect.md) had restored publishing:
+
+1. **LAN clients were shown as Remote.** Root cause, measured: `LanNetworksBandwidth` was **ABSENT**,
+   so Plex had no definition of a local network and classified *everything* as WAN — **468 requests
+   logged `Allowed Network (WAN)` and ZERO `(LAN)`**, including Seerr dialling from inside the
+   cluster. `allowedNetworks` (the `ALLOWED_NETWORKS` env) only waives **auth**; it does not make a
+   client local. Two different settings, and only one of them has an env hook.
+2. **The advertised URL was a single-purpose, fragile value.** The previous fix advertised
+   `https://<dashed-ip>.<serverhash>.plex.direct:32400`, which works, but hard-codes the VIP *and*
+   the server's certificate hash into a cluster-var — two values nothing could keep in sync.
+
+**Operator design adopted (2026-10-04): ONE hostname, `plex-vip.${BASE_DOMAIN}`, serving both
+audiences by split-horizon DNS** — PiHole answers it with `${PLEX_L4_VIP}` on the LAN, Cloudflare
+answers it publicly with the WAN address, NATed to the same VIP.
+
+- **`catalog/plex.yaml`:** `ADVERTISE_IP` → `https://plex-vip.${BASE_DOMAIN}:32400`. New
+  `lifecycle.postStart` hook (level-triggered, same idiom as pihole's and qbittorrent's) that
+  re-asserts on **every** start: `LanNetworksBandwidth`, and the three `customCertificate*` prefs
+  after converting the PEM to the PKCS#12 bundle Plex requires. New `PLEX_TLS_CRT`/`PLEX_TLS_KEY`
+  env from the `plex-vip-tls` Secret, both `optional: true`.
+- **`apps/wave-0/plex-cert.yaml` (new, hand-written):** a cert-manager `Certificate` for exactly
+  `plex-vip.${BASE_DOMAIN}` from ClusterIssuer `letsencrypt-prod`. Non-catalog because the schema has
+  no field for a Certificate. A single name, not the wildcard: the wildcard's Secret lives in
+  `gateway` and Secrets are not readable across namespaces, and re-requesting `*.${BASE_DOMAIN}`
+  would burn a duplicate-certificate slot for no benefit.
+- **`schemas/servicedefinition.schema.json` → 1.11.0:** adds `optional` to
+  `env.valueFrom.secretKeyRef` / `configMapKeyRef`. **This is load-bearing, not tidiness.** The
+  Certificate must be issued into the `plex` namespace, and that namespace is created by the
+  rendered `apps/wave-0/plex.yaml` itself — so it cannot be issued in an earlier tier and lands in
+  the *same* reconcile as the workload. With hard refs, Plex would sit in
+  `CreateContainerConfigError` until a certificate issued, and `apps-wave-0` is `wait: true` with
+  `apps-wave-1` dependsOn it, so a slow or failed DNS01 challenge would have **stalled the whole app
+  tier** — the hazard `catalog/plex.yaml` already documents for its GPU resource.
+- **`catalog/pihole.yaml`:** `address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}` added to
+  `FTLCONF_misc_dnsmasq_lines`. Required, not cosmetic: measured 2026-10-04,
+  `plex-vip.${BASE_DOMAIN}` resolved to **`192.168.2.20`** (the Gateway VIP) via the zone wildcard,
+  and a connect to `:32400` there returned **`http=000`**.
+- **`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml`:** `PLEX_ADVERTISE_URL`
+  **deleted** — the hostname makes it unnecessary, which removes the cert-hash fragility outright.
+  `PLEX_L4_VIP`'s note now records its second consumer (pihole).
+- **`catalog/render.py`:** `plex-cert` EXTRA_UNITS entry; **`apps/wave-0/kustomization.yaml`:**
+  `plex-cert.yaml` hand-added (NOT via `--emit-waves`, which would also onboard `keycloak` and
+  `vm-demo` — the pre-existing drift recorded on 2026-10-03).
+
+**A correction to my own earlier reasoning, recorded because it was wrong in the handoff log.** The
+previous entry asserted a bare-IP URL "can NEVER validate" and treated a SAN match as a
+*requirement*. The operator pointed out it is not: `httpsRequired=0` and `secureConnections` default
+to Preferred, so clients can fall back to cleartext. The measurements still stand (a validating
+client did fail, and real clients did send `certificate unknown`) — but the correct framing is that a
+matching certificate keeps the remote leg **encrypted**, not that it is required for Plex to
+function. That is why this branch obtains a real certificate for the advertised name rather than
+choosing the name to suit Plex's built-in one.
+
+**SENSITIVE PATHS — touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK until an
+approving review (or the operator's `/approve-override <reason>` posted as a PR *review* of type
+"Comment"). Deliberately not self-posted: CLAUDE.md §1.**
+
+## What I verified
+
+Live prod, 2026-10-04, read-only. Secrets per CLAUDE.md §6: prefs were read by name-scoped grep,
+the server token was used inside the pod and never printed, and no `Preferences.xml` was dumped.
+
+- **The LAN-classification root cause, directly:** `LanNetworksBandwidth=<ABSENT>`, and request
+  tagging of **468 × `Allowed Network (WAN)` vs 0 × `(LAN)`** — including `10.0.2.33` (Seerr), an
+  in-cluster pod. This is the measurement the whole hook rests on.
+- **All four prefs the hook sets EXIST on this build**, read from the server's own `/:/prefs`
+  (token-authenticated, output filtered to ids): `LanNetworksBandwidth`, `customCertificatePath`,
+  `customCertificateKey`, `customCertificateDomain`, all `type="text"`. I had earlier been unable to
+  confirm this (an unauthenticated `GET /:/prefs` returns **401**) and said so rather than assuming.
+- **There is NO env path for these settings** — the reason the hook exists. Grep of the image's
+  entire init surface: `setPref` is called for exactly six prefs (`MachineIdentifier`,
+  `ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`,
+  `TranscoderTempDirectory`) and the only `PLEX_*` env it reads are CLAIM/CLAIM_FILE/UID/GID/
+  UPDATE_CHANNEL and five INFO/HOME vars. No `PLEX_PREFERENCE*` of any kind; zero files mention
+  `LanNetworks`.
+- **The DNS gap was real:** `plex-vip.${BASE_DOMAIN}` resolved to `192.168.2.20` (Gateway VIP), and
+  `https://192.168.2.20:32400` returned `http=000` (nothing listening). PiHole's wildcard is why.
+- **`openssl` is present in the image** (`/usr/bin/openssl`, 3.0.13) — the hook's PEM→PKCS#12
+  conversion has a tool to run.
+- **The issuer exists and works:** ClusterIssuer `letsencrypt-prod` `READY=True` (28d), and the
+  existing `gateway/wildcard-home` Certificate is `Ready`/not-expired, which is what makes a DNS01
+  issuance for a new name in this zone a reasonable expectation.
+- **The renderer's substitution guard caught a real defect in my own work**, which is recorded
+  because it is the second time this class of bug has appeared here: my hook's explanatory comment
+  contained brace-wrapped names, and `assert_substitution_safe` refused to emit
+  (`'PX' is not a declared cluster-vars key — envsubst will replace it with the EMPTY STRING`).
+  Reworded; the rendered hook now contains exactly one brace-wrapped name, `BASE_DOMAIN`, verified
+  by parsing the emitted manifest.
+- **A `set -u` bug I introduced and then removed.** With `optional: true` the TLS env vars are
+  genuinely UNSET when no certificate has issued; under `set -u` the first `[ -n ... ]` would abort
+  the hook, and a postStart hook exiting non-zero makes kubelet KILL the container — the exact
+  failure `optional` exists to prevent. Verified by parsing the rendered manifest that no `set -u`
+  **command** remains (only two mentions in comments).
+- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
+  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
+  **pass**; the rc 3 is the two known environmental skips (`gitea-settings` — no local
+  `SETTINGS_API_TOKEN`; `conformance` L2/L3 — `CLUSTER_AVAILABLE` unset). **Not green. Reported as
+  rc 3.** `l0` passing is the specific proof the schema edit admits `optional` and still validates
+  every catalog file; `render-golden` passing is the proof catalog and committed goldens agree.
+- **Only plex and pihole goldens moved.** No other service's manifest changed, and no service was
+  silently onboarded.
+
+## What I did NOT verify
+
+- **THE OUTCOME, ENTIRELY. Nothing in this branch has been observed working.** It is unreconciled:
+  no Certificate exists in prod, `plex-vip` still resolves to the Gateway VIP, `LanNetworksBandwidth`
+  is still absent, and the hook has NEVER RUN. Every claim above is about the inputs, not the result.
+- **Whether the certificate will actually ISSUE.** DNS01 for a new name in this zone is inferred from
+  the wildcard's success, not demonstrated. If the Cloudflare credential or solver is scoped in a way
+  that fails for this name, Plex will start (by design) with no TLS and the advertised `https://` URL
+  will be unservable — i.e. **this branch could leave the advertised URL broken in a new way**. First
+  reconcile must be watched.
+- **Whether `openssl pkcs12 -export` produces a bundle PLEX ACCEPTS.** OpenSSL 3 defaults to AES-256
+  + SHA-256, which some consumers reject. Untested against Plex. If it is rejected, Plex may fail to
+  load its TLS listener — the hook guards against a *missing* file, NOT against a present-but-
+  unreadable one. **This is the single largest unverified risk in the change.**
+- **Whether the certificate prefs take effect without a restart.** Plex may need to re-initialise its
+  listener after `customCertificatePath` changes. If so, the first reconcile applies the prefs and
+  the SECOND start serves the cert. Converges either way, but the first start may look wrong.
+- **Whether `LanNetworksBandwidth` actually flips the Remote badge.** It is the correct setting by
+  Plex's own documentation and the measured absence explains the symptom exactly, but I did not set it
+  live and observe a client reclassify.
+- **SNAT makes the LAN definition coarser than it looks.** With `externalTrafficPolicy: Cluster`, Plex
+  sees the ingress NODE's address for everything arriving via the VIP, so once `10.0.0.0/8` and
+  `192.168.0.0/16` are LAN, GENUINE REMOTE clients arriving that way will ALSO be classified local
+  and remote bandwidth limits will not bind on them. This is the metering consequence already
+  accepted on 2026-10-03, now made stronger rather than weaker. Not measured.
+- **The public Cloudflare record for `plex-vip` is off-repo and absent from this branch**, as is the
+  WAN tcp/32400 port-forward. Remote access cannot work until the operator creates both; nothing here
+  can verify either, and this repo has no off-LAN vantage point.
+- **The private key is exposed in the container's environment** (`/proc/1/environ`) because the schema
+  and renderer support no secret VOLUME — measured: `storage` accepts only
+  [class, export, mountPath, name, readOnly, size, subPath] with `additionalProperties: false`, and
+  no rendered app mounts a Secret. A volume would be the better shape; adding secret-volume support
+  is the clean follow-up and I did not do it.
+- **The PKCS#12 passphrase is a hardcoded literal** (`plexvip`) in the rendered manifest. It protects
+  nothing — Plex needs it in cleartext in `customCertificateKey` and it sits beside the bundle on the
+  same PVC — but it is worth knowing it is not a secret and was never treated as one.
+- **Four places must agree and nothing enforces it:** `ADVERTISE_IP`, the hook's
+  `customCertificateDomain`, `plex-cert.yaml`'s `dnsNames`, and PiHole's answer. No gate stage or
+  conformance test checks that they match, or that PiHole's `plex-vip` answer still equals
+  `${PLEX_L4_VIP}`.

decisions/2026-10-05-agents-work-in-their-own-workspace.md

@@ -0,0 +1,147 @@
+---
+date: 2026-10-05
+branch: docs/agents-work-in-their-own-workspace
+title: One workspace per agent run — the shared checkout is now documented as off-limits
+---
+
+## What changed
+
+Agents have been doing their work in the operator's checkout at
+`/var/home/keith/source/home-network-k3s-impl` and clobbering each other. Nothing in the tree
+told them not to: worktrees appeared in CLAUDE.md only as §3's *gate workarounds*, i.e. as an
+awkward edge case, and `decisions/README.md` asserted worktree isolation as a fact about how
+agents already behave rather than as a rule they must follow. Nowhere was the rule stated.
+
+Documentation only — no code, no manifests, no pipeline logic.
+
+- **`CLAUDE.md`: new `## 0. Work in your OWN workspace. Never in the shared checkout.`**
+  Placed before §1 so the section numbers every other file and handoff entry cites (§1, §2b,
+  §3, §7) all keep their meaning. It gives the three commands (`git fetch origin main`;
+  `git worktree add -b <type>/<slug> <path outside the repo> origin/main`; `cd`), an explicit
+  do-not list for the shared checkout (no `checkout`/`switch`, no edits, no
+  `add`/`commit`/`stash`/`restore`, no removing others' worktrees), a measurement table, the
+  "put it outside the repo root" rule, cleanup, what to do if you are already standing in the
+  shared checkout with edits, and what a PM must put in a brief it writes for another agent.
+- **`CLAUDE.md` intro** now says "Start with §0", with the reason: it is the rule you can
+  break before you have read anything else.
+- **`CLAUDE.md` §3** retitled "Running the gate from a worktree" and reframed as the normal
+  path rather than an edge case, since §0 now makes every gate run a worktree run. Its list
+  grew from two breakages to three: `spec-immutability`'s skip is *avoidable*, not merely
+  tolerated, if the `ENGINE` shim also mounts the sibling spec repo read-only and passes
+  `-e SPEC_REPO=/home-network-k3s`.
+- **`CLAUDE.md` §4** gained "One branch, one workspace, one PR".
+- **`README.md`**: a fourth non-negotiable rule ("No work in a shared checkout"), and
+  "Working on this repo" now opens with how to get out of the checkout before describing the
+  dev container that runs in it.
+- **`decisions/README.md`**: the worktree-isolation sentence now cites §0 as the requirement
+  instead of describing it as something agents happen to do.
+
+I did this work the way the new §0 requires — a worktree at
+`<session scratchpad>/wt-workspace-rule`, outside the repo root, branched from `origin/main`,
+with the shared checkout left on the branch and with the untracked files I found in it.
+
+## What I verified
+
+Every number in the new §0 table was measured in the shared checkout on 2026-10-05, not
+recalled:
+
+| Claim | Command | Value |
+|---|---|---|
+| shared HEAD is yanked between branches | `git reflog \| grep -c 'checkout: moving'` | **107** |
+| it is not sitting on `main` | `git worktree list` | `feat/pm-agent-issue-to-pr` |
+| other runs' uncommitted work is in it | `git status --porcelain` | **2** trees: `infra/agent/` (22:04), `platform/pm-agent/` (22:09) — `stat -c %y` |
+| its `main` is stale | `git rev-parse main` vs `origin/main` | `4386d14` vs `22c94ef`, **2 behind** |
+| worktrees registered / already merged | `git worktree list --porcelain` + `git merge-base --is-ancestor <b> main` | **42** branches, **31** merged |
+| nested worktrees are a gate hazard | `grep -rl '\.claude' conformance/tests platform/monitoring/tests` | **9** scripts carry an explicit exclusion |
+| the 18× double-count | read `platform/monitoring/tests/run.sh:115-123` | in-tree comment, observed 2026-09-12 |
+
+**The failure mode was observed live, mid-session, not just reconstructed.** The two untracked
+trees I measured at 22:04 and 22:09 were committed at 22:20 by another run — `5334abe`
+`feat(pm-agent): an in-cluster AI Program Manager...`, 1,000+ lines across `infra/agent/`,
+`clusters/`, `catalog/`, `apps/`, `docs/plans/` and its own `decisions/` entry — onto the shared
+checkout's checked-out branch. Measured after the fact: the shared checkout's HEAD moved
+`22c94ef` → `5334abe` and its working tree went clean between my first `git status` and my last,
+with `git log -1` naming a commit that is not mine. That is §0's "shared HEAD" row happening
+during the session that wrote §0, and it is now cited in the doc.
+
+**A claim of mine was wrong and is corrected in the doc.** I had it recorded that
+`git worktree remove` is blocked by the agent sandbox. Provoked rather than assumed: created
+a throwaway detached worktree in the scratchpad and removed it — **rc 0**, and
+`git worktree list` no longer shows it. §0 therefore tells agents to remove their own, and
+scopes the "blocked" note to what it actually covers: *other* sessions'
+`.claude/worktrees/agent-*`, some of which are additionally `locked` on git's own terms. The
+`rework/<pr-number>` escape hatch is now scoped to someone else's stale worktree holding a
+branch, which is a real and separate problem (31 of them). Both of my own worktrees in this
+session were removed cleanly at rc 0 after pushing.
+
+`spec-immutability` is avoidable-not-tolerated: taken from
+`decisions/2026-10-01-render-selftest-runs-in-gate.md`, which records
+`stage spec-immutability: pass` under the two-mount shim plus `-e SPEC_REPO=/home-network-k3s`.
+
+**Gate — `rc 3`, UNDETERMINED, and I am not calling it green.** Pinned toolbox, from this
+worktree, `DEV_KUBECONFIG=/dev/null` and an `ENGINE` shim bind-mounting both the main
+checkout's `.git` (at its own host path) and the sibling spec repo read-only with
+`-e SPEC_REPO=/home-network-k3s`:
+
+    ENGINE=<two-mount shim> DEV_KUBECONFIG=/dev/null       ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'   # rc 3
+
+| stage | result |
+|---|---|
+| spec-immutability | **pass** — it genuinely RAN; the second mount is what makes it run from a worktree |
+| approval-policy | pass, but **N/A**: `event='none'`. It cannot see a PR locally, so this green says nothing. See below |
+| decisions-log | pass — the three required headings and the frozen-archive pin |
+| lint · l0 · render-golden · l1 | pass (L1: 3 offline checks, 0 failures, 0 skips, 0 unwired) |
+| gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset; it exists only in CI |
+| conformance (L2/L3) | SKIPPED — `CLUSTER_AVAILABLE!=true`; the decision notes no cluster-needing requirement was touched |
+
+`gitea-settings` is the *only* stage that did not run, and it cannot be made to run from a
+workstation, so rc 3 is the ceiling for any local run — not a defect in this branch. The
+first gate attempt returned rc 3 for a different and real reason (`HEAD is not ahead of
+origin/main`: I had not committed yet, so the touched-requirement set was empty by default
+rather than by verification); it was re-run after committing, and the table above is the
+post-commit run. §3 now records both expected skips, because it previously named only
+`spec-immutability`.
+
+## What I did NOT verify
+
+- **GATE: UNVERIFIED at the time of writing.** The pinned-toolbox result for this branch and
+  its exit code are reported in the PR, not predicted here. **A local rc 3 is not green** and
+  will not be described as one.
+- **CI gate: RED, measured — `MERGE GATE: BLOCKED`, one failing stage.** Run 245 on PR #56,
+  head `b988da8b`, 1m47s. The only failure is `approval-policy`: *"a sensitive change lacks a
+  non-author approval"*, `sensitive paths touched: CLAUDE.md`, author `gitea-admin`. Every other
+  stage passed — `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, and `gitea-settings`
+  (which can only run in CI, and reported `required_approvals = 0`,
+  `dismiss_stale_approvals = true`). `spec-immutability` SKIPPED in CI (no sibling spec repo in
+  the CI workspace — the inverse of the local run, where the two-mount shim made it pass) and
+  `conformance` SKIPPED. **Note `gate / conformance (pull_request)` reports status `success`
+  with description "Has been skipped"** — CLAUDE.md §2b's trap, live on this very PR. Nothing
+  in this branch is broken; the red is the structural human gate, below. *Measured at
+  `b988da8b`; this entry was then amended to record it, so the head SHA in the PR is newer than
+  the run quoted here and CI re-ran on the amended commit.*
+- **This PR is a sensitive-path PR and is CI-RED on `approval-policy` until a human acts.**
+  `CLAUDE.md` is in `infra/gitea/approval-policy.json`'s `sensitive_paths` ("the delivery
+  contract agents follow"), so the stage requires an approving review from someone who is not
+  the author, or an `/approve-override <reason>` posted by an administrator **as a PR review
+  of type "Comment"**, not in the plain comment box. `ci/approval-policy.sh` run locally
+  reports N/A (it cannot see a PR), so the local gate cannot catch this and a green local run
+  says nothing about it. **I have not posted an override and must not** — CLAUDE.md §1 forbids
+  an agent clearing this on its own PR even though `gitea-admin` holds `owner` and could.
+- **Nothing here is enforced.** §0 is prose. There is no gate stage that detects work done in
+  the shared checkout, no check that a worktree is outside the repo root, and no check that a
+  branch was cut from `origin/main`. Every other rule in CLAUDE.md that agents kept breaking
+  (`DECISIONS-LOG.md` appends, the three handoff headings) ended up as a gate stage, and that
+  is the honest follow-up here. I did not write one: the ask was documentation, and an
+  enforcement stage lands in `ci/**`, which is itself a sensitive path and deserves its own PR.
+- **I did not clean up the 31 stale merged worktrees or the 35 nested under
+  `.claude/worktrees/`.** They are other sessions' workspaces; removing them is an operator
+  action, and §0 explicitly tells agents not to remove worktrees they did not create.
+- **I did not touch the untracked `infra/agent/` and `platform/pm-agent/` trees** sitting in the
+  shared checkout. They are another run's in-flight work. Whose, and whether they are wanted,
+  is unknown to me.
+- **`docs/plans/ssd-etcd-execution-runbook.md:18` still points at a `.claude/worktrees/` path**
+  inside the repo. Left as-is deliberately: that runbook is marked COMPLETED/historical and
+  describes the now-decommissioned dev cluster, so the path is a record of what was done, not
+  an instruction. It does mean a grep for in-repo worktree paths is not yet clean.
+- **No runtime, cluster or Flux behaviour was touched or checked.** This branch changes three
+  Markdown files; `kubectl` was not run.

decisions/2026-10-05-cilium-roll-out-on-config-change.md

@@ -0,0 +1,88 @@
+---
+date: 2026-10-05
+branch: feat/cilium-roll-out-on-config-change
+title: Cilium config changes were silently inert — roll the agents when cilium-config changes, which also finally activates the Hubble exporter
+---
+
+## What changed
+
+**A merged Cilium config change did nothing, and everything reported success.** The Hubble flow
+exporter (PR #50, merged 2026-10-04) wrote its settings into `cilium-config` and never took effect.
+
+| Measured 2026-10-05, over a day after the merge | |
+|---|---|
+| HelmRelease `cilium` | `Ready=True`, `UpgradeSucceeded`, *"Helm upgrade succeeded for release kube-system/cilium.v6"* |
+| `cilium-config` keys | `hubble-export-file-path: stdout`, `hubble-export-allowlist: {"destination_port":["32400"]} {"source_port":["32400"]}` — **all present** |
+| cilium agent pods | **all three still the PRE-MERGE pods**, `startTime 2026-10-04T07:11Z` |
+| exported flow lines in agent stdout | **0, 0, 0** |
+| Flux Kustomizations | all `Ready=True` |
+
+Cause: the chart's `rollOutCiliumPods` defaults to **`false`** (cilium 1.20.1 `values.yaml`: *"Roll
+out cilium agent pods automatically when configmap is updated"*), so Helm rewrites the ConfigMap and
+leaves the DaemonSet's pod template untouched — and **cilium-agent reads that ConfigMap only at
+startup**. The feature was merged, believed delivered, green everywhere, and doing nothing at all.
+
+That is the same class of failure this repo already guards against elsewhere: a green signal sitting
+over something that never ran.
+
+- **`platform/cilium/helmrelease.yaml`** (authoritative): `rollOutCiliumPods: true`.
+- **`platform/cilium/values.yaml`** (reference mirror) and
+  **`infra/ansible/roles/cilium/templates/cilium-values.yaml.j2`** (the `[parity]` source of truth):
+  the same, so a bootstrap render does not differ from what Flux reconciles.
+
+**The cost is real and is the point.** Every future Cilium config change now triggers a cluster-wide
+agent rollout instead of waiting for someone to remember a manual restart. That is strictly more
+honest — the change applies when merged, or it fails visibly — but it makes config edits disruptive
+by default, so they want a quiet moment. Mitigation already in place:
+`updateStrategy.rollingUpdate.maxUnavailable: 1` means the DaemonSet will not proceed past an unready
+agent, so a bad render costs one node rather than the cluster.
+
+**Merging this is itself one rollout**, because adding the ConfigMap checksum annotation changes the
+pod template — and that rollout is what finally activates the Hubble exporter.
+
+## What I verified
+
+- **Every row of the table above**, read from the live HelmRelease status, the live `cilium-config`
+  ConfigMap, the agent pods' `startTime`, and a grep for `"flow"` in each agent's stdout.
+- **The chart value name, default and semantics**, quoted from cilium **v1.20.1**'s own
+  `values.yaml` — the version actually running here (`quay.io/cilium/cilium:v1.20.1`), not a guess
+  from another release: `rollOutCiliumPods: false` with the comment *"Roll out cilium agent pods
+  automatically when configmap is updated."*
+- **All three value files carry the key** and both YAML files still parse, with
+  `spec.values.rollOutCiliumPods` resolving to `true`.
+- **The blast-radius mitigation is genuinely present**, not assumed:
+  `updateStrategy.rollingUpdate.maxUnavailable: 1` is already set in the HelmRelease, with a long
+  existing comment explaining that a CNI rollout has no healthy-peer fallback.
+- **THE MECHANISM IS CONFIRMED BY CONTRAST, not just reasoned.** The cgroup fix (PR #42,
+  2026-10-03) DID take effect: both the DaemonSet and the running pod mount
+  `cilium-cgroup hostPath=/sys/fs/cgroup`, matching what the HelmRelease declares. The difference is
+  exactly the one claimed above — #42 changed the POD TEMPLATE (a volume hostPath), which rolls the
+  DaemonSet on its own, while #50 changed ONLY the ConfigMap, which does not. It also explains the
+  agents' `startTime 2026-10-04T07:11Z`: that was #42's rollout, and nothing has rolled them since.
+  So the inertness is specific to ConfigMap-only changes, which is precisely what this flag fixes.
+- **Flux is healthy before this change** — every Kustomization `Ready=True`, so this lands on a clean
+  graph rather than into an existing cascade.
+- **Toolbox gate**: recorded in the PR.
+
+## What I did NOT verify
+
+- **THE OUTCOME, ENTIRELY. Unreconciled.** No agent has rolled, no flow has been exported. This
+  change is an argument that the rollout will happen, not evidence that it did.
+- **Whether `filePath: stdout` actually works** — still the open question from #50, and it cannot be
+  answered until the agents restart. If the literal is not special-cased, the agent may create a
+  FILE named `stdout` and nothing reaches Loki: a silent no-op, not a crash. Fallback documented in
+  #50: keep the chart's default path and add a `loki.source.file` target to Alloy with a hostPath
+  mount. **Check for flow lines in the agent logs immediately after the rollout.**
+- **THE ROLLOUT'S IMPACT ON LAN DNS IS UNMEASURED AND IS THE MAIN RISK.** The node holding the L2
+  lease for the Gateway VIP, **PiHole's DNS VIP** and Plex's VIP loses them briefly as it cycles.
+  `maxUnavailable: 1` bounds it to one node at a time, and BPF maps persist across an agent restart,
+  but I have not measured how long a VIP is unreachable during a lease move, nor whether the two
+  PiHole replicas cover the gap. Merge it at a quiet moment.
+- **Whether enabling this is the operator's preferred trade.** The alternative — leaving it `false`
+  and restarting by hand — keeps config edits non-disruptive at the cost of them being silently
+  inert. I judged silent inertness the worse failure, but it is a judgement, not a measurement.
+- **Whether anything ELSE in `cilium-config` is currently inert.** I checked the one other recent
+  Cilium change and it is fine (see below), but I did not audit every key against the running
+  agents' actual state.
+- **`hubble.relay.rollOutPods` / `hubble.ui.rollOutPods`** are left at their `false` defaults. Both
+  have the same silently-inert property for their own ConfigMaps; not addressed here.

decisions/2026-10-05-oauth2-sidecar-scrape-verified.md

@@ -0,0 +1,90 @@
+---
+date: 2026-10-05
+branch: docs/oauth2-sidecar-scrape-verified
+title: The gate-sidecar scrape is verified working on prod — and four places in the tree still said it wasn't
+---
+
+## What changed
+
+Documentation only. No manifest, no rendered output, no behaviour.
+
+PR #49 merged as `21c88b0` and the fix works, but four places in the tree still asserted the
+*pre-fix* state as current — in the present tense, in capitals, in the files a reader would consult
+first. This is the stale-documentation failure mode this repo has been burned by twice already
+(`git push origin main` in the runbooks, "append to DECISIONS-LOG.md" in the old handoffs), so:
+
+- **`platform/oauth2-proxy/podmonitor.yaml`** — the banner read *"READ THIS BEFORE BELIEVING THE
+  TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY"* and *"SO THE TARGETS BELOW WILL
+  READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS"*. Both now false, and the first one actively
+  instructs the reader to disbelieve a correct `up == 1`. Replaced with a `STATUS: THE SCRAPE WORKS`
+  block carrying the post-fix numbers; the diagnosis is kept below it under an explicit `HISTORY`
+  heading, because it is why the fix is shaped the way it is. Added the drop fingerprint to
+  recognise if it regresses — `lastScrapeDuration` pinned at *exactly* the `scrapeTimeout` with
+  `context deadline exceeded`, versus 1–6 ms when healthy.
+- **`catalog/render.py:159`** — *"THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED … Do not
+  read this flag as 'metrics are working'."* Rewritten to enumerate the three pieces a working
+  scrape needs and which PR landed each. The warning is **kept, deliberately**: the flag alone
+  really is insufficient, and "1 without 3" is exactly what caused the 63.7h outage. Only the
+  "not yet scraped" claim was false.
+- **`catalog/render.py:1365`** — the sidecar arg comment, same correction, one line.
+- **`platform/oauth2-proxy/README.md`** — its two observability subsections are *dated*
+  ("measured 2026-09-30", "Update 2026-10-01"), so they are honest records rather than lies. But a
+  reader hits the stale body long before any correction, so a `> SUPERSEDED` pointer now sits
+  directly under the heading, and an `### Update 2026-10-05` section at the end carries the
+  before/after table, the TSDB series counts and the alert outcome.
+
+Two substantive errors fixed while there, not just tense:
+
+1. The README's *"To close the gap for an app, three things are needed"* listed **a Service port
+   and a ServiceMonitor**. Wrong — a PodMonitor targets pod ports directly, which is the whole
+   reason no Service exposes 44180 and the renderer needed no Service change. Following that
+   paragraph would have sent someone to edit `catalog/render.py`'s Service rendering for no reason.
+2. `podmonitor.yaml` ended by deferring the fix to *"the catalog/renderer, which is in flight on
+   another branch"* — that branch does not exist and never did; the work happened in PR #49. It now
+   names `r_monitoring_ingress_policy` directly, and a new `CRAFTY` section explains why crafty is
+   the one namespace this file cannot help.
+
+## What I verified
+
+- **The fix itself, on prod 2026-10-05, read-only** (this is the evidence the docs now quote):
+  Flux applied `21c88b0` through the real path — `apps-wave-0` `lastAppliedRevision` at 21:37:01Z,
+  `apps-wave-1` at 21:37:46Z, in dependency order — and all five `<app>-allow-monitoring-scrape`
+  CNPs exist, each `ports=['44180']`, `from=[['monitoring']]`.
+- **All 6 targets `up == 1`**, `health=up`, `lastError=''`, `lastScrapeDuration` 0.001–0.006s —
+  against 10.0s and `context deadline exceeded` before.
+- **`oauth2_proxy_*` is in the TSDB for these namespaces for the first time:** 5 metric names,
+  148 series; `sum by (namespace) (oauth2_proxy_requests_total)` = qbittorrent 35072, radarr 23560,
+  sonarr 1284. (Queried with `count by (__name__)` plus a matcher, not label-values.)
+- **Alerts:** all 5 `TargetDown` and 5 of 6 `OAuth2ProxyGateUnobserved` resolved; one series
+  remains, `OAuth2ProxyGateUnobserved{namespace="crafty"}`, firing.
+- **The rendered output is byte-identical.** `catalog/render.py` changed only in comments, so this
+  branch touches no `apps/` file and no golden. Confirmed by `render-golden` and `l1` both passing
+  with no regeneration, and by `git status` showing nothing under `apps/`.
+- **This branch touches no sensitive path**, unlike PR #49: `platform/oauth2-proxy/**`,
+  `catalog/**` and `decisions/**` are none of them in `infra/gitea/approval-policy.json`. So it
+  should merge on a green gate with no human approval — the opposite of the PR it documents, and
+  the reason the two were kept apart.
+- `render.py --selftest`: PASS.
+- Toolbox gate: see the handoff's own rule — reported below rather than claimed here.
+
+## What I did NOT verify
+
+- **I did not re-measure the 63.7h figure or the ~22 emails/day rate**; both are carried forward
+  from the 2026-10-04 measurements, now quoted inside the docs as historical. The 15-day
+  `query_range` that established "one episode, not flapping" was run once, on 2026-10-04.
+- **`oauth2_proxy_requests_total` is still absent for jackett and pihole.** I assert in the README
+  that this is because the counter is not created until the gate serves its first request. That is
+  the documented behaviour of a Prometheus counter with no initialisation, and it is consistent
+  with the other three namespaces having traffic — but I did not force a request through jackett's
+  gate and watch the series appear. If it is still absent after real use, that is a finding.
+- **The auth-churn rule is still unwritten.** The README now says it is "writable rather than
+  decorative"; that is a claim about the metrics being present, not a rule anyone has authored or
+  unit-tested. The thing the 2026-09-30 incident actually needed does not exist yet.
+- **crafty remains unobserved.** I read its container spec on 2026-10-05 and confirmed no
+  `--metrics-address`; I did not attempt the fix, and nothing here changes its alert.
+- **I did not audit the whole tree for other stale "metrics are not scraped" claims** beyond the
+  four sites a targeted grep found (`not yet reachable`, `UNTIL AN INGRESS`, `up == 0`,
+  `not reachable by Prometheus` and neighbours). `runbooks/oauth2-proxy-auth-failures.md` is
+  described in the README as "the by-hand substitute meanwhile" and I left it alone — the by-hand
+  log procedure still works and is not wrong, but whether it should now be rewritten against the
+  new metrics is unexamined.

decisions/2026-10-05-plex-resolver-independent-advertise.md

@@ -0,0 +1,105 @@
+---
+date: 2026-10-05
+branch: fix/plex-resolver-independent-advertise
+title: Plex down LAN-wide — the advertised name existed only in PiHole, so clients that bypass it landed on the router's %D cert
+---
+
+## What changed
+
+**I took Plex down for the whole LAN.** Reported symptom: every internal client showed a certificate
+error with `%D` as the subject name and could not connect.
+
+**Root cause: the advertised connection depended on our own resolver.** `ADVERTISE_IP` was
+`http://plex-vip.${BASE_DOMAIN}:32400` — a name that exists ONLY in PiHole's split-horizon answers
+and in no public zone. A Plex client resolves the advertised host *itself*, with whatever resolver it
+happens to use, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default,
+phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve
+the name, the SOHO router answered the failed lookup with **itself**, and they all landed on the
+router's web UI.
+
+| Measured 2026-10-05 | Result |
+|---|---|
+| `192.168.2.1:443` (router, LAN side) | `C = ZZ, ST = Somewhere, O = %D342e7e6b, CN = %D` |
+| `136.51.115.252:443` (router, WAN side) | the same `%D` cert |
+| PiHole **pihole-0** | `plex -> 192.168.2.20`, `plex-vip -> 192.168.2.24` ✅ |
+| PiHole **pihole-1** | identical ✅ (both behind the `.23` VIP) |
+| Gateway `.20:443` cert | `CN = *.home.keithjackson.dev`, valid to 2026-12-12 ✅ |
+| `http://192.168.2.24:32400/identity` | `200` ✅ |
+| Plex's published connections | all port **32400** — nothing on 443 |
+
+**The DNS was right; the dependency on it was the bug.** PiHole answered correctly from both
+replicas the entire time. And because nothing Plex advertises listens on 443, the clients had plainly
+left the Plex path altogether and were talking to the router.
+
+**The fix: go back to `<dashed-ip>.<serverhash>.plex.direct`, which is RESOLVER-INDEPENDENT.** This
+is Plex's own mechanism and the reason it exists — `plex.direct` is a public zone that returns the
+private address embedded in the label, so *every* resolver (PiHole, the router, 1.1.1.1, a DoH
+browser) answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no split-horizon
+entry, no Cloudflare record and no certificate work, because the wildcard cert Plex already serves on
+32400 covers exactly that name.
+
+- **`catalog/plex.yaml`:** `ADVERTISE_IP` →
+  `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`, with the
+  resolver-independence requirement written at the top of the block as the first thing to check
+  before anyone changes this field again.
+- **PiHole's `plex-vip` answer is KEPT.** It is harmless, costs nothing, and is useful for reaching
+  the VIP by name from a LAN shell. It is simply not what Plex advertises.
+
+**Why not keep the nicer hostname.** It cannot be made correct for clients that bypass PiHole: a
+public record for it resolves to the WAN address, and with no NAT hairpin on this router a LAN client
+sent there fails too. The name is right only for PiHole users and wrong for everyone else, and there
+is no configuration of it that is right for both. `plex.direct` is right for both.
+
+**The honest cost, restated rather than buried:** the value embeds both the VIP and the server's
+certificate hash as a literal — envsubst cannot dash an IP and nothing in the tree can derive the
+hash, so nothing checks that they still agree. That fragility is exactly why the hostname was tried.
+It is accepted because an unresolvable name is a *worse* failure: silent, LAN-wide, and it points
+users at their router.
+
+## What I verified
+
+Live prod, 2026-10-05, read-only.
+
+- **Every row of the table above**, including the router's `%D` cert on both its LAN and WAN sides —
+  which is what identifies the router as what the clients were reaching.
+- **THE FIX ITSELF, with full TLS validation (no `curl -k`)**: the plex.direct name resolves to
+  `192.168.2.24` and `/identity` returns **200, TLS verified**. Re-checked today, not carried over
+  from yesterday's measurement.
+- **Both PiHole replicas are identical and correct**, with the same five `address=` lines live in
+  `/etc/pihole/dnsmasq.conf`, and both IPs present behind the `pihole-dns-tcp`/`-udp` endpoints. I
+  checked only pihole-0 at first and deliberately went back for pihole-1, because a split-brain pair
+  would have produced exactly this intermittent-looking symptom.
+- **The cluster was never the problem:** all eight LoadBalancer VIPs still assigned, L2
+  announcements intact (prod-3 holds `.20`/`.23`/`.24`), `wildcard-home` Certificate `Ready=True`,
+  and PR #51 had reconciled (`customConnections="http://plex-vip.home.keithjackson.dev:32400"` was
+  live, confirming the bad value was in effect).
+- **`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is still set**, so the postStart hook survived
+  the roll and keeps re-asserting.
+- **The Cilium rollout from #50 STILL has not happened** — agents are unchanged since
+  2026-10-04T07:11, so the Hubble export is inactive and `filePath: stdout` remains unproven.
+
+## What I did NOT verify
+
+- **THE OUTCOME. Unreconciled.** No client has been observed connecting after this change. The
+  argument is that the advertised name now resolves for every resolver and serves a matching
+  certificate — both measured — but I have not seen a Plex client complete a connection, and I
+  cannot see the user's clients at all.
+- **THE CLIENT-SIDE RESOLVER PATH IS INFERRED, NOT OBSERVED.** I could not query a public resolver
+  from here — my attempt returned NXDOMAIN even for a `plex.direct` name that certainly resolves, so
+  the probe was broken (egress policy or a missing tool) and I discarded it rather than report it.
+  The chain "client bypasses PiHole → NXDOMAIN → router answers with itself → `%D`" is the only
+  explanation consistent with all the measurements, but the NXDOMAIN-hijack step in particular is
+  deduced from the router serving `%D` on its LAN address, not from watching a lookup.
+- **Which clients bypass PiHole, or how many.** If in fact every client uses PiHole, then my
+  explanation is wrong and something else sent them to the router — in which case the next thing to
+  check is whether the clients were using `plex.home.keithjackson.dev` (port 443, the Gateway) rather
+  than an advertised connection at all.
+- **Whether plex.tv will re-publish promptly** after the value changes, and whether stale connection
+  lists are cached on clients. A client holding the old list may keep failing until it refreshes.
+- **Mixed content.** The previous `http://` value would also have been blocked by the browser for
+  anyone using `https://app.plex.tv`; that is fixed incidentally by returning to `https`, but I did
+  not test it.
+- **The Remote badge — the symptom that began this whole sequence — remains unconfirmed.**
+  `LanNetworksBandwidth` is set, but Plex still tags traffic `Allowed Network (WAN)`. Unchanged and
+  still unexplained.
+- **`ManualPortMappingMode="1"` with an absent port** is still there, untouched.

decisions/2026-10-05-pm-agent-issue-to-pr.md

@@ -0,0 +1,259 @@
+---
+date: 2026-10-05
+branch: feat/pm-agent-issue-to-pr
+title: An in-cluster AI Program Manager that turns Gitea issues into pull requests — authored, nothing deployed
+---
+
+## What changed
+
+A new service, `pm-agent`, that runs in the cluster, reads the issues of this Gitea
+repository, decides what to do about each thread **as the Program Manager persona this
+repository already works under**, and runs one worker agent per assignment that authors a
+change, runs the merge gate and opens a pull request. It reacts to issue creation and to
+comments; the assignment and all correspondence travel through the issue thread and the PR.
+It authenticates with an Anthropic **subscription** token. It never pushes to `main` and
+never merges.
+
+Design, alternatives and threat model: `docs/plans/pm-agent-issue-to-pr.md`. Operating
+surface: `platform/pm-agent/README.md`. Operator procedures:
+`runbooks/pm-agent.md`.
+
+**New — the agent itself** (`infra/agent/`): one image (`agent-runner`) layered on the pinned
+dev toolbox plus node and the Claude Code CLI, with two fixed entrypoints and no argument
+surface. `pm-dispatch` is the PM; `pm-worker` is one assignment. `pm-persona.md` is the PM
+persona, assembled from `CLAUDE.md`, the frozen log's PM guardrails and the spec repo's
+`agents/roles.md`. `worker-job-template.yaml` is the only Job shape the dispatcher may
+create, baked into the image rather than mounted, because that template *is* the shape the
+admission policy protects. `versions.env` pins only what the toolbox lacks — deliberately not
+a second copy of the toolchain pins. `build-image.sh` builds, verifies in-image as uid 65532
+on a read-only rootfs, and **never pushes**.
+
+**New — the deployment** (`platform/pm-agent/`): namespace (no `app-namespace` label — this
+is CI plumbing like `gitea-actions`; PodSecurity `restricted` is enforced), `rbac.yaml` (the
+dispatcher may `create/get/list/watch jobs` in its own namespace and nothing else; the worker
+identity is bound to **nothing** and mounts no token), the dispatcher Deployment (1 replica,
+`Recreate` — two dispatchers would double-trigger every thread), `configmap.yaml` (every knob,
+with its reasoning), `networkpolicy.yaml` (default-deny; egress to gitea:3000,
+kube-apiserver for the dispatcher only, tcp/443; **no ingress at all**), and
+`pm-agent-credentials.sops.yaml` holding **placeholders**.
+
+**New — the boundary** (`platform/kyverno/admission/pm-agent-job-shape.yaml`, `Enforce` +
+`failurePolicy: Fail`): Kubernetes places no `use` check on `serviceAccountName`, so "may
+create Jobs" is "may run arbitrary code as the namespace's most privileged identity". Six
+rules admit only the committed shape. It is listed in `platform-config/kustomization.yaml`,
+which is the only list that actually applies ClusterPolicies — adding it to
+`platform/kyverno/admission/kustomization.yaml` alone is a silent no-op, which is exactly how
+the conformance boundary shipped green and never reached the cluster on 2026-09-29.
+
+**Wiring:** `clusters/base/pm-agent.yaml` (dependsOn `apps-wave-0` — what it needs is Gitea,
+not Keycloak or the Gateway; `wait: false`, because the pod is *expected* to be unhealthy
+while its credentials and image digest are placeholders), its line in
+`clusters/base/kustomization.yaml`, a SOPS decryption patch in **both** overlays, and an
+`inClusterClients` entry for the `pm-agent` namespace on `catalog/gitea.yaml` with
+`apps/wave-0/gitea.yaml` re-rendered by the canonical command.
+
+Three design choices worth naming, with the alternatives recorded in the plan: it **polls**
+rather than receiving webhooks (Gitea's webhook SSRF guard, lost deliveries, and no need for
+inbound reach); it keeps **no state** — the trigger is "the newest thing in this thread was
+not written by me", so the thread is the state and there is no PVC and no backup obligation;
+and the **model holds no tools** — its reply is one JSON object that this code validates, so
+nothing in the path executes text.
+
+## What I verified
+
+All offline, in the pinned toolbox (`./infra/devcontainer/dev.sh run`), never on the host.
+
+- **`kyverno test platform/kyverno/tests/pm-agent` → 28 passed, 0 failed** under the pinned
+  CLI (v1.19.0). The first run was **27 passed, 1 failed** and caught a real defect: the
+  volume rule asserted `length(emptyDir) != 0`, which denies the valid `emptyDir: {}`. Fixed
+  by asserting the volume's **key set** instead, which also denies every other source type
+  without enumerating them. The suite includes the two cases a careless policy admits — a
+  second container while `containers[0]` is compliant, and a volume *named* `workspace` whose
+  source is a `hostPath`.
+- **`./ci/gate.sh` → exit 3, MERGE GATE: UNDETERMINED. Not green, and not reported as
+  green.** Per-stage, quoted from the run: spec-immutability **pass** (the spec repo is a
+  sibling of this checkout and is clean at the pinned SHA), decisions-log **pass**, lint
+  **pass**, l0 **pass**, render-golden **pass** (10 services reproduce), l1 **pass**,
+  approval-policy **pass** (N/A off a pull-request event), conformance **declared-skip** (no
+  cluster — Phase 0), and `gitea-settings` **SKIPPED**: `SETTINGS_API_TOKEN` is not set on
+  this workstation, so branch protection and runner registration are UNVERIFIED by this run.
+  The gate names that single stage as the reason for rc 3 ("Stages that did not run:
+  gitea-settings"), so nothing in the change is unexamined — but rc 3 is not a pass and this
+  entry does not call it one.
+- **The PR's OWN CI run, read from its log rather than from the status tick** (PR #57, head
+  `8b5e950b`, `.gitea/workflows/gate.yml` with `GATE_STRICT_SKIPS=true` and
+  `REQUIRE_GITEA_SETTINGS=true`): decisions-log **pass**, lint **pass**, l0 **pass**,
+  render-golden **pass**, l1 **pass**, **gitea-settings pass** — including
+  `block_admin_merge_override = true`, which is the one stage my workstation cannot run at
+  all — spec-immutability **SKIPPED** (the spec repo is absent in CI, as its own step
+  tolerates), conformance **SKIPPED**, and **approval-policy FAIL → MERGE GATE: BLOCKED**.
+  The failure is *only* the sensitive-path rule: "This PR changes the pipeline or the
+  security posture it rests on, and has NO approving review from anyone other than its author
+  ('gitea-admin')", over eleven paths — `clusters/{base,dev,prod}/*`,
+  `platform-config/kustomization.yaml`, `platform/kyverno/{admission,tests}/pm-agent*`,
+  `platform/pm-agent/rbac.yaml` and `platform/pm-agent/pm-agent-credentials.sops.yaml`.
+  **That is the gate working, not a defect in the change**, and it is the one thing an agent
+  must not clear for itself: the operator posts `/approve-override <reason>` as a PR *review*,
+  or a non-author reviews it. CI is therefore red and will stay red until a human acts.
+- **The commit-status API did not reflect any of that.** While the run completed and its log
+  carried the verdict above, `GET /commits/{sha}/status` still reported
+  `gate / gate (pull_request): pending — Waiting to run`. Worth knowing before anyone builds
+  automation on that endpoint: on this Gitea the status tick lagged the run indefinitely, so
+  the log is the source of truth. A fresh instance of §2b — "a green tick is not proof
+  anything ran"; here the tick was not even wrong, it was absent.
+- **The gate caught two real defects in this change before I did**, which is worth recording
+  because both were in the parts I was most confident about:
+  1. `flux-postbuild-substitution` → 1 UNDEFINED + 1 FATAL-PARSE. My ConfigMap's *comment*
+     explaining the envsubst hazard contained two dollar-brace tokens, and envsubst runs over
+     comment lines too — so the paragraph warning about the trap was itself the trap. Rewritten
+     without braces.
+  2. `no-plaintext-secrets` → the unencrypted `.example` template was read as REFERENCED by a
+     kustomization, because the kustomization's comment named it by filename. Correctly
+     refused: the distance between a filename in a comment and a line under `resources:` is
+     one careless edit. The comment now points at it in prose.
+- **`infra/agent/probe-validation.sh` → 17 refusals, 0 failures.** A new, committed
+  negative-path probe for `pm-worker`'s input guards — the things that stand between an
+  injected environment variable and a clone from the wrong host or a push to the wrong ref.
+  Each case asserts a non-zero exit AND the specific message, so a guard that stopped firing
+  cannot pass by failing somewhere else. It refuses: an argument on the pinned command;
+  `main`, a wrong prefix, `feat/x;id`, `../../etc/passwd`, too-short slugs, and a branch equal
+  to the base; a redirected git host, a real-and-reachable wrong one (`github.com` — the
+  egress policy permits tcp/443, so this one is genuinely reachable), and the right host with
+  a smuggled path; a malformed owner or repository; an unknown mode; a non-numeric and a zero
+  issue number; and each missing credential. `build-image.sh` also runs it inside the built
+  image, where the interpreter is the one that will run in the cluster.
+- **Writing that probe corrected an overclaim in the code.** The `PM_BRANCH != "main"` guard
+  is UNREACHABLE while the pattern check above it holds (`main` has no `feat|fix|docs|chore/`
+  prefix), so the comment claiming it as the defence against pushing to main was wrong about
+  which line does the work. It is kept — as an assertion that does not depend on the regex
+  being right — and the comment now says so.
+- **Two more defects found by self-review of the unattended paths, both fixed:** the worker's
+  `EXIT` trap ended in a failing test, which under `set -e` can replace the script's real exit
+  code with that failure; and a provenance *log line* read a file through a redirect that
+  `set -o pipefail` would have turned into a dead worker if the file were absent.
+- **A self-review of the trigger found and fixed one real defect** before any of this ran:
+  `last_activity()` compared RFC 3339 timestamp STRINGS to find the newest comment, which is
+  only correct while every offset Gitea returns is identical — and the failure mode is the
+  worst available one, the PM answering its own last comment in a loop. It now takes the last
+  element of the comments list, which is the order the endpoint returns, so there is nothing
+  about timezones to be right about.
+- **`check-jsonschema --schemafile schemas/servicedefinition.schema.json catalog/gitea.yaml`
+  → ok.**
+- **The golden reproduces:** `python3 catalog/render.py --all catalog -o apps --wave-layout
+  --no-image-automation` leaves `apps/wave-0/gitea.yaml` byte-identical to what is committed,
+  and the only diff versus `main` is the new `gitea-allow-pm-agent-ingress`
+  CiliumNetworkPolicy.
+- **Syntax:** `bash -n infra/agent/pm-worker` and `ast.parse` on `pm-dispatch.py` both clean.
+  `yaml.safe_load_all` parses every new manifest.
+- **The spec repo has no git remote** — `git remote -v` in `../home-network-k3s` prints
+  nothing. That is *why* a worker's gate can never run `spec-immutability` in-cluster, and it
+  is measured rather than assumed. The worker's acceptance rule names that stage and
+  `gitea-settings` explicitly and refuses any other skip.
+- **`PM_SELFTEST=1 python3 infra/agent/pm-dispatch.py` → exit 0, 26 assertions, all ok**
+  (run in the toolbox against the repository's own files, with `PM_WORKER_JOB_TEMPLATE` and
+  `PM_PERSONA_FILE` pointed at `infra/agent/`). It covers the branch vocabulary in both
+  directions (`feature/x`, `main`, `feat/AB`, `feat/x;rm -rf /` and `../../etc/passwd` all
+  refused), the five result-envelope shapes `extract_decision` has to survive plus three it
+  must reject, the committed Job template against its whole contract (pinned command, no
+  args, the powerless SA, no mounted token, the volume set, the security context, and every
+  env name inside the dispatcher's allowlist), and that the persona still says the four
+  things it must. `build-image.sh` runs the same selftest inside the image; that invocation
+  is the one that has not happened, because the image has not been built.
+- **Claude Code facts were read from the current documentation, not from memory:**
+  `claude setup-token` mints a one-year subscription OAuth token consumed via
+  `CLAUDE_CODE_OAUTH_TOKEN`; `ANTHROPIC_API_KEY`/`ANTHROPIC_AUTH_TOKEN` rank **above** it in
+  credential precedence (hence the unsets and the env-name allowlist); `--bare` does **not**
+  read it (hence neither entrypoint uses that flag); `--permission-prompts none` requires
+  >= 2.1.259, which is why the image asserts a version floor. The pin `2.1.285` is the npm
+  registry's `stable` dist-tag, measured on 2026-10-05.
+
+## Merged `main` in, after PR #55 and #56 landed
+
+One conflict, in `clusters/base/kustomization.yaml`: PR #55 (vpn-ingress) and this branch
+both appended a resource line to the same tail. Resolved by keeping both, main's first —
+`vpn-ingress` is a platform-config-tier leaf and `pm-agent` dependsOn `apps-wave-0`, which
+is later. Nothing else conflicted; `runbooks/kustomization.yaml` auto-merged with both
+entries, and the handoff entries are separate files with nothing to merge, which is exactly
+what `decisions/` being a directory buys.
+
+**PR #56 added `CLAUDE.md` §0 — work in your own worktree, never the shared checkout —
+while this branch was open, and it named this run's uncommitted directories as its
+measured example.** Two consequences were handled rather than noted:
+
+- This merge, and everything after it, was done in a worktree outside the repository root
+  (`/var/home/keith/source/wt-pm-agent`), per §0.
+- **§0 would have misled a worker agent, so `infra/agent/worker-rules.md` now says so
+  explicitly.** A worker reads `CLAUDE.md` first, and §0's instruction is to create a
+  worktree outside the shared checkout. There is no shared checkout in a worker's container
+  — it holds a fresh clone of its own, which is what §0 is actually asking for — and a
+  worker that obeyed §0 literally would `git worktree add`, edit somewhere the wrapper does
+  not look, and have its work committed by nobody while the issue got a "changed nothing"
+  report. The rules file now tells it §0 is already satisfied and not to make a worktree.
+
+**The operator's approving review goes STALE on each new head, and the gate does not care.**
+I first wrote here that it would need re-approving. That was wrong, and the correction is
+worth keeping because it is a fact about this repository's gate rather than about this
+change:
+
+- Measured on the live branch protection: `dismiss_stale_approvals = true`,
+  `ignore_stale_approvals = false` — so pushing the merge did stale `jakman85`'s review
+  (confirmed: `GET /pulls/57/reviews` → `APPROVED`, `stale: true`).
+- But `ci/approval-policy.sh` selects reviews on `.state == "APPROVED"` and
+  `.user.login != author` and reads **nothing about staleness** (line 147). So a stale
+  approval still satisfies the sensitive-path rule.
+- And `required_approvals = 0`, so Gitea's own merge button never required one either.
+
+Measured, not inferred: CI run **#257** on the merged head printed
+`approved by 'jakman85' (author 'gitea-admin') — sensitive change is human-reviewed.` →
+`stage approval-policy: pass`, with `gitea-settings: pass` as well. The job's own exit was
+`MERGE GATE: UNDETERMINED` (rc 3 — `spec-immutability` and `conformance` skipped, as the
+workflow tolerates with `|| [ $? -eq 3 ]`), so the required check
+`gate / gate (pull_request)` is **green** and PR #57 reports `mergeable: true`.
+
+Nothing is therefore waiting on the operator except **the merge itself**, which is theirs
+by CLAUDE.md §1 and never an agent's. Worth noting for whoever reads this later: the rule
+"push again and you must override again" applies to the `/approve-override` **directive**,
+which is staleness-checked against the head commit — not to an ordinary approving review,
+which this gate accepts stale.
+
+## What I did NOT verify
+
+**Nothing in this change has run.** That is the headline, and everything below is a
+consequence of it.
+
+- **The image has never been built.** The nodejs module-stream install, the `npm i -g
+  @anthropic-ai/claude-code@2.1.285` layer, the version-floor assertion, the `COPY`
+  ownership assertions and `build-image.sh`'s whole in-image verification (including
+  `PM_SELFTEST=1 pm-dispatch`) are **unexercised**. Expect the build to need iteration.
+- **It has never been pushed,** so the committed image digests are the deliberate all-zero
+  placeholder. Nothing has pulled this image.
+- **No pod has started.** The dispatcher has never authenticated to Gitea, never polled,
+  never triaged an issue, never created a Job. No worker has cloned, branched, run a session,
+  run the gate in-cluster, pushed a branch, opened a pull request or commented on an issue.
+  Every claim about runtime behaviour is a design claim.
+- **The admission policy has not met a live API server.** 28 offline assertions prove the
+  policy *refuses*; nothing proves the API server *enforces* it here. That is an L2 test and
+  it needs a cluster. Note also that `CLAUDE.md` §2b applies to me as much as to anyone: I
+  did not query whether Kyverno is healthy, whether `pm-agent-job-shape` reconciles, or
+  whether `platform-config` is Ready.
+- **The network policies have not been exercised.** Whether `toEntities: kube-apiserver`
+  actually permits `kubectl` from this namespace on k3s, and whether `world:443` reaches
+  `api.anthropic.com` through this cluster's egress, are both unmeasured.
+- **Neither credential exists.** The Gitea bot account has not been created, so its
+  permission has not been measured — and *that measurement is the control* that keeps the
+  agent from clearing its own sensitive-path review. Do it before the first real run.
+- **Gitea's webhook `ALLOWED_HOST_LIST` was not queried.** It is cited as a reason webhooks
+  were rejected; it is not a measured fact about this Gitea.
+- **The `write`-can-merge gap is real and unclosed.** No Gitea token scope separates "push a
+  branch and open a pull request" from "merge it". The service issues no merge call, which is
+  a property of the code, not of the credential.
+- **The worker's credentials are inside the model session's reach.** Same container, same
+  uid; no isolation exists and none is claimed. Bounded by token scope and egress only.
+- **This branch touches sensitive paths** — `clusters/**`, `platform/kyverno/**`,
+  `platform-config/kustomization.yaml`, `**/rbac.yaml`, `**/*.sops.yaml` — so it needs an
+  approving review from a non-author, or an administrator's `/approve-override` **posted by
+  the operator**. An agent must not clear this one on its own PR; that is precisely the
+  self-approval the stage exists to prevent.
+- **No observability.** No metrics endpoint, no ServiceMonitor, no alert, and the namespace's
+  default-deny would drop a scrape from `monitoring` if one were added. Follow-ups are listed
+  in the plan, in the order they are worth doing.

decisions/2026-10-05-pm-agent-registry-pull-prereq.md

@@ -0,0 +1,127 @@
+---
+date: 2026-10-05
+branch: docs/pm-agent-registry-pull-prereq
+title: pm-agent reconciled exactly as designed — and is the first workload ever to pull from the in-cluster registry, which does not work yet
+---
+
+## What changed
+
+Documentation only. No manifest, no policy, no code.
+
+PR #57 merged and `pm-agent` reconciled. I measured what landed, and the measurement turned
+up **two prerequisites for the next operator step that nothing in this repository
+declares** — and that the handoff entry for #57 could not have known, because it listed the
+image pull as unverified and that is exactly where it bit.
+
+So this entry records the post-merge measurement, and
+`platform/pm-agent/README.md`, `docs/plans/pm-agent-issue-to-pr.md` and
+`runbooks/pm-agent.md` now name both prerequisites where they say "build, push, pin the
+digest". Before this, that step read as one action; it is three, and the two new ones are
+not an agent's to perform.
+
+**The finding is not specific to pm-agent.** `conformance-runner` has the identical
+registry path (`gitea.home.keithjackson.dev/home-network/...`), the identical "operator
+pushes, a digest goes in a PR" model, and has never been pulled either — so the same two
+blockers sit in front of the L2/L3 conformance wiring. Measured: **no pod anywhere in the
+cluster references that registry** except the pm-agent dispatcher, which is therefore the
+first workload ever to try it.
+
+## What I verified
+
+Read-only, through the pinned toolbox, against the live prod cluster.
+
+**The merge reconciled, completely and exactly as designed.**
+
+- `GitRepository/flux-system` artifact revision: `main@sha1:95597bd6883637…` — the merge
+  commit. `platform-config`, `pm-agent` and `apps-wave-0` all report that same
+  `lastAppliedRevision` with `Ready=True`. Compared field to field, per CLAUDE.md §2b,
+  rather than assumed from the merge.
+- **The security boundary is live.** `ClusterPolicy/pm-agent-job-shape`:
+  `validationFailureAction=Enforce`, `failurePolicy=Fail`, `Ready=True`, and all six rules
+  present (`worker-must-be-the-committed-shape`,
+  `worker-must-be-one-container-with-no-args`,
+  `worker-volumes-must-be-workspace-or-tmp-emptydirs`,
+  `worker-may-not-declare-init-or-ephemeral-containers`,
+  `worker-env-names-must-be-the-committed-set`, `worker-may-not-use-envfrom`). This is the
+  check that mattered most: a ClusterPolicy added to the wrong kustomization list ships
+  green and never reaches the cluster, which is what happened to the conformance boundary
+  on 2026-09-29.
+- Namespace contents: both ServiceAccounts, `pm-agent-config` (17 keys), the
+  `pm-agent-credentials` Secret, the `default-deny` NetworkPolicy, and both
+  CiliumNetworkPolicies with `VALID=True`.
+- **SOPS decryption works through the new overlay patch.** Checked by KEY NAMES and VALUE
+  LENGTHS only, never values (CLAUDE.md §6): keys `gitea-token`, `claude-code-oauth-token`,
+  lengths 76 and 52 bytes — which are the base64 lengths of the committed placeholder
+  strings, so the Secret holds the decrypted placeholders and not ciphertext. A
+  ciphertext apply succeeds silently and would have left `ENC[AES256_GCM,…]` values an
+  order of magnitude longer.
+- **`pm-agent-worker` really is bound to nothing.** The namespace has exactly one
+  RoleBinding (`pm-agent` → Role `pm-agent`, subject `ServiceAccount/pm-agent`), and no
+  ClusterRoleBinding anywhere names either pm-agent ServiceAccount.
+- `gitea-allow-pm-agent-ingress` is applied in the `gitea` namespace, matching
+  `io.kubernetes.pod.namespace: pm-agent`.
+- The dispatcher is `ImagePullBackOff` on the all-zero placeholder digest — the designed,
+  visible failure.
+
+**The two prerequisites, measured.**
+
+1. **The nodes cannot verify the registry's TLS certificate.** containerd's error on the
+   pull is not about the digest:
+
+       failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:000…":
+       tls: failed to verify certificate: x509: certificate is not valid for any names,
+       but wanted to match gitea.home.keithjackson.dev
+
+   "not valid for **any** names" means the certificate presented to the node carries no
+   matching SAN at all. From the operator's workstation the same name is healthy: it
+   resolves to `192.168.2.20` and serves a Let's Encrypt certificate
+   `CN=*.home.keithjackson.dev` with SANs `*.home.keithjackson.dev, home.keithjackson.dev`
+   — and `Certificate gateway/wildcard-home` is `Ready=True` for exactly those names. So
+   the certificate that *should* be served is fine, and what the NODE reaches is not what
+   the workstation reaches. **Correcting the digest will not clear this.**
+2. **Anonymous pulls from this registry are refused.** Not inferred from a bare 401 — that
+   would have been wrong, because this Gitea answers 401 for *everything* anonymously:
+   `/v2/home-network/agent-runner/manifests/latest`, `/v2/home-network/conformance-runner/…`,
+   a randomly-named nonexistent repository, and `nosuchowner/nosuchimage` all return 401,
+   and `/api/v1/packages/Home-Network` returns `{"message":"token is required"}`. The
+   registry *does* issue a token to an anonymous client, but its `access` claim is `null` —
+   no pull scope. So the conclusion that holds is the simple one: **an unauthenticated
+   client gets nothing from this registry**, so a pull with no credential cannot succeed.
+   And `grep` over the tree finds **no `imagePullSecrets` anywhere** — not in
+   `platform/pm-agent/deployment.yaml`, not in `infra/agent/worker-job-template.yaml`, not
+   in `conformance/incluster/`.
+
+## What I did NOT verify
+
+- **Why the node sees a nameless certificate.** I measured that it does, and that the LAN
+  does not. I did not diagnose it: I did not read the nodes' resolver or
+  `/etc/rancher/k3s/registries.yaml` (node files, not reachable read-only through the API),
+  and I did not check whether the node resolves the name to the same `192.168.2.20` the
+  workstation does or to something else.
+
+  **A correction to a remark I first wrote here:** I cited
+  `cluster-vars`'s `GATEWAY_VIP: 192.168.1.53` as suspicious next to the nodes'
+  `192.168.2.x`. That was noise — the operator confirms those base `cluster-vars` values are
+  **dev-era leftovers for a cluster that is decommissioned**, so the 192.168.1.x addresses in
+  that file say nothing about prod and should not be read as evidence about anything. The
+  measured facts stand on their own and none of them depended on that knob.
+- **Whether the package would be pullable once authenticated.** Nothing has ever been
+  pushed to that path, so "a credential is sufficient" is untested. The only thing
+  established is that no credential is insufficient.
+- **Which fix the operator wants.** Two routes, and they are not equivalent:
+  (a) an `imagePullSecret` — a `dockerconfigjson` holding a Gitea token, SOPS-encrypted
+  like `pm-agent-credentials`, referenced from the Deployment *and* from the committed
+  worker Job template; note the Job template's shape is pinned by admission, and
+  `imagePullSecrets` is a field that policy does not currently constrain, so adding one is
+  a template change that needs a deliberate look at whether the policy should pin it too.
+  (b) allowing anonymous reads of that package in Gitea — no manifest change at all, but it
+  makes the image world-readable to anything that can reach the host.
+  I have implemented neither: both need an operator action I must not take, and (b) is a
+  posture decision. The docs now present both.
+- **Everything the #57 entry listed as unverified about RUNTIME remains unverified.** The
+  image has still never been built or pushed. No dispatcher has authenticated to Gitea, no
+  issue has been triaged, no worker Job has been created — so the admission policy, though
+  now provably applied and Enforce, has still never refused or admitted a real Job. The
+  network policies have not carried traffic.
+- **I did not re-run the merge gate against `main` after the merge.** This branch's own gate
+  run is the evidence for this change; the merged state's is CI run on PR #57.

decisions/2026-10-05-vpn-remote-access-wireguard.md

@@ -0,0 +1,225 @@
+---
+date: 2026-10-05
+branch: feat/vpn-remote-access-wireguard
+title: Remote-access VPN — WireGuard on udp/51820, hostNetwork because a pod identity would 403 every Gateway hostname
+---
+
+## What changed
+
+New platform component `platform/vpn-ingress/`: a **WireGuard** remote-access VPN so the operator's
+Android / Linux / Windows / macOS devices can dial in from anywhere on the internet and behave like
+machines on the LAN — full reach to `192.168.2.0/24` plus internet egress out the house WAN.
+
+This is the **opposite direction** to the existing `platform/vpn-egress` (pods *out* through PIA,
+ADR-0015). They share nothing but the word "VPN"; both READMEs now say so, and `README.md`'s layout
+block labels the two.
+
+**Transport: WireGuard, and the security argument is the handshake's failure mode, not "it is
+encrypted".** WireGuard's first handshake message carries a MAC keyed with the responder's own
+public key, so a sender that does not already know it gets **no reply at all** — no error, no reset,
+no banner. From the internet, udp/51820 is indistinguishable from a closed port: nothing to
+brute-force, no version string, no amplification. That is what makes an unfiltered WAN port
+defensible here, and it is why there is deliberately no `loadBalancerSourceRanges`-style allowlist
+(which would also break roaming clients). It also has first-party clients on all four named
+platforms, which OpenVPN (needs a CA + per-device certs + a CRL) and IKEv2 do not.
+
+**`hostNetwork: true`, and this is THE decision — forced by a measured property of this cluster.**
+A pod-networked WireGuard server must SNAT client traffic to its own pod IP (Cilium drops a packet
+whose source is not the endpoint's address), so every remote client would have arrived wearing *that
+pod's* security identity. `decisions/2026-10-04-plex-advertise-plexdirect.md` measured what that
+costs: a request to the shared Gateway VIP **from inside a pod returns `403 Access denied` from
+Envoy for all 16 Gateway hostnames**, because Cilium's Gateway preserves the original source
+identity and enforces the *destination's* ingress policy, and every app admits `reserved:ingress`
+plus named peers — never an arbitrary pod. So the obvious build would have produced a tunnel that
+came up healthy, showed connected peers, routed packets, and 403'd every cluster web UI — and
+`*.home.keithjackson.dev` would not even have resolved, because `apps/wave-0/pihole-dns.yaml`'s
+ingress policy deliberately omits the `cluster` entity. In the host netns there is no pod identity
+to carry: forwarded traffic is SNATed to the node address and arrives as `host`/`remote-node`,
+byte-for-byte what a LAN client looks like. Three further problems vanish with it: `ip_forward` is
+already 1 on these nodes (so no `--allowed-unsafe-sysctls` kubelet change — it is *not* a safe
+sysctl); no LoadBalancer VIP, so no L2-announcement lease lottery and no `externalTrafficPolicy`
+trade-off; and no double encapsulation, which is what makes `MTU = 1420` simply correct.
+
+`hostNetwork` is permitted by `restrict-host-namespaces`' own stated carve-out (it matches
+`app-namespace=true` only, and names Cilium/node-exporter as the precedent); `namespace.yaml` is
+deliberately unlabelled. **The cost is stated, not buried:** a hostNetwork pod has no Cilium
+endpoint, so **no NetworkPolicy constrains this workload**. `networkpolicy.yaml` says that at
+length rather than shipping an `allow world → 51820` rule that would read as the control and
+enforce nothing; the real boundary is WireGuard's peer authentication.
+
+**A DaemonSet, not a Deployment.** The WAN forward names one node address by hand, off-repo (same
+shape as the plex tcp/32400 forward). A floating single replica would silently invalidate it on
+every reschedule; an identical listener on all three nodes makes **failover a one-line router
+change** with no cluster work and no re-issued client profiles. A pinned LoadBalancer VIP was the
+considered alternative and was rejected: it reintroduces the L2 lease, SNAT and an extra node hop
+into the one path whose failure mode is "I am away from home and cannot get in".
+
+**Authorisation is split so that it stays reviewable.** Peer **public** keys live in
+`platform/vpn-ingress/peers.yaml` in **plaintext** — a public key grants nothing by itself, and this
+makes "who can get into my house" a PR diff with `git log` as its audit trail. The only real secret,
+the server's private key, is the `wg-server-key` Secret, **created out of band by the operator**: an
+agent must not mint a long-lived private key (CLAUDE.md §6), and a key that passed through a session
+transcript would have to be treated as compromised. The two halves are joined only at pod start, by
+an init container, into a `medium: Memory` emptyDir — so the assembled config holding the key never
+touches a node's disk and dies with the pod.
+
+**There are no peers yet, and that is the correct initial state**, for the same reason. The VPN ships
+INERT-BUT-HEALTHY: a zero-peer interface listens and refuses every handshake.
+
+Files:
+
+- `platform/vpn-ingress/` (new): `namespace.yaml`, `peers.yaml` (the plaintext peer register),
+  `daemonset.yaml` (hostNetwork listener + config assembly), `networkpolicy.yaml` (namespace
+  baseline, with the honest caveat), `kustomization.yaml`, `README.md` (the design record).
+- `clusters/base/vpn-ingress.yaml` (new): its own **non-gating post-platform** Flux Kustomization
+  (`dependsOn platform-config`, `wait:false`). Non-gating is load-bearing rather than conventional
+  here: the DaemonSet *cannot* start until the operator creates the server key, so in the gating
+  `platform` tier an unfinished manual step would have stalled every apps wave on merge.
+- `clusters/base/kustomization.yaml`: the new tier added.
+- `clusters/base/cluster-vars.yaml`: new `WG_CLIENT_CIDR` (`10.90.0.0/24`), declared next to
+  `LAN_CIDR` because it is the one range that must be collision-checked against every other; the
+  comment enumerates the five it avoids. **No prod override, deliberately** — unlike every VIP, this
+  range is private to the tunnel, so the same value is correct in both environments and a prod patch
+  would imply otherwise. Substitution inventory updated.
+- `infra/ansible/roles/node_baseline/defaults/main.yml`: `wireguard` added to
+  `node_baseline_kernel_modules`. The pod has `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so
+  it cannot autoload the module — it CrashLoops with a clear error rather than quietly degrading to
+  userspace wireguard-go (which would need `/dev/net/tun`, a device a hostPath mount cannot actually
+  grant, as the `/dev/dri` attempt proved).
+- `infra/vpn-ingress/new-peer.sh` (new): enrols a device. Generates the keypair on the operator's
+  machine, writes a 0600 profile **outside** the checkout (it refuses to write into the repo),
+  prints a QR code for phones, and prints only the public `[Peer]` stanza. It also refuses a
+  duplicate address, because two devices on one address does not error anywhere — the server routes
+  to whichever handshook last, so both appear to work intermittently.
+- `runbooks/vpn-remote-access.md` (new) + its `kustomization.yaml` generator entry + `README.md`
+  index line: the three manual steps, the four off-LAN verification checks, revocation, and why no
+  connection alert exists.
+- `README.md`: `vpn-ingress` added to the layout, with the egress/ingress distinction spelled out.
+
+**THREE MANUAL STEPS REMAIN AND THE VPN DOES NOT WORK UNTIL ALL THREE ARE DONE** (all off-repo by
+necessity; `runbooks/vpn-remote-access.md`): run `node_baseline` to load the kernel module, create
+the `wg-server-key` Secret, and forward WAN udp/51820 to one node address on the SOHO router.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (three files), so `ci/approval-policy.sh` will
+BLOCK it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
+"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
+PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
+
+## What I verified
+
+Live prod, read-only, 2026-10-05, plus the offline gate.
+
+- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
+  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
+  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
+  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
+  Run from an isolated git worktree with both CLAUDE.md §3 workarounds applied
+  (`DEV_KUBECONFIG=/dev/null` plus an `ENGINE` shim bind-mounting the main checkout's `.git` and
+  the sibling spec repo) — which is why `spec-immutability` is **green here and skipped in CI**,
+  where the runner has no spec checkout. `approval-policy` passing locally is an offline N/A, not a
+  verdict; the real verdict is the CI run below.
+- **The CI gate ran on PR #55 (run 244, head `1a67fc1b`) and `approval-policy` is the ONLY failing
+  stage** — read from the job log, not from the red tick. Verbatim: "This PR changes the pipeline or
+  the security posture it rests on, and has NO approving review from anyone other than its author
+  ('gitea-admin')", naming exactly the three files I predicted:
+  `clusters/base/cluster-vars.yaml`, `clusters/base/kustomization.yaml`,
+  `clusters/base/vpn-ingress.yaml`. Every other stage that ran reported green —
+  `gitea-settings`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`. So the branch is blocked
+  on the operator's `/approve-override <reason>` (posted as a PR **review** of type "Comment") and
+  on nothing else. **I have not posted it** — CLAUDE.md §1.
+- **Branch protection is live, measured by CI's own `gitea-settings` stage** (it has the token this
+  workstation lacks): `required_approvals = 0`, `enable_status_check = true`,
+  `status_check_contexts = ["gate / gate (pull_request)"]`, `block_admin_merge_override = true`,
+  `enable_push = false`, `dismiss_stale_approvals = true`, and **1 runner online**. That is the §2b
+  re-measurement of CLAUDE.md §2's table for this branch, not a restatement of it.
+- **`flux-postbuild-substitution` (L0) actually exercised the new wiring, and caught a real defect
+  in it.** On the first run it FAILED, naming `platform/vpn-ingress/daemonset.yaml:118` and
+  attributing it to "Flux Kustomization 'vpn-ingress' (clusters/base/vpn-ingress.yaml, path
+  ./platform/vpn-ingress)". Two things are proven by that message: the new tier's
+  `postBuild.substituteFrom` declaration is correctly detected (so `${WG_CLIENT_CIDR}` will be
+  substituted, not shipped literally), and the check reads comments as well as values — the hazard
+  was a comment that spelled out the forbidden braced form literally while claiming not to use it.
+  Fixed; the check now passes 444/444 with 0 hazards.
+- **The `wireguard` kernel module is available on these nodes by construction**: all three run
+  Debian 13 (trixie) on kernel `7.1.8+deb13-amd64` (measured), and WireGuard has been in-tree since
+  5.6, so it is present in the distro kernel and only needs loading.
+- **`net.ipv4.ip_forward = 1` is already written by `node_baseline`**
+  (`roles/node_baseline/tasks/main.yml`, `/etc/sysctl.d/99-k3s-cilium.conf`), and **no kubelet
+  carries `--allowed-unsafe-sysctls`** (grepped across `roles/k3s_server`, `roles/node_baseline` and
+  `group_vars`: zero hits). That pair is what makes the host netns the only option that does not
+  require a node-restart change.
+- **Cilium's datapath, measured from `cilium-config` rather than from the values file**:
+  `enable-ipv4-masquerade=true`, `enable-bpf-masquerade=true`, `routing-mode=tunnel`,
+  `tunnel-protocol=vxlan`, `ipv4-native-routing-cidr` EMPTY, Cilium `v1.20.1`. The vxlan underlay is
+  where `MTU = 1420` comes from, and the empty native-routing CIDR is why pod→LAN is masqueraded
+  today.
+- **`WG_CLIENT_CIDR` collides with nothing.** Pod CIDR `10.0.0.0/16` (`CLUSTER_POD_CIDR`), service
+  CIDR `10.43.0.0/16` (measured: `kubernetes` Service clusterIP is `10.43.0.1`), the PIA tunnel's
+  `10.72.192.0/18` (`platform/vpn-egress/README.md`), and both LANs `192.168.1.0/24` /
+  `192.168.2.0/24`. `10.90.0.0/24` is outside all five.
+- **The egress-class label matters and is set to `internet`.** Read live:
+  `CiliumEgressGatewayPolicy vpn-egress-steering` selects `podSelector:
+  homelab.keithjackson.dev/egress-class: vpn` with **no namespace scope**, so `vpn` here would have
+  steered every remote client's internet traffic out PIA.
+- **All three nodes are untainted** (`spec.taints: None` on each), so the DaemonSet covers the
+  cluster with no tolerations.
+- **`new-peer.sh` was run end to end** against a throwaway server key in the scratchpad (deleted
+  immediately after; it never existed in the repo). It rendered a correct profile — `Address
+  10.90.0.7/32`, `DNS 192.168.2.23`, `Endpoint wan.home.keithjackson.dev:51820`, `MTU 1420`,
+  `PersistentKeepalive 25`, `AllowedIPs 0.0.0.0/0` — at mode 0600, printed the matching public-key
+  stanza, and both argument-validation paths (missing name, out-of-range octet) refuse correctly.
+  The private key was redacted from every line I displayed.
+- **The endpoint name is resolver-independent, which is the specific trap this avoids.**
+  `wan.${BASE_DOMAIN}` is an unproxied public Cloudflare A record kept current by
+  `platform/externaldns/ddns.yaml` (checked every 5 min), so it resolves from a cellular connection
+  — unlike any name PiHole answers, which a remote client cannot resolve until it is already
+  connected. (This is the `plex.direct` lesson applied.)
+- **Velero does cover this namespace.** `platform/velero/helmrelease.yaml`'s `daily` schedule
+  excludes only `monitoring` and `unifi`, so `vpn-ingress` and the `wg-server-key` Secret are
+  captured off-cluster to R2. Read from the manifest, not from a backup listing.
+
+## What I did NOT verify
+
+- **THE ACTUAL FIX. Nothing in this branch has been proven to carry a single packet.** The
+  DaemonSet has never been reconciled, no `wg0` has ever existed, and no handshake has ever been
+  attempted. Three blockers are off-repo and manual (kernel module, server key, WAN forward), and
+  this branch removes none of them — it only makes them the *only* remaining work.
+- **Outside-in reachability, at all.** This repo has **no off-LAN vantage point**, which is exactly
+  the gap the plex remote-access work hit. Step 6 of the runbook (a phone on cellular) is the only
+  honest test and it has not been run.
+- **The in-pod `iptables` → host-netfilter interaction, which is my biggest single doubt.** Debian 13
+  uses nf_tables; the container writes MASQUERADE/FORWARD rules into the **host's** tables via the
+  image's own `iptables`. If that binary is the legacy backend while the host is nft, the rules can
+  apply to a table the kernel does not consult — and the symptom would be the nastiest available:
+  handshakes succeed, `wg show` looks perfect, and every forwarded connection hangs. **Check this
+  first if the tunnel comes up but nothing is reachable:** `iptables-save -t nat | grep MASQUERADE`
+  and `nft list table ip nat` on the node, from the host, not from the pod.
+- **Whether `allowPrivilegeEscalation: false` is compatible with the image's s6 init.** It is the
+  correct posture and I set it, but this repo has already been bitten by `no-new-privileges`
+  misbehaving (`platform/externaldns/ddns.yaml`'s README note). If the pod fails before
+  `wg-quick` runs, that flag is the first thing to try flipping.
+- **Whether `NET_ADMIN` alone is sufficient.** Reasoned from what `wg-quick` does (create the link,
+  set a sysctl, write two netfilter rules), not measured. If `iptables` complains, `NET_RAW` is the
+  likely addition — deliberately not granted pre-emptively.
+- **The image's custom-config path.** `lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124` (tag
+  confirmed to exist and to be the current one, published 2026-10-01) is expected to bring up
+  whatever is in `/config/wg_confs/` when `PEERS` is unset. That is its documented behaviour; it has
+  not been run here, and the init container's assembly of that file has never executed.
+- **Whether the probe is right.** `wg show wg0` is asserted to exit non-zero when the interface is
+  absent. Not executed.
+- **Cilium's reaction to a new `wg0` in the host netns.** `platform/cilium/values.yaml` does not pin
+  `devices`, so Cilium auto-detects; whether it attaches to `wg0` — and whether that matters — is
+  unknown. `wg` interfaces are already filtered out by `node_health_collect.py`, so something in
+  the estate anticipates them, but that is not evidence about Cilium.
+- **Zero peers means the end-to-end authorisation path is untested**, including revocation. The
+  first enrolment is also the first test of `peers.yaml`'s format being what `wg-quick` accepts.
+- **No peer-level observability exists.** An exporter publishing
+  `wireguard_latest_handshake_seconds` is the top follow-up; until then "who is connected" is an
+  on-demand `wg show` and nothing alerts on an unexpected session. Pod-down is covered only by
+  kube-prometheus-stack's generic `KubePodNotReady` / `KubeDeploymentReplicasMismatch`, which I did
+  not confirm fire for a DaemonSet in this namespace.
+- **The security consequence I am most keen for a human to review rather than take from me:** an
+  enrolled device reaches the three k3s nodes themselves — apiserver 6443, kubelet 10250, SSH 22 —
+  because "reach the entire local network" includes them. That is stated in the README and the
+  runbook with the narrowing recipe, but it is a posture decision, not a verified fact, and it is
+  the right thing to argue about on this PR.

decisions/2026-10-05-vpn-runbook-ansible-command.md

@@ -0,0 +1,112 @@
+---
+date: 2026-10-05
+branch: fix/vpn-runbook-ansible-command
+title: vpn-remote-access runbook — step 1 named a script that does not exist, and step 2 cited the wrong reason the key is gitignored
+---
+
+## What changed
+
+Two defects in `runbooks/vpn-remote-access.md` (merged in #55), both authored by me, both reported
+by the operator the first time they tried to follow it.
+
+**1. Step 1 named a command that does not exist.** It said:
+
+    cd infra/ansible && ./site.sh --tags system --limit k3s_prod
+
+There is **no `site.sh` anywhere in this repo** (`find . -name site.sh` → nothing). The invocation
+was wrong in four separate ways at once, which is what makes it worth recording rather than just
+patching: wrong kind of thing (this repo drives Ansible with `ansible-playbook <playbook>`, never a
+wrapper script — `playbooks/site.yml` exists and is the full bring-up chain, not this); wrong
+playbook (the module work lives in `playbooks/node-baseline.yml`); wrong targeting flag (`--limit`
+instead of this repo's `-e k3s_hosts=...` convention, which `infra/ansible/README.md` "Targeting
+prod" documents for every playbook); and no toolbox wrapper, though Ansible exists only inside the
+pinned container and `ansible.cfg` is only read from `infra/ansible`.
+
+Replaced with the real command, plus the three things about it that are load-bearing:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
+
+- `-e k3s_hosts=k3s_prod` — the playbook defaults to `k3s_servers`, which is the **dev** group;
+  dev is decommissioned, so an unqualified run does nothing useful.
+- `--tags system`, never untagged — the two kernel-module tasks carry that tag, and an untagged run
+  would also hit the `disks` tasks, which assert on block-device variables prod deliberately leaves
+  as placeholders (the playbook's own header warns about this). `--tags system` is broader than the
+  modules alone (sysctls, iscsid, qemu-guest-agent, swap masking) but each is idempotent and
+  already in its desired state on a live node.
+
+**2. Step 2 gave the wrong reason the server key is safe from git — and the right reason is
+fragile.** It told the operator to stage the key at `/work/.cache/server.key` "inside the repo's
+gitignored cache". `.cache/` is **not** gitignored: `git check-ignore -v .cache/serverkey.txt`
+matches nothing. The file was ignored only incidentally, by `.gitignore` line 11 `*.key` — so the
+path happened to be safe for a reason I had not identified, and **any operator who renamed it
+(`server.pem`, `wg-server`, `privatekey`) would have had a committable WireGuard private key sitting
+in the checkout.** The gate would not have caught it either; `no-plaintext-secrets` does not
+recognise a bare WireGuard key.
+
+The step now states the real rule, marks the `.key` suffix as the thing doing the work, and gives
+the operator a verification they can run before trusting it (`git check-ignore -v` must print a
+rule, and `git status --short` must not list the file).
+
+Nothing outside `runbooks/vpn-remote-access.md` changed; no manifest, no behaviour.
+
+## What I verified
+
+- **`site.sh` does not exist**: `find . -name 'site.sh' -not -path './.git/*'` → no output.
+- **The real surface**: `infra/ansible/playbooks/` contains `node-baseline.yml` and `site.yml` (and
+  11 others); `node-baseline.yml`'s own header documents both
+  `-e k3s_hosts=k3s_prod` and the tag-scoping hazard. The `-e <playbook>_hosts=` convention is
+  confirmed by `infra/ansible/README.md` §"Targeting prod" and by the existing
+  `runbooks/node-memory-swap.md`, which invokes `ansible-playbook playbooks/node-health.yml -e
+  node_health_hosts=...`.
+- **`k3s_prod` is a real inventory group**: `[k3s_prod:children]` in `inventory/hosts.ini`, with
+  `[k3s_prod_primary]`/`[k3s_prod_secondary]`.
+- **`--tags system` covers the kernel-module tasks**: both "Persist kernel modules…" and "Load
+  kernel modules now" carry `tags: [system]` in `roles/node_baseline/tasks/main.yml`.
+- **The gitignore claim, measured both directions**:
+  `git check-ignore -v .cache/server.key` → `.gitignore:11:*.key`;
+  `git check-ignore -v .cache/serverkey.txt` → no match. So the suffix is the only thing ignoring it.
+- **Every path the component's docs reference resolves**: all six relative markdown links in
+  `platform/vpn-ingress/README.md` (checked from that file's own directory), and both Ansible
+  playbook paths.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; the rc 3 is the two
+  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no `CLUSTER_AVAILABLE`).
+  **Not green. Reported as rc 3, not as pass.**
+- **This branch touches NO sensitive path** (`runbooks/**` and `decisions/**` are not in
+  `infra/gitea/approval-policy.json`), so unlike #55 it needs no `/approve-override`.
+
+## What I did NOT verify
+
+- **The corrected command has not been RUN.** Ansible is an operator action, and this is the
+  decisive gap: I have verified the playbook, the group, the tag and the flag all exist and are
+  the documented convention, but not that the run succeeds end to end or that
+  `lsmod | grep '^wireguard'` then reports the module. It is a better-evidenced command, not a
+  proven one.
+- **Whether `--tags system` is entirely side-effect-free on a live prod node.** Each task it runs is
+  idempotent by construction, but I did not dry-run it (`--check`) to confirm it reports zero
+  changes beyond the module load.
+- **The rest of the runbook's commands past step 2** are unchanged and still unexercised — in
+  particular `kubectl exec ds/wireguard -- wg show`, which cannot be reached while the pods sit in
+  `Init` waiting on the server key.
+- **Whether any OTHER document I wrote in #55 contains an invented command.** I re-checked every
+  file path and markdown link in the component (all resolve) and every script the docs tell an
+  operator to run, but "the referenced file exists" is weaker than "the command is correct" — which
+  is exactly the gap that produced `site.sh`.
+
+## Post-merge state of #55, recorded here since its own entry predates the merge
+
+Measured on live prod after #55 merged as `ad3f56e`:
+
+- Flux applied it by the real path: Kustomization `vpn-ingress` `lastAppliedRevision:
+  main@sha1:ad3f56e`, equal to the GitRepository's artifact revision, `Ready=True`.
+- All **18** Flux Kustomizations `Ready=True` — the new tier disturbed nothing, and `wait: false`
+  behaved as intended (`vpn-ingress` is Ready while its own pods are not, so nothing downstream
+  stalled on it).
+- **`hostNetwork` is confirmed in effect, not merely configured**: the three pods report pod IPs
+  `192.168.2.10/.11/.12` — the node addresses, one per node.
+- **It fails closed exactly as designed**, with a legible cause: all three pods `Init:0/1`,
+  `MountVolume.SetUp failed for volume "wg-server-key" : secret "wg-server-key" not found`, and no
+  Secret in the namespace. The #55 entry's "the DaemonSet has never been reconciled" is superseded
+  by this paragraph; its remaining unverified claims (the tunnel carrying a packet, the
+  iptables/nftables backend question) still stand.

decisions/2026-10-05-vpn-server-key-in-sops.md

@@ -0,0 +1,187 @@
+---
+date: 2026-10-05
+branch: fix/vpn-server-key-in-sops
+title: VPN server key moves into SOPS-encrypted git — the out-of-band Secret broke R-SEC-01 and the DR story
+---
+
+## What changed
+
+**Operator correction (2026-10-05): "secrets are kept in source through sops".** They are right, and
+#55 shipped the WireGuard server private key the wrong way: as a Secret the operator was told to
+`kubectl create` out of band, referenced by name only. This branch moves it into the tree as
+`platform/vpn-ingress/wg-server-key.sops.yaml`, where every other secret in this repo lives.
+
+**Why the original choice was wrong, not merely unconventional.** Two concrete costs, both of which
+#55's own README half-admitted and then shipped anyway:
+
+1. **R-SEC-01 / spec docs/07 says secrets live in source, SOPS-encrypted, reaching the cluster only
+   through Flux.** An out-of-band Secret is outside the GitOps tree by construction.
+2. **It broke DR for the one credential that cannot be re-derived.** A cluster rebuild restores
+   everything in git; it does not restore a hand-made Secret. Losing this key means **re-enrolling
+   every device**, because each client profile pins its public half. #55 papered over this with
+   "keep a copy in your password manager", which is a workaround for a self-inflicted gap.
+
+Note `platform/kyverno/admission/secret-provenance.yaml` did **not** forbid the original shape —
+it scopes to `app-namespace=true` namespaces and `vpn-ingress` is deliberately a platform namespace,
+so admission would have accepted the hand-applied Secret. The rule it violated was the written
+requirement, not an enforced control. Recording that because "the gate let it through" was never
+evidence it was correct.
+
+**How the unfillable-key problem is solved.** An agent still must not mint a long-lived private key
+(CLAUDE.md §6), so the file is committed **already encrypted, with `privatekey: REPLACE-ME`**. That
+makes the wiring complete and the gate green from the first commit — the resource line and both
+overlay decryption patches are present — and leaves the operator exactly one act: replace the value.
+The alternative, a commented-out resource line to uncomment later, is precisely the "file in git with
+no generator entry: reviewed, merged, apparently done, still broken" shape this repo keeps getting
+bitten by (`alert-runbooks-resolvable` exists because of it).
+
+**The init-container guard is now a validation, not an emptiness check.** It was `[ -z "$key" ]`.
+Because the committed placeholder is non-empty, the likely failure is now a value that is present
+and *wrong*, so the guard pipes the value through `wg pubkey` — which accepts only a real 32-byte
+curve25519 private key — and names which of four cases it hit:
+
+| case | why it matters |
+|---|---|
+| `REPLACE-ME` placeholder | the expected fresh-install state |
+| empty | blanked value |
+| **starts with `ENC[`** | **the Secret reached the cluster UNDECRYPTED.** A missing overlay decryption patch applies cleanly and leaves every Flux condition green — this is the silent failure the guard exists for |
+| rejected, none of the above | mangled value, or a *public* key pasted by mistake |
+
+Files:
+
+- `platform/vpn-ingress/wg-server-key.sops.yaml` (new) — SOPS-encrypted to the repo's age recipient,
+  `stringData.privatekey` only, placeholder value.
+- `platform/vpn-ingress/wg-server-key.sops.yaml.example` (new) — plaintext template + operator
+  hand-off, following the `crafty-oauth2-proxy-credentials.sops.yaml.example` convention.
+- `platform/vpn-ingress/kustomization.yaml` — the resource added; the "DELIBERATELY ABSENT: a
+  Secret" block replaced with the overlay-patch warning.
+- `platform/vpn-ingress/daemonset.yaml` — the `wg pubkey` guard; volume comment rewritten.
+- `clusters/dev/kustomization.yaml`, `clusters/prod/kustomization.yaml` — **the `vpn-ingress`
+  `decryption:` patch, in both.** This is the load-bearing half: #55's own kustomization comment
+  predicted that forgetting it applies ciphertext cleanly.
+- `clusters/base/vpn-ingress.yaml` — its "NO decryption stanza is needed" comment was now false.
+- `platform/vpn-ingress/README.md`, `runbooks/vpn-remote-access.md` — step 2 rewritten.
+- `infra/vpn-ingress/new-peer.sh` — stale "step 1" pointer.
+
+**A second finding: `wg` and `sops` were not in the same place — and the first fix for that was
+also wrong.** Measured: `wg` existed only on the host (`/usr/bin/wg`), `sops` only in the dev
+container, `qrencode` on neither. So `sops -d … | wg pubkey` could not run in one shell, and #55's
+step 2 — which assumed it could — would have failed.
+
+My first fix was to **split every step by where it runs**, and to tell the operator to
+`dnf install qrencode` on their workstation. **Operator correction: "sop, wg, and any other tool
+requirement is supposed to be baked into the devcontainer."** That is CLAUDE.md §5 and the
+devcontainer README's "Don't install the toolchain on the host", and the split fix broke the
+invariant rather than honouring it — it also produced a procedure that could never be reproduced in
+CI, and which no one should have to follow.
+
+**The real fix:** `wireguard-tools` and `qrencode` are now distro packages in
+`infra/devcontainer/Containerfile`, alongside ansible/openssh/diffutils. Both are in Fedora 44's
+repos (`wireguard-tools` 1.0.20260223 — the same upstream version as the container image this
+component runs — and `qrencode` 4.1.1), unpinned like every other dnf package there, because the
+pinned thing is the Fedora base and neither is a gate tool whose version changes what green means.
+
+Three things got *simpler* rather than merely relocated, which is the tell that the operator's rule
+was the right one:
+
+1. **Step 2 is now one pipeline in one shell:** `sops set … "\"$(wg genkey)\""` generates the key
+   straight into the already-encrypted document. The plaintext never lands on disk, never reaches a
+   terminal, and there is no staged copy to shred. The previous version needed a copy in
+   `.cache/server.key`, a `git check-ignore` warning about the `*.key` suffix being the only thing
+   protecting it, and a sha256 comparison to prove the right bytes arrived. All of that is gone.
+2. **`new-peer.sh` now derives the server's public key from `wg-server-key.sops.yaml` itself**
+   (`sops -d --extract … | wg pubkey`), so there is no local key copy to keep in step with the
+   encrypted one. It previously required the operator to keep the private key at a magic host path.
+3. **`.cache/` is now gitignored at the repo root** (mirroring `infra/ansible/.gitignore`), so the
+   container has a persistent, non-committable scratch directory. `new-peer.sh` writes client
+   profiles to `.cache/wg-peers/` by default — necessary because the repo bind-mount is the
+   container's ONLY persistent read-write mount, so a profile written to `/tmp` in there is gone
+   before it can be moved to a desktop. Its old blanket "refuse to write inside the checkout" guard
+   is replaced by `git check-ignore`, which tests the actual property instead of a proxy.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two overlays) AND adds a `**/*.sops.yaml`, so
+`ci/approval-policy.sh` will BLOCK it.** It needs the operator's `/approve-override <reason>` as a
+PR **review** of type "Comment". I have deliberately not posted it (CLAUDE.md §1).
+
+## What I verified
+
+- **The encrypted file is real and correctly targeted.** `sops -e -i` in the pinned toolbox
+  produced `stringData.privatekey: ENC[AES256_GCM,...]` with cleartext keys, `encrypted_regex
+  ^(data|stringData)$`, and exactly one age recipient:
+  `age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg` — the recipient in the root
+  `.sops.yaml`, i.e. the key Flux holds as `sops-age`.
+- **Round-trip, without exposing anything**: `sops -d` then a Python read of key names only →
+  `stringData keys: ['privatekey']`, and `value is still the placeholder: True`.
+- **Both overlay patches are present and name the right Kustomization** (`name: vpn-ingress`,
+  `namespace: flux-system`, `decryption.secretRef.name: sops-age`), inserted after each overlay's
+  `velero` stanza, mirroring headlamp/crafty/velero.
+- **`kubectl kustomize platform/vpn-ingress` builds** and now emits five objects — Namespace,
+  ConfigMap, **Secret**, DaemonSet, NetworkPolicy — with `${WG_CLIENT_CIDR}` still an unexpanded
+  token for Flux.
+- **The tool gap, measured on both host and container before the fix**: `wg` host-only, `sops`
+  container-only, `qrencode` on neither.
+- **Both packages exist in the image's repos**, queried from inside the running container:
+  `wireguard-tools` 1.0.20260223 (repo `updates`) and `qrencode` 4.1.1 (repo `fedora`).
+- **The image REBUILDS with them and both tools resolve inside it** —
+  `./infra/devcontainer/dev.sh build` succeeded, and `wg --version` / `qrencode --version` /
+  `sops --version` all answer in the rebuilt container. This is the claim the change rests on, so
+  it was executed rather than reasoned.
+- **`new-peer.sh`'s new SOPS path fails correctly against the placeholder.** Run in the rebuilt
+  container it refused with "could not derive the server public key … it still holds the REPLACE-ME
+  placeholder", which is both the expected pre-key behaviour and a live test of the
+  `sops -d | wg pubkey` pipeline the whole design now depends on.
+- **`.cache/` is ignored regardless of filename** now: `git check-ignore -v .cache/anything.conf`
+  and `.cache/serverkey.txt` both match `.gitignore:46:.cache/`. Before this change only `*.key`
+  matched, which is the fragility the previous runbook had to warn about.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
+  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no cluster).
+  **Not green. Reported as rc 3, not as pass.**
+- **The L0 gate caught THREE real defects in this branch before it could merge — including one
+  in the fix for another — and I had asserted one of them was fine without running it.** Both are recorded because the gate's value here is the
+  whole point of the exercise:
+  1. **`no-plaintext-secrets` FAILED**: "`wg-server-key.sops.yaml.example`: an unencrypted .example
+     template is referenced by a kustomization — it would be applied as a plaintext Secret". My
+     kustomization named the template in a *comment*, and the check matches comments too (correctly:
+     naming one is how a plaintext Secret gets applied by accident). `platform/crafty/kustomization.yaml`
+     never names its own example either — the convention was there to copy and I missed it. Fixed by
+     not naming the file; the hand-off lives inside the example, where it belongs.
+     **An earlier draft of this entry claimed this check passed. It did not. The claim was written
+     from expectation rather than from a run.**
+  2. **`flux-postbuild-substitution` FAILED** on `${key#ENC[}` in my new guard
+     (`daemonset.yaml:155`), flagged `[SILENT-EMPTY]`: envsubst consumes every `${...}`, so my
+     shell prefix-strip would have rendered to the empty string — the ciphertext branch of the guard
+     would never have fired, which is precisely the silent failure the guard was added to catch.
+     Rewritten with `case`, which uses no braces and so has nothing to escape.
+  3. **And then it failed a THIRD time, on the fix's own comment** — `[FATAL-PARSE]
+     daemonset.yaml:156 ${...}`, the prose explaining the rule, written with the literal form. That
+     class is worse than it looks: a `FATAL-PARSE` hazard fails the **entire** Kustomization, every
+     resource in the path, not just the line. Rewritten as "dollar-brace expansion". This is the
+     second time in this component that writing the hazard literally inside a comment about the
+     hazard has been the defect (the first was in #55), so the habit to build is: **in any
+     substituted path, describe the construct in words, never type it.**
+- **After both fixes, `no-plaintext-secrets` and `flux-postbuild-substitution` pass** (446 and 198
+  assertions respectively), and `l0` is green.
+
+## What I did NOT verify
+
+- **That Flux decrypts this file on prod.** The overlay patches have never reconciled. This is the
+  whole risk of the change and it has exactly one observable outcome: after merge the pods should
+  CrashLoop reporting the **`REPLACE-ME` placeholder** (proving decryption worked), *not* reporting
+  **`ENC[`** (which would prove a patch is wrong). I built the guard to tell those apart because I
+  cannot test them.
+- **That `wg pubkey` accepts the operator's real key through this path.** The guard is reasoned from
+  `wg pubkey`'s behaviour and has never run — `wg` is absent from both this workstation's container
+  and the gate, so the first execution will be in the pod.
+- **The `sops set` command in the hand-off.** I verified `sops -e -i` and `sops -d` in the toolbox;
+  I did **not** run `sops set` with the quoting shown, and nested quoting inside
+  `dev.sh run bash -c` is exactly where a documented command tends to break. If it misbehaves, the
+  editor form (`sops <file>`) is the fallback and does not depend on quoting.
+- **The sha256 comparison in step 2 step 4** is untested end to end (it needs the real key).
+- **Whether anything else in #55 assumed a tool is available where it is not.** I checked `wg`,
+  `sops` and `qrencode` because step 2 depended on them. I did not audit every command in the
+  runbook the same way — and that class of defect has now produced two corrections in a row
+  (`site.sh`, then `sops | wg`), so assume more remain until each step has actually been run.
+- **The live VPN still carries no traffic**, and the earlier gaps stand unchanged: the
+  iptables-vs-nftables question, `allowPrivilegeEscalation: false` versus the image's s6 init, and
+  any off-LAN reachability.

decisions/2026-10-06-new-peer-ignores-template-comments.md

@@ -0,0 +1,73 @@
+---
+date: 2026-10-06
+branch: fix/new-peer-ignores-template-comments
+title: new-peer.sh refused the first VPN address because its duplicate check matched its own commented-out example
+---
+
+## What changed
+
+The operator's first real enrolment failed:
+
+```
+$ ./infra/vpn-ingress/new-peer.sh keith-phone 2
+new-peer.sh: 10.90.0.2/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet.
+```
+
+**Nothing was allocated.** `peers.yaml` had zero peers — the pod's own init container reported
+`wg0.conf assembled: 0 peer(s)`. The duplicate-address check grepped the WHOLE file, and
+`peers.yaml` line 66 is a **commented-out example stanza** I wrote, containing literally
+`#   AllowedIPs = 10.90.0.2/32`. So `.2` — the first address anybody would ask for, and the one my
+own runbook uses in its example — was permanently un-allocatable, with a message asserting the
+opposite of the truth.
+
+Both halves fixed, because either alone leaves the trap half-set:
+
+- **`infra/vpn-ingress/new-peer.sh`**: the check strips comment lines before matching
+  (`grep -vE '^[[:space:]]*#' | grep -q …`). That is the correct predicate rather than a patch: a
+  real stanza's `AllowedIPs` line is never commented, while both the template and every peer's
+  `# <name> — <owner>` label always are.
+- **`platform/vpn-ingress/peers.yaml`**: the example now uses placeholders that cannot match
+  anything real — `# <device-name> — <owner>, enrolled <YYYY-MM-DD>` and
+  `AllowedIPs = 10.90.0.<octet>/32`. A future grep cannot be fooled by it either.
+
+The bug class is worth naming: **a documentation example that is byte-identical to real data will
+eventually be parsed as real data.** The same shape caused two earlier defects in this component —
+`flux-postbuild-substitution` flagged a `${...}` written inside a comment *explaining* that hazard,
+and `no-plaintext-secrets` flagged the `.sops.yaml.example` because the kustomization named it in a
+comment. Three instances now; the rule is to make examples structurally unmatchable, not merely
+commented out.
+
+No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `decisions/**`), so this
+merges on a green gate with no override.
+
+## What I verified
+
+Run in the pinned toolbox against the live tree, 2026-10-06.
+
+- **`.2` now works**, and it went through the REAL key path: `new-peer.sh keith-phone 2` rendered a
+  profile with `Address 10.90.0.2/32`, `AllowedIPs = 10.90.0.2/32` in the printed stanza, full
+  tunnel, at `.cache/wg-peers/keith-phone.conf` (0600). The throwaway keypair was deleted after.
+- **This incidentally closes a gap left open in #60**: the profile could only have been rendered by
+  deriving the server's public key via `sops -d --extract … | wg pubkey` from
+  `wg-server-key.sops.yaml` — so that pipeline is now proven **against the operator's real key**,
+  not just against a placeholder it was expected to reject.
+- **A real duplicate is still refused.** Appending an uncommented
+  `AllowedIPs = 10.90.0.5/32` stanza and asking for octet 5 produced the correct refusal; the tree
+  was restored afterwards and `git diff --stat` showed only the intended template edit.
+- **Zero active allocations remain**: `grep -vE '^[[:space:]]*#' peers.yaml | grep -c AllowedIPs`
+  → `0`.
+- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
+  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips (`gitea-settings`,
+  conformance L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **No peer is enrolled by this branch.** The keypair I generated was a throwaway and was deleted;
+  the operator must run the script themselves, since the device's private key must exist only on
+  the device. So the end of the enrolment path — a stanza merged into `peers.yaml`, Flux rolling
+  the DaemonSet, and `wg show` listing a peer — is still unexercised.
+- **The duplicate-NAME warning** (`grep -qi "^ *# *${NAME} "`) deliberately still inspects
+  comments, because per-peer labels are comments. It is no longer tripped by the template, but I
+  did not test it against a real enrolled peer.
+- **Nothing about client traffic.** The WAN port-forward is still not configured and there is still
+  no off-LAN vantage point, so the tunnel has carried no packet.

decisions/2026-10-06-peers-configmap-content-hash.md

@@ -0,0 +1,103 @@
+---
+date: 2026-10-06
+branch: fix/peers-configmap-content-hash
+title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
+---
+
+## What changed
+
+The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
+2026-10-06, with the peer already on `main`:
+
+| observation | value |
+|---|---|
+| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
+| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
+| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
+| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
+| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
+| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
+| init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
+
+The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
+fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
+no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
+into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
+would have had **no effect at all** until something unrelated happened to restart the pods. For a
+*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
+see a green PR, and the device would keep working.
+
+My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
+never true.
+
+**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
+*"The generated name carries kustomize's content hash, so editing any runbook rolls the server
+automatically and a stale projection is impossible."*
+
+So the register becomes a **content-hashed generator**:
+
+- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
+  plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
+  carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
+  documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
+  stripped.
+- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
+  ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
+  `nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
+  the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
+  `wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
+  by building the output and reading the volumes block.
+- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
+  register is now **fatal** instead of skipped. The old guard was
+  `[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
+  duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
+  as a defect in its own right.
+- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
+  `networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
+  operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
+- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
+  `generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
+  merge.
+
+Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
+
+No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
+`decisions/**`), so this merges on a green gate with no override.
+
+## What I verified
+
+Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
+
+- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
+  namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
+  `namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
+  bearing, measured both ways.
+- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
+  append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
+  and reversible, so an edit necessarily changes the pod template.
+- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
+  `peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
+  `main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
+- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
+  NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
+  transformer did not rewrite it).
+- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
+  "append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
+  real `.2` now occupies it — the first time the duplicate check has been exercised against real
+  data rather than against its own example.
+- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
+  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
+  as rc 3.**
+
+## What I did NOT verify
+
+- **That the roll actually happens on the cluster.** The whole point of the change is unproven
+  until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
+  `wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
+- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
+  port-forward is reportedly configured but has never been exercised by a packet I can see, and I
+  have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
+- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
+  should now roll the pods and drop the peer; untested.
+- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
+  `prune: true` on the Kustomization, not observed.

decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md

@@ -0,0 +1,126 @@
+---
+date: 2026-10-06
+branch: docs/pm-agent-registry-pull-prereq
+title: The nodes resolve gitea to the WAN, so the failing certificate is the router's — the second node process to need the issuer pin
+---
+
+## What changed
+
+The image pull for `pm-agent` failed TLS verification, and the previous entry on this branch
+honestly recorded it as **not diagnosed**. The operator asked for the diagnosis before any
+registry design. This is it, and the fix turns out to be a pattern this repository already
+has.
+
+`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins
+`gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the
+Keycloak issuer pin that is there for the identical reason.
+
+**It reaches the nodes the way every other node setting in this project does — through
+Ansible, from the inventory.** Node configuration is Ansible's (and, for the dev VMs,
+OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the
+*only* surface: `infra/PROD-PROVISIONING.md` records that prod is 3× bare metal and that the
+dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no
+cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command:
+
+    cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
+      -e k3s_hosts=k3s_prod --tags hosts
+
+`--tags hosts` runs ONLY the `/etc/hosts` task, so applying this pin cannot drift any other
+baseline setting. Running the play is an operator action (it has become/root on prod), but
+the configuration itself is declared, reviewed and version-controlled — not typed on a node.
+
+The docs that described this as an undiagnosed blocker now describe the cause and the fix:
+`platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and
+the play), and `docs/plans/pm-agent-issue-to-pr.md`.
+
+Also recorded, in the two places an operator reads when creating the credential: **the agent
+authenticates to Gitea with a local account and a scoped PAT, by decision** (operator,
+2026-10-06), not with a federated identity. Written down because the alternative looks
+obviously better until you check whether it is possible — see below.
+
+## What I verified
+
+Every step measured, read-only, in this order.
+
+**The failing certificate is the router's, not the Gateway's.**
+
+- containerd's error names TLS, not the digest: `tls: failed to verify certificate: x509:
+  certificate is not valid for any names, but wanted to match gitea.home.keithjackson.dev`.
+  "not valid for **any** names" means the presented certificate carries no SANs at all.
+- `192.168.2.20` is the shared Gateway's own LoadBalancer VIP
+  (`gateway/cilium-gateway-shared`, from the Cilium pool `192.168.2.20-30`), and its `https`
+  listener serves `*.home.keithjackson.dev` with `wildcard-home-tls`. From the operator's
+  workstation the name resolves there and serves the real Let's Encrypt
+  `CN=*.home.keithjackson.dev` with correct SANs. **So the certificate and the Gateway are
+  healthy.**
+- **The node asks a different resolver and gets a different answer.** Prod's
+  `node_baseline_dns_servers` is the LAN gateway, not Pi-hole — deliberately, as that file
+  explains. Public DNS for this zone is a CNAME to `wan.home.keithjackson.dev`: measured via
+  both `1.1.1.1` and `8.8.8.8`, `gitea.home.keithjackson.dev` →
+  `wan.home.keithjackson.dev.` → `136.51.115.252`, and the apex answers the same.
+- **That address serves a certificate with no names.** Measured from the workstation against
+  the WAN address with the right SNI:
+
+      subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
+      issuer=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
+      No extensions in certificate
+
+  No extensions means no `subjectAltName`, which is exactly "not valid for any names". The
+  `CN=%D` is the router's placeholder.
+
+**This repository had already measured the same thing, for Keycloak, and I did not need to
+re-derive it.** `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` says, above the pin:
+"Since platform/gateway/gateway.yaml set an external-dns `target`, every public name is a
+CNAME to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from
+the LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
+k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18)." Same cause, same symptom,
+one name later. The pin exists precisely for node processes that must reach the Gateway
+directly; containerd is one, and nobody had added it.
+
+**The parity the pin has to satisfy:** `192.168.2.20` equals the live
+`GATEWAY_VIP`, and `k3s_base_domain` equals the live `BASE_DOMAIN` — both read from the
+cluster, not from the comment that claims prod serves `prod.home.keithjackson.dev` (it does
+not; `flux-system/cluster-vars` reports `BASE_DOMAIN=home.keithjackson.dev` and
+`GATEWAY_VIP=192.168.2.20`, and Ansible's `k3s_base_domain` is `home.keithjackson.dev`).
+That comment is stale history.
+
+**The credential decision, and why the obvious alternative is not available.** Gitea's API
+accepts basic auth, PATs, SSH signatures and its own OAuth2 bearer tokens; as an OAuth2
+provider Gitea implements **only the Authorization Code Grant** (plus PKCE/OIDC), so there
+is no client-credentials grant, and nothing in Gitea validates a token issued by a
+third-party OIDC provider. A Keycloak service-account token therefore cannot authenticate to
+Gitea's API or to git-over-HTTP. Read from Gitea's own documentation
+(`docs.gitea.com/development/api-usage`, `/development/oauth2-provider`), not assumed.
+
+**Gate:** `./ci/gate.sh` in the pinned toolbox from a §0 worktree — see the branch's run; rc 3
+with `gitea-settings` the only stage that did not run.
+
+## What I did NOT verify
+
+- **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the
+  nodes is written by the `node_baseline` play's `hosts`-tagged task, which is an operator
+  action. I did confirm the mechanism is not inert — `roles/node_baseline/tasks/main.yml`
+  renders `node_baseline_hosts_pins` into a marked `blockinfile` with `backup: true`, and
+  `state: absent` when the list empties, so a removed entry really leaves the node. Until that runs
+  and an image exists, "the node now reaches the Gateway" is a design claim. The evidence
+  that it *will* is the Keycloak pin working for the same cause — which I also did not
+  re-measure.
+- **Why public DNS answers at all for an `internal` service.** `catalog/gitea.yaml` sets no
+  `exposure`, so it inherits `internal`, and the renderer contract says an internal service's
+  HTTPRoute carries `external-dns.alpha.kubernetes.io/controller: none` and publishes no
+  public record. Yet `gitea.home.keithjackson.dev` resolves publicly to the WAN CNAME. I did
+  not chase that: it may be a zone-level wildcard rather than a published record, and it did
+  not change the diagnosis. It is worth someone's attention, because "internal services
+  publish no public DNS" is an L0-checked claim and this looks like a case where the *name*
+  resolves regardless of what the renderer withheld.
+- **Registry authentication.** Unchanged and still open: an unauthenticated client gets
+  nothing from this registry, and no `imagePullSecret` exists anywhere in the tree. With the
+  credential model now decided (local Gitea accounts with PATs) the shape is a
+  `dockerconfigjson` for an account whose token carries `read:package`, referenced from the
+  Deployment and the committed worker Job template — and the worker Job template's shape is
+  pinned by admission, so whether the policy should also pin `imagePullSecrets` deserves a
+  deliberate look rather than a quiet addition. Not built: minting that token is an operator
+  action, and I did not want to guess at a second credential's scope.
+- **Anything about runtime.** The image still does not exist. No dispatcher has
+  authenticated, no issue has been triaged, no worker Job has been created, and the admission
+  policy — live, `Enforce`, six rules — has still never admitted or refused a real Job.

decisions/2026-10-06-pm-agent-registry-pull-secret.md

@@ -0,0 +1,104 @@
+---
+date: 2026-10-06
+branch: feat/pm-agent-registry-pull-secret
+title: The kubelet's pull credential — the second half of making agent-runner pullable, and the seventh field the admission policy had not read
+---
+
+## What changed
+
+The registry had two blockers. The first — the nodes resolving the registry hostname to the
+non-hairpinned WAN and failing TLS against the router's `CN=%D` certificate — was diagnosed
+and fixed by an `/etc/hosts` pin, merged in #59 and applied by the operator. This is the
+second: **the pull is unauthenticated, and this registry refuses unauthenticated clients
+outright.**
+
+`pm-agent-registry` is a new `kubernetes.io/dockerconfigjson` Secret, SOPS-encrypted with
+placeholder values, referenced by `imagePullSecrets` on both the dispatcher Deployment and
+the committed worker Job template. It is a **second Secret holding the same Gitea token** as
+`pm-agent-credentials`, because the two consumers need two formats: the application reads a
+raw token from env, the kubelet reads a dockerconfigjson before any container starts. That
+duplication is the established pattern here (the Keycloak client secret is mirrored the same
+way, for the same reason) and it is documented in both `.example` files.
+
+The operator's part is the value, and one scope: the Gitea token must now carry
+**`read:package`** alongside `write:repository` and `write:issue`. Steps are in
+`platform/pm-agent/pm-agent-registry.sops.yaml.example`, including the trap that
+`.dockerconfigjson`'s registry key must be the bare host as the image reference spells it —
+with a scheme or a port it matches nothing, the kubelet pulls anonymously, and the failure is
+indistinguishable from the Secret being absent.
+
+**The admission policy gained a seventh rule.** `imagePullSecrets` is a pod-spec field that
+no container references, which is exactly why it is easy to leave unread — the same shape of
+omission as `initContainers`, `ephemeralContainers`, `env` and `envFrom` before it. The
+kubelet reads the named Secret, in this namespace, before anything runs. Rule 1's pattern now
+REQUIRES the committed entry and rule 7 forbids any other or any second one; both halves are
+needed because a `foreach` over an absent list yields no result.
+
+The leverage is small and worth saying so: the image is digest-pinned, so a different pull
+credential cannot change *what* runs, and the only Secrets in this namespace are the agent's
+own. It is pinned anyway, because "only the committed shape" is this policy's whole premise
+and the rule costs four lines.
+
+## What I verified
+
+- **`kyverno test platform/kyverno/tests/pm-agent` → 33 passed, 0 failed** under the pinned
+  CLI (was 28 before this change). Three new fixtures, each a real path rather than a style
+  violation: `extra-pull-secret` (the committed entry PLUS a second one — rule 1's array
+  pattern is satisfied by the first element, which is why the count is asserted separately),
+  `wrong-pull-secret` (a different Secret in this namespace), and `no-pull-secret` (absent
+  entirely, refused by rule 1's pattern because rule 7's foreach cannot see it).
+- **`PM_SELFTEST=1 pm-dispatch` → exit 0**, now also asserting that the committed Job
+  template names exactly `pm-agent-registry` — so the template and the policy cannot drift
+  apart without the image build failing.
+- **`./ci/gate.sh` in the pinned toolbox, from a §0 worktree → exit 3 (UNDETERMINED)**, with
+  `gitea-settings` the only stage that did not run (no `SETTINGS_API_TOKEN` on this
+  workstation). Every other stage green, `spec-immutability` included. rc 3 is not a pass.
+- **The gate caught me shipping the Secret in plaintext.** I wrote the placeholder
+  `pm-agent-registry.sops.yaml` and never ran `sops -e -i` on it. `no-plaintext-secrets`
+  refused it with both halves named — "has no top-level `sops:` block with a mac" and
+  "Secret value '.dockerconfigjson' is not SOPS ciphertext" — which is exactly the check
+  earning its place: the file looked finished, and the value was a placeholder, so nothing
+  about reading the diff would have made it obvious. Encrypted, and the stage is green.
+- **The registry really does refuse anonymous clients**, measured before building this
+  (2026-10-05): the anonymous registry token's `access` claim is `null`, every `/v2/…`
+  path answers 401 whether it exists or not, and `/api/v1/packages/Home-Network` replies
+  `{"message":"token is required"}`. That is why this is a credential and not a posture
+  tweak.
+
+## What I did NOT verify
+
+- **That the pull now works.** Nothing has been pushed to that registry and the committed
+  digest is still the all-zero placeholder, so neither half of the registry fix has been
+  exercised end to end. The first real pull is what proves both at once — and if it fails, the
+  error names which half: `x509 … not valid for any names` is the resolution pin, `401` /
+  `unauthorized` is this credential.
+- **That the `/etc/hosts` pin from #59 took effect.** The operator applied it; I could not
+  confirm the effect, because the dispatcher pod's retained events are all repeating
+  `BackOff` with no fresh `Pulling`, so the `x509` message still visible is the one from
+  before the play ran. Not evidence either way.
+- **Whether `read:package` is sufficient** for a pull from a Gitea package of this type. It is
+  the documented read scope for packages; nothing here has exercised it, because no package
+  exists yet.
+- **The dockerconfigjson placeholder is not a valid credential and is not meant to be.** Its
+  `auth` field is a `REPLACE-ME` string rather than base64, so a pull with it will fail at
+  authentication rather than be mistaken for working.
+- **Anything about runtime.** Unchanged from the previous entries: the image has never been
+  built or pushed, no dispatcher has authenticated to Gitea, no issue has been triaged, no
+  worker Job has ever been created, and the admission policy — live, `Enforce`, now seven
+  rules — has still never admitted or refused a real Job.
+
+**Addendum 2026-10-30 (rebased onto #81/#82/#83; see `decisions/2026-10-30-pm-agent-live.md`).**
+Three statements above are now out of date, and one was wrong when written:
+- *"Nothing has been pushed to that registry"* — no longer true. CI published agent-runner in
+  images run 340 (`sha-523cd9118f95` = `main`, digest `sha256:c4eba5b3…6eb42`), and this
+  branch now pins it.
+- *"That the `/etc/hosts` pin from #59 took effect"* — now measured: all three prod nodes
+  carry the line, resolve the name to `192.168.2.20`, and get `401` with `ssl_verify_result=0`
+  from `https://gitea.home.keithjackson.dev/v2/` (see the new entry).
+- *"Whether `read:package` is sufficient"* — the scope is necessary but not the whole answer:
+  org-package access is decided by account permission (team Packages unit, or org
+  visibility). Handled in the registry `.example`.
+- **Wrong when written:** the scope list. Gitea 1.24.3 puts `GET /api/v1/user` — the
+  dispatcher's first call — behind the `user` scope category, so a token with only
+  `write:repository`, `write:issue` and `read:package` CrashLoops the dispatcher on startup.
+  The token needs `read:user` too.

decisions/2026-10-06-vpn-ingress-ip-forward.md

@@ -0,0 +1,113 @@
+---
+date: 2026-10-06
+branch: fix/vpn-ingress-ip-forward
+title: The VPN tunnel never came up — a hostNetwork pod cannot write net.ipv4.ip_forward, and wg-quick treats a failed PostUp as fatal
+---
+
+## What changed
+
+The remote-access VPN reconciled for the first time with a real server key (#60 then #62) and
+**the tunnel did not come up**. One line in my own manifest was the whole cause.
+
+Measured in the `wireguard` container on prod, 2026-10-06:
+
+```
+[#] ip link add dev wg0 type wireguard
+[#] wg addconf wg0 /dev/fd/63
+[#] ip -4 address add 10.90.0.1/24 dev wg0
+[#] ip link set mtu 1420 up dev wg0
+[#] sysctl -q -w net.ipv4.ip_forward=1
+sysctl: permission denied on key "net.ipv4.ip_forward"
+[#] ip link delete dev wg0
+**** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! ****
+```
+
+Everything up to that point worked. **The cause is not a missing capability.** With
+`hostNetwork: true` the runtime mounts `/proc/sys` **read-only** in the container — confirmed in the
+pod's own `/proc/mounts`: `proc /proc/sys proc ro,nosuid,nodev,noexec,relatime` — precisely because
+those sysctls are the *host's* globals and a container must not rewrite them. `CAP_NET_ADMIN` cannot
+lift a read-only mount, so this can never succeed from here. And `wg-quick` treats any failed
+`PostUp` as fatal, so it deleted the interface it had just finished building.
+
+**The line was never needed.** `node_baseline` writes `net.ipv4.ip_forward=1` to
+`/etc/sysctl.d/99-k3s-cilium.conf`, and reading the value from inside the pod returns `1`. I had
+written the header comment *correctly* — "ip_forward is ALREADY 1 on these nodes … so no pod sysctl
+is needed" — and then emitted the sysctl anyway, justified in its own comment as "re-asserted so
+this component does not silently depend on another role's sysctl file surviving". That defensive
+re-assertion is the entire outage. **The lesson is narrow and reusable: in a hostNetwork pod,
+verify host state, never write it.**
+
+Fix:
+
+- **`platform/vpn-ingress/daemonset.yaml`**: the `PostUp = sysctl …` line is gone, replaced by a
+  **read** in the init container — `grep -qx 1 /proc/sys/net/ipv4/ip_forward` — which fails closed
+  with a message naming the `node_baseline` role. Reading `/proc/sys` is permitted; only writing is
+  not. Fail-closed still matters: a tunnel that comes up but cannot forward would satisfy every
+  probe in this component and silently drop every client's traffic.
+- The header bullet that said "no pod sysctl is needed" now also says **do not attempt one**, with
+  the mechanism, so the next author does not re-add it for the same reason I did.
+- The `securityContext` comment claimed NET_ADMIN was for "create the wg0 link, set the host sysctl,
+  and write the netfilter rules". The middle clause was false; corrected.
+- The image's `chown: … Operation not permitted` startup run — plus its own "Permissions could not
+  be set … we will not provide support for it" — is now documented as **expected and harmless**, in
+  both the manifest and the runbook, so nobody chases it. Capabilities are dropped to `NET_ADMIN`
+  only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs as root,
+  and `wg-quick` proceeds past it (as it did here — the failure was 30 lines later).
+- **`runbooks/vpn-remote-access.md`**: a "what a healthy start looks like" block with the real log
+  lines; `ip_forward is 0` added to the init-failure table; and the note that `0/1 Running` with
+  `Startup probe failed: Unable to access interface: No such device` is the signature of `wg-quick`
+  tearing the tunnel down — read the `[#]` line it died on.
+- **`platform/vpn-ingress/README.md`**: a table of what the first reconcile settled (below).
+
+No sensitive paths: `platform/vpn-ingress/**`, `runbooks/**` and `decisions/**` only. This merges on
+a green gate with no override.
+
+## What I verified
+
+Live prod, read-only, 2026-10-06. **Four questions this component's design rested on are now
+measured rather than assumed — three went the right way, one was the defect above.**
+
+- **The GitOps path is proven end to end.** `Kustomization vpn-ingress` `lastAppliedRevision:
+  main@sha1:cf7d5e4` equals the GitRepository artifact revision. The Secret carries
+  `kustomize.toolkit.fluxcd.io/name: vpn-ingress`, so Flux delivered it.
+- **SOPS decryption works, proven by digest rather than by eye.** The value in
+  `platform/vpn-ingress/wg-server-key.sops.yaml` on `main`, decrypted, is 44 bytes, is **not** the
+  placeholder, and **`wg pubkey` accepts it**. Its SHA-256 equals the SHA-256 of the live Secret's
+  decoded `data.privatekey` (`d7c8f8de6f252649…` both sides). No key material was printed at any
+  point — lengths, booleans and digests only.
+- **The `wg pubkey` guard fired correctly, and its `ENC[` branch was never needed.** During the
+  kubelet window before the updated Secret propagated, all three pods reported *"It is still the
+  committed REPLACE-ME PLACEHOLDER"* — the guard distinguishing a stale/unfilled value from
+  ciphertext, exactly as designed. It cleared on its own once the mount refreshed; **a transient
+  `REPLACE-ME` CrashLoop for a minute or two after the key PR merges is expected, not a fault.**
+- **The kernel module reaches the pod**: `ip link add dev wg0 type wireguard` succeeded and
+  `/proc/modules` lists `wireguard`. So the `node_baseline` addition works and `CAP_SYS_MODULE` is
+  correctly not required.
+- **`allowPrivilegeEscalation: false` does NOT break the image's s6 init** — the container reaches
+  `wg-quick` normally. The flag stays; the fear recorded in #55 is closed.
+- **The iptables backends MATCH, which was my single biggest documented doubt.** The image ships
+  `iptables v1.8.13 (nf_tables)` via `/usr/sbin/xtables-nft-multi`, and Debian 13 is nf_tables. The
+  nightmare case — rules accepted into a table the kernel never consults, so handshakes succeed and
+  every forwarded connection hangs — cannot occur on this pairing.
+- **`net.ipv4.ip_forward` is `1`** in the host netns the pod shares, and `/proc/sys` is `ro` there.
+  Both read directly from the pod.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
+  environmental skips (`gitea-settings`, conformance L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **That the tunnel now comes up.** This fix has not reconciled. The next reconcile is the test, and
+  its success criterion is specific: pods `1/1 Running`, and `wg show` reporting `interface: wg0`
+  with `listening port: 51820`.
+- **That the MASQUERADE and FORWARD rules actually apply.** They are the next statements after the
+  one that failed, so **they have never executed**. The backends matching (above) removes the worst
+  failure mode but proves nothing about the rules themselves. First check after the pods go Ready:
+  `iptables -t nat -S POSTROUTING | grep 10.90.0.0/24` on a node.
+- **Anything about client traffic.** Still zero peers enrolled, still no WAN port-forward, still no
+  off-LAN vantage point. The VPN has carried no packet.
+- **Whether the transient `REPLACE-ME` window has an upper bound.** It cleared within ~2 minutes
+  here; I did not determine the kubelet refresh behaviour, so a longer stall after a key change is
+  uncharacterised.
+- **The `sops set` hand-off command**, still. The operator filled the key in by their own route
+  (#62), so the exact quoting I documented inside `dev.sh run bash -c` remains unexercised.

decisions/2026-10-07-bulk-nfs-truenas.md

@@ -0,0 +1,85 @@
+---
+date: 2026-10-07
+branch: fix/bulk-nfs-truenas
+title: Repoint bulk NFS at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
+---
+
+## What changed
+
+Both ZFS pools were physically moved off the Proxmox host onto a standalone TrueNAS Community
+Edition at **192.168.2.30**, which sits on prod's own `192.168.2.0/24`. `BULK_NFS_SERVER` moves from
+`192.168.2.102` (the pfSense DNAT prod used to mount through) to `192.168.2.30`.
+
+- `clusters/prod/kustomization.yaml` — `BULK_NFS_SERVER` → `192.168.2.30`. The comment block that
+  justified the DNAT, and the "NOT YET DECIDED / THIS PLACEHOLDER IS LOAD-BEARING" banner, are
+  replaced with the measured export configuration and the recreate requirement below.
+- `clusters/base/cluster-vars.yaml` — base default `192.168.1.9` → `192.168.2.30`. That host no
+  longer exists.
+- `clusters/dev/kustomization.yaml` — same; marked explicitly **inert**, since dev is decommissioned
+  and its nodes are gone. It carries the real address only so nothing in the tree renders a dead one.
+- `docs/BULK-NFS-STORAGE.md` — §1 and §5 rewritten for TrueNAS (exports, service settings, the ZFS
+  `sharenfs` properties that had to be cleared, and the ACL hazard); §7 rewritten now that the
+  placeholder is gone; §9 retargeted from dev to prod; **new §10** documents the PV
+  delete-and-recreate. §2/§3/§4/§6 are unchanged by the move.
+- `catalog/qbittorrent.yaml`, `platform/monitoring/node-pressure-dashboard.yaml` — two comment/prose
+  references to `192.168.2.102` as the live export address.
+
+**The headline operational fact: repointing this knob is NOT a plain reconcile.** A bound PV's NFS
+source is immutable, so Flux's apply fails rather than updating and the pods keep their old, dead
+mount. The five PVs must be deleted and recreated. That sequence is §10 and is an operator action;
+it is **not** performed by this PR.
+
+## What I verified
+
+**On the NAS (192.168.2.30), measured 2026-10-07 by the operator at my request:**
+
+- `zfs list` — `Storage` → `/mnt/Storage`, `Storage/plex` → `/mnt/Storage/plex`,
+  `Storage/nextcloud` → `/mnt/Storage/nextcloud`. The PV paths resolve unchanged.
+- Ownership intact: `/mnt/Storage/plex`, `tv/`, `downloads/`, `movies/`,
+  `downloads/complete/`, `nextcloud/nextcloud/` all `drwxr-xr-x 1000:3000`. `movies/` 1171 entries
+  and `tv/` 107, against 1165 / 107 measured on Proxmox.
+- `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`), `aclinherit=passthrough`, and no
+  non-trivial ACLs — mode bits still govern.
+- Three stale `sharenfs` dataset properties (`plex`, `nextcloud`, `docker-volumes`) carried over from
+  Proxmox; cleared with `zfs inherit`, re-checked empty.
+- `exportfs -v`: both shares to `192.168.2.10/.11/.12` with `sec=sys,secure,root_squash,no_all_squash`,
+  `rw` on plex and `ro` on nextcloud.
+- `/proc/fs/nfsd/versions` → `+3 +4 +4.1 +4.2`, so the PVs' `nfsvers=4.1` negotiates.
+- **A defect was found and fixed during setup:** the plex share listed `192.168.2.13` — an address no
+  node holds — and so omitted `192.168.2.12` (k3s-prod-3). Corrected and re-verified.
+
+**From the cluster:**
+
+- Node InternalIPs are `192.168.2.10/.11/.12` (`kubectl get nodes`), confirming `.13` was wrong.
+- All five NFS PVs currently name `192.168.2.102`.
+- `192.168.2.30:2049` and `:111` accept connections; `192.168.2.102:2049` does not.
+- **`spec.nfs` immutability, by server-side dry-run** — this is the finding that shapes the change:
+  `spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation`.
+- The pods report `Ready` while their mounts are dead: a `stat /media` answers from attribute cache,
+  but a `readdir` blocks indefinitely (the PVs mount `hard`). Hence "scale to zero first" in §10.
+
+**Gate:** see the PR; run in the pinned toolbox from this worktree.
+
+## What I did NOT verify
+
+- **Nothing has mounted 192.168.2.30.** No pod has touched the new server — the PVs still point at
+  the old address and cannot be updated in place. Every §9 probe (numeric-id passthrough, the uid-1000
+  write, the hardlink/EXDEV check, Plex's read-only posture) is **outstanding**, not passing. The
+  equivalents passed against the old host on 2026-09-12; that is not evidence about this one.
+- **The §10 sequence is reasoned, not executed.** Retain + the renderer's pre-set `claimRef` are read
+  from the manifests and from `catalog/render.py`; I have not watched a PVC rebind after a PV delete
+  in this cluster.
+- **`root_squash` is a behaviour change from the Proxmox export**, which carried `no_root_squash`. The
+  argument that nothing needs root on the tree (LSIO drops to 1000; the images chown only their
+  Longhorn `/config`; both subPath directories already exist at 0755) is reasoned from the images and
+  the measured permissions. It has not been exercised. If a mount or subPath resolution fails with
+  EACCES, Maproot `root:root` reproduces the old export and is the first thing to try.
+- **Whether NFSv3 can be disabled.** Enabled Protocols is `NFSv3, NFSv4`; the PVs only need v4.1, so
+  narrowing would close the rpcbind/mountd/statd surface. Not done — it is a change to make after the
+  cutover is proven, not during it.
+- **Whether pfSense / the legacy stack is actually retired.** I measured only that
+  `192.168.2.102:2049` stopped answering. The other legacy consumers of these datasets
+  (`192.168.1.6`, `10.123.1.10`, `192.168.1.18`) were dropped from the new exports; if any is still
+  live, it has lost access. Not checked.
+- **L2/L3 conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline subset
+  only. Nothing in the gate exercises an NFS mount.

decisions/2026-10-17-automated-upgrades.md

@@ -0,0 +1,300 @@
+---
+date: 2026-10-17
+branch: feat/automated-upgrades
+title: Automated upgrades — kured + security-only unattended-upgrades, and Renovate finally gets a runner
+---
+
+## What changed
+
+Two halves of one process. Neither is useful alone and they must land together.
+
+### (a) OS upgrades and reboots
+
+**`platform/kured/`** — the Kubernetes Reboot Daemon, chart `6.1.0` / app `1.23.0`, in its own Flux
+Kustomization (`clusters/base/kured.yaml`, `dependsOn: [platform]`, `wait: true`), registered in
+`clusters/base/kustomization.yaml`. Not a line in `platform/kustomization.yaml`: that tier is
+`wait: true` and gates `platform-config` → every apps wave, so a wedged reboot daemon placed there
+could stall every application in the cluster. A component added to make maintenance safer must not
+be able to cause an outage.
+
+**`infra/ansible/roles/unattended_upgrades/`** + `playbooks/unattended-upgrades.yml`, enabled for
+prod in `inventory/group_vars/k3s_prod/vars.yml`. A **new** role rather than an extension of
+`node_baseline` — different lifecycle, and other open branches touch that role.
+
+**The design, and how it answers this repo's two standing objections to reboot automation.**
+`roles/node_baseline/tasks/main.yml:61-67` says "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm
+reboot 2026-09-03; physical power button required), and `docs/plans/cluster-upgrade-plan.md:291-299`
+rejected system-upgrade-controller for "automating away the human gate between nodes". Both still
+stand. **The human gate is the sentinel.** unattended-upgrades is security-only with the kernel
+blacklisted, so it essentially never produces `/var/run/reboot-required`; that file appears when an
+operator deliberately runs `node-baseline.yml --tags kernel`. kured then performs only the
+*mechanical* half the operator would otherwise type by hand — including the ordering rule
+`node_baseline`'s own output has to shout ("NEVER node 2 first"), which the operator keeps by
+running the kernel play with `--limit <host>` one node at a time. `cluster-upgrade-plan.md:329-330`
+names the condition under which its own rejection reverses (fast disks), and prod is that cluster.
+
+**Quorum: concurrent reboots are structurally impossible.** All three nodes are control-plane+etcd
+(quorum 2/3). `concurrency: 1` plus kured's single cluster-wide lock annotation on its own DaemonSet
+means a node reboots only while holding that lock, enforced by API-server optimistic concurrency
+rather than by timing. **`lockTtl: 0`** (expiry disabled) is the deliberate choice: a TTL exists so
+reboots continue after the lock holder dies, which on 3 etcd nodes is exactly the catastrophe — a
+hung node's lock expiring would let a second node take the cluster to 1/3. So a hang stops the cycle
+forever and pages a human. `lockReleaseDelay: 30m` holds the lock after a node returns.
+
+**The Longhorn gate.** Measured: 25 volumes, all `numberOfReplicas: 3`, one replica per node, with
+`replica-soft-anti-affinity: false` and `disable-scheduling-on-cordoned-node: true`. So every drain
+takes **all 25 volumes to degraded 2/3**, and recovery is serialized —
+`replica-replenishment-wait-interval: 600`, `concurrent-replica-rebuild-per-node-limit: 1`. Reboot
+the next node too early and volumes reach 1/3, where `node-drain-policy:
+block-if-contains-last-replica` can block a drain forever. The inter-node gate is therefore **volume
+robustness, not pod readiness**, and it is doubled: `lockReleaseDelay: 30m` (unconditional, covers
+the 600s wait) plus kured's Prometheus alert gate (semantic, covers a rebuild longer than 30m).
+
+**`alertFilterMatchOnly: true` is load-bearing.** The flag inverts the regexp's meaning: `false`
+(chart default) makes it an *ignore* list and kured blocks on **any** firing alert. Measured
+2026-10-17, five alerts were firing with no end in sight (`CNPGPrimariesCoLocated`,
+`KubeDeploymentReplicasMismatch`, `KubeDeploymentRolloutStuck`, `KubePodNotReady`,
+`OAuth2ProxyGateUnobserved` — the last three all tracing to `pm-agent`, ImagePullBackOff on a
+placeholder digest for 12 days). Under the default, kured would have been blocked on day one and
+never rebooted anything, indistinguishable from "no reboots needed". The allow-list is anchored:
+`^(LonghornVolumeDegraded|LonghornVolumeFaulted|KubeNodeNotReady|NodeExporterDown)$`.
+`NodeEtcdRequestsSlow` and the `NodeIOPressure*` family are **deliberately excluded** despite being
+exactly what you'd want to pause for — prod-1 has a documented etcd WAL I/O starvation history and
+including them risks a permanent wedge that looks like safety.
+
+**The kernel floor (the highest-risk item), handled with two independent exclusions.** Measured: all
+three nodes run `7.1.8+deb13-amd64` (`7.1.8-1~bpo13+1`, the backports metapackage), far above the
+6.13 memattr floor — **but `linux-image-6.12.107` and `linux-image-6.12.94` are also still
+installed, below the floor, bootable, and `apt-mark showhold` is empty on every node.** So: (1) the
+allowed-origins list is `label=Debian-Security` only, and the running kernel comes from
+`o=Debian Backports` (pin 100, the `NotAutomatic`+`ButAutomaticUpgrades` signature), so u-u cannot
+*see* a kernel upgrade; (2) `^linux-` is blacklisted anyway, so widening origins later — the most
+likely future edit — cannot re-arm the hazard. The role **asserts both against `apt-config dump`**,
+apt's own parse of the whole directory, rather than trusting the file it just wrote. Also blacklisted:
+`intel-microcode`, `amd64-microcode`, `grub-*`. `Remove-Unused-Kernel-Packages` is **false** against
+the Debian default, because those older kernels are the documented rollback path. None of this stops
+the kernel moving — the operator's deliberate Ansible run still does that, unchanged.
+
+**The silent-no-op trap this closes.** Measured: `update-notifier-common`, `needrestart` and
+`unattended-upgrades` were **absent on all three nodes**, so *nothing could create
+`/var/run/reboot-required`*. kured deployed alone would have been permanently idle while 1/1 Running
+and reporting `kured_reboot_required=0` — green forever. The role installs the sentinel writer and
+**asserts a hook exists** by discovery (not a hard-coded path), failing loudly if not.
+`needrestart` runs in auto mode with `k3s.service` blacklisted — auto-restart is what makes a
+security-only policy effective, but restarting the control plane unattended is worse than a delayed
+library fix. (k3s is not an apt package at all here: no `k3s`, no `containerd` in dpkg.)
+
+**Alerts + runbook.** `platform/kured/alerts.yaml` — three rules over kured's *only* metric
+(`kured_reboot_required`, one GaugeVec, read from `cmd/kured/main.go:92-96`) plus `kube_node_info`:
+`KuredNotReporting` (`absent()` — the pipeline is dead and the other two are inert),
+`KuredNodeCoverageGap` (`count(kured) < count(nodes)` — a node silently uncovered, which `absent()`
+cannot see), `KuredRebootPending` (`for: 36h` — a reboot was offered and blocked). Companion
+promtool suite `alerts.promtool-test.yaml` pins all three both ways, including that the rule
+**stops** when a reboot succeeds. `runbooks/node-reboot-automation.md` registered in
+`runbooks/kustomization.yaml`.
+
+**The window is DAILY 03:00-06:00, and that choice is driven by the alert.** A weekly window means a
+pending reboot legitimately waits 6 days, forcing `for: 8d`+ — longer than this cluster's 7d
+Prometheus retention and reset by any restart past the 1h for-outage-tolerance. Daily caps the
+legitimate wait under 24h and makes `for: 36h` both robust and meaningful.
+
+### (b) Image / dependency automation
+
+**Renovate was aspirational. It now has a runner.** `.gitea/workflows/renovate.yaml` — scheduled
+daily 03:15 UTC, `workflow_dispatch` with `dry_run` defaulting to **true** for a safe first run.
+Preflight **fails hard (exit 1)** if `RENOVATE_TOKEN` is missing rather than skipping, because a
+skipped Gitea job reports `success` and that is precisely how `renovate.json` stayed inert.
+
+**`renovate.json`: Renovate now owns both halves, and Flux image-automation is formally closed.**
+ADR-0009 gave container image tags to Flux image-automation; that cannot be delivered on this
+repo's terms. (1) `ImageUpdateAutomation` **pushes commits, it does not open PRs**, so it bypasses
+the CLAUDE.md §1 gate; pointed at a side branch instead, nothing opens the PR. (2) It cannot work
+here anyway: `main` is protected `user_can_push: false` and the `flux-system` GitRepository has **no
+`spec.secretRef`** — the source is anonymous and read-only. (3) Enabling it means an in-cluster git
+**write** credential for the repo that defines the cluster. So `docker`/`kubernetes` — disabled
+*only* to avoid fighting that controller — are enabled, with `kubernetes.managerFilePatterns`
+covering `platform/`, `platform-config/`, `apps/`, **`catalog/`** (apps are generated from it; a bump
+only to the rendered output is reverted by the next render and fails `ci/render-golden.sh`),
+`runbooks/serving/` and `infra/agent/`. Added a no-automerge `blast-radius` rule for `kured`,
+`longhorn`, `cilium`, `cloudnative-pg`, `kube-prometheus-stack`, because the gate is static and does
+not boot a node or form a quorum. `platform/kyverno/tests/**` added to `ignorePaths`.
+
+### Sensitive paths — this PR needs a non-author approving review
+
+It touches **`clusters/**`**, **`conformance/**`** and **`.gitea/**`**, all in
+`infra/gitea/approval-policy.json`. It will **not** merge on a green gate alone. Per CLAUDE.md §1 I
+have **not** posted `/approve-override` and must not — that is the self-approval the stage exists to
+prevent.
+
+**The `conformance/` edit, justified.** `+34/-0` in
+`alert-metrics-have-scrape-targets/check.sh`: one new `SCRAPE_OWNERS` entry claiming `kured_`. It
+**weakens nothing** — no existing owner's evaluation changes — and it *tightens* the check for my
+own component, adding two `require` patterns (`service.create: true`, `metrics.create: true`; the
+chart defaults **both** to false) plus a `companion` requirement that `networkpolicy.yaml` stay in
+the kustomization. It had to be touched: assertion (3) fails any alert over an unclaimed metric, and
+the escape hatch `KNOWN_UNWIRED_METRICS` is **shrink-only**, so declaring `kured_*` as debt was not
+permitted and would have been wrong anyway. The alternative was shipping reboot automation with no
+alerting, which for a component whose every failure mode is silent is the worse choice. The entry
+carries a dated comment saying all of this.
+
+## What I verified
+
+Measured against the live cluster and nodes, read-only, on 2026-10-17.
+
+- **Flux image-automation controllers ARE installed and idle.** `-n flux-system get deploy` shows
+  `image-automation-controller` and `image-reflector-controller` both `1/1`, age **43d**; all three
+  `image.toolkit.fluxcd.io` CRDs exist. And **zero** `ImageRepository`/`ImagePolicy`/
+  `ImageUpdateAutomation` objects — "No resources found" for each. The controllers have run for 43
+  days with nothing to do. `platform/kustomization.yaml:52` has the component commented out.
+- **Renovate is aspirational, confirmed three ways**: no workflow in `.gitea/` or `ci/` (grep),
+  no Renovate CronJob/Deployment/Job in any namespace, and `renovate.json` itself disabled the
+  `docker`+`kubernetes` managers — so every plain `image:` tag was maintained by **nothing**, while
+  **seven files** claim "Renovate maintains this pin".
+- **`flux-system` GitRepository has no `secretRef`** (empty), URL is the in-cluster Gitea, and its
+  artifact revision was `main@sha1:1a0b208…`, matching my base.
+- **Nodes**: 3× Debian 13 trixie, all `control-plane,etcd`, k3s `v1.36.4+k3s1`, kernel
+  `7.1.8+deb13-amd64`. `apt-mark showhold` **empty**; `/etc/apt/preferences.d/` **empty**;
+  `/var/run/reboot-required` absent; `unattended-upgrades`/`update-notifier-common`/`needrestart`
+  all **absent**; `linux-image-6.12.107` and `6.12.94` installed alongside the backports kernel;
+  no `k3s`/`containerd` apt package; `apt-cache policy` origins and pins as quoted above.
+- **No node taints** on any of the three (so no tolerations needed), and `system-node-critical`
+  exists.
+- **Longhorn**: 1.12.1; `node-drain-policy: block-if-contains-last-replica`;
+  `replica-soft-anti-affinity: false`; `disable-scheduling-on-cordoned-node: true`;
+  `replica-replenishment-wait-interval: 600`; `concurrent-replica-rebuild-per-node-limit: 1`;
+  25 volumes all 3-replica/healthy; 25 replicas per node.
+- **6 PDBs with 0 allowed disruptions**: 3× `longhorn-system/instance-manager-*` (structurally 0 —
+  single-node selector, `minAvailable: 1`) and 3× CNPG `*-primary`. All three Postgres primaries
+  were on `k3s-prod-1`.
+- **kured drain flags do not exist.** Read `cmd/kured/main.go:438-449`: `Force`,
+  `DeleteEmptyDirData` and `IgnoreAllDaemonSets` are **hardcoded true**. Passing
+  `--ignore-daemonsets`/`--delete-emptydir-data`/`--force` as extraArgs would CrashLoop the
+  DaemonSet. All three are genuinely needed (12 DaemonSets, 135 emptyDir pods measured).
+- **kured exposes exactly one metric** — the only `MustRegister` in the project.
+- **Prometheus selectors are all `{}`** (`serviceMonitorSelector`, `serviceMonitorNamespaceSelector`,
+  `ruleSelector`, `ruleNamespaceSelector`), so a `PrometheusRule`/`ServiceMonitor` in a new namespace
+  is picked up with no label plumbing.
+- **`count(kube_node_info) == 3`** and `count(longhorn_volume_robustness) == 100`, queried through a
+  read-only port-forward. The 5 chronically-firing alerts were read from `/api/v1/alerts`.
+- **`LonghornVolumeDegraded`** is `longhorn_volume_robustness{state="degraded"} == 1`, `for: 10m` —
+  shorter than Longhorn's own 600s replenishment wait, so degradation reliably outlasts the latch.
+- **kured chart 6.1.0 → appVersion 1.23.0** is the newest of 80 entries in the chart index.
+- **Kyverno/PSA will not block it**: every policy is scoped
+  `app-namespace: "true"`; the namespace sets it explicitly `"false"`, and
+  `restrict-host-namespaces.yaml`'s own header states that carve-out. No cluster-wide PSA baseline.
+- **`renovate.json` parses** as valid JSON after my edits (`json.load`).
+- **kured is not installed** and the string `kured` appears nowhere in the repo — greenfield.
+
+### CORRECTION 2026-10-18 — the first real run FAILED, and the sentinel design was wrong
+
+`ansible-playbook playbooks/unattended-upgrades.yml -e k3s_hosts=k3s_prod` failed on all three
+nodes at the package task:
+
+```
+fatal: [k3s-prod-1]: FAILED! => "No package matching 'update-notifier-common' is available"
+```
+
+**`update-notifier-common` is an UBUNTU package and does not exist in Debian.** Measured on trixie:
+`apt-cache policy update-notifier-common` → `Candidate: (none)`. The original entry asserted this
+package "owns the hook that creates /var/run/reboot-required" — true on Ubuntu, false here, and the
+role could never have worked. This was the exact risk the first version of this entry flagged ("the
+`find` patterns are a guess at which package ships what") landing in a worse place than predicted:
+not a wrong glob, a nonexistent package.
+
+Three further measurements settled what to do instead:
+
+* `/usr/share/update-notifier/notify-reboot-required` is **absent**.
+* **No `linux-image-*` postinst references `notify-reboot-required`** — invoking it is an Ubuntu
+  patch, so merely placing a script at that path would never be called.
+* No hook under `/etc/apt/apt.conf.d` or `/etc/kernel/` mentions `reboot-required`.
+
+So Debian trixie has **no stock mechanism at all**, and the role now ships the writer itself:
+`/usr/local/sbin/k3s-notify-reboot-required` (compares the running kernel against the newest
+installed via `linux-version`, from `linux-base`, which every `linux-image` depends on — measured
+`linux-base 4.12.1` present) plus a `DPkg::Post-Invoke` hook at
+`/etc/apt/apt.conf.d/99-k3s-reboot-required`. `/run` is a tmpfs (measured), so the sentinel
+self-clears on boot and needs no cleanup task.
+
+The verify assert changed shape too: it was a `find` glob over `/etc/kernel/postinst.d` and
+`/etc/apt/apt.conf.d` for `*update-notifier*` / `*unattended-upgrades*`. That glob was **weaker
+than it looked** — it would have matched Debian's own `unattended-upgrades` apt conffile and
+reported "sentinel writer present" while no writer existed. It now stats the two exact paths this
+role owns and requires the script to be executable.
+
+**Newly verified (this correction):** the sentinel script was executed on prod-1 as the
+unprivileged `k3s` user under `sh -x`. It resolved `running=7.1.8+deb13-amd64`,
+`newest=7.1.8+deb13-amd64`, took the equal branch, exited 0 and wrote nothing — correct, since the
+running kernel *is* the newest. `sh -n` and `bash -n` clean.
+
+**Still NOT verified after this correction:** the *writing* half. Because running == newest on all
+three nodes, the `touch` path has never executed, so I have not seen the sentinel actually created,
+nor seen kured react to it. The honest test is the operator's next kernel upgrade. Also unverified:
+whether a `DPkg::Post-Invoke` hook fires in the `unattended-upgrades` code path specifically (it
+runs dpkg, so it should, but that is inference) — and whether `linux-version sort --reverse` orders
+a mixed backports/stable kernel set the way I assume when a 6.12 and a 7.1 are both installed. It
+produced the right answer with the current set; a future set could differ.
+
+## What I did NOT verify
+
+The honest list. Most of this needs an actual upgrade cycle, which I did not and could not run.
+
+- **I never ran an OS upgrade, never rebooted a node, and never watched kured take a lock.** No part
+  of the runtime behaviour of either half has been observed. `sudo` needs a password I do not have,
+  so I could not run `apt`, `unattended-upgrade --dry-run`, or install anything. **Everything about
+  (a) working is inference from configuration and measured cluster state, not observation.**
+- **The Ansible role has never been executed.** Not syntax-checked by `ansible-playbook --syntax-check`
+  either (no inventory access from the gate). Its asserts, its `apt-config dump` parsing, and the
+  sentinel-hook discovery are all unproven. The `find` patterns for the sentinel hook are a *guess at
+  which package ships what* — I could not read `update-notifier-common`'s file list (packages.debian.org
+  returned no such package for that suite/arch), which is exactly why the role discovers rather than
+  hard-codes. **If no hook matches, the role fails loudly — but it may fail on a correct node.**
+- **Whether CNPG actually switches the primary in response to a drain-induced eviction.** All three
+  `*-primary` PDBs are structurally 0-disruption and `.spec.nodeMaintenanceWindow` is unset on all
+  three clusters. If the operator does not switch over, draining `k3s-prod-1` blocks on three PDBs at
+  once and hits `drainTimeout`. **This is the most likely way the first real reboot fails.** It
+  should be tested with one manual, watched `kubectl drain` before trusting the timer.
+- **Whether `drainTimeout: 15m` is long enough.** Unmeasured. The `unifi` VMI has
+  `evictionStrategy: LiveMigrate` with `migrationMethod: BlockMigration`, i.e. a full root-disk copy
+  over the network on every `k3s-prod-1` drain. I do not know how long that takes.
+- **Whether the alert gate actually blocks as intended.** I never saw kured query Prometheus. The
+  assumption that it treats a query error as "do not proceed" is from its documented behaviour, not
+  observed. The NetworkPolicy allowing kured→Prometheus:9090 and Prometheus→kured:8080 is
+  **unproven**; if either is wrong the symptom is silence, which is why `KuredNotReporting` exists —
+  but that alert is itself unproven against a live scrape.
+- **The alert expressions have not run against real `kured_*` data**, because the metric does not
+  exist in this cluster yet. They pass promtool unit tests against synthetic series whose shape I
+  read from kured's source. `absent()` and the `count() < count()` comparison are the two most
+  likely to behave differently against real label sets.
+- **`enabledManagers: ["regex"]` may be the wrong name.** Modern Renovate expects `custom.regex` for
+  `customManagers`. I left the pre-existing value alone rather than guess, but **the inline
+  `# renovate:` markers may simply not run.** Unverifiable without executing Renovate. Two markers
+  are additionally known-broken independent of this: `external-snapshotter`'s sits two comment lines
+  above its `newTag`, and Keycloak's `kustomization.yaml` has no marker at all despite a comment
+  claiming Renovate maintains it.
+- **Renovate has still never run.** I added the runner; `RENOVATE_TOKEN` does not exist yet, so the
+  first scheduled run **will fail** (by design). Nothing about the config is proven correct until a
+  `dry_run: true` dispatch is read by a human. Expect it to want to open ~30 PRs.
+- **No alert for "automatic patching silently stopped".** u-u's success lives in
+  `/var/log/unattended-upgrades/` on each node and nothing reads it. Likewise **no alert for the
+  Renovate schedule ceasing to fire**. Both are real gaps I did not close.
+- **`coredns` is 1 replica and I did not fix it.** It is k3s-managed, out of this PR's scope, and it
+  means every `k3s-prod-1` reboot causes a cluster-wide in-cluster DNS gap. Recommended follow-up.
+- **kured cannot honour "NEVER node 2 first".** It has no ordering control. The mitigation — run the
+  kernel play with `--limit` one host at a time — is a *procedure*, documented in the runbook, not an
+  enforced property. If the operator flags all three nodes at once, kured picks the order.
+- **kured cannot press a power button.** The 2026-09-03 warm-reboot hang is unmitigated. Bounded, not
+  prevented: quorum holds at 2/3, the lock is held forever so nothing cascades, and
+  `KuredRebootPending` eventually fires.
+- **Upstream version claims are a snapshot, partly from search results**, not all from authoritative
+  release APIs. I could not determine latest for versitygw, crafty, nginx-unprivileged, act_runner,
+  `virt-template-*`, the linuxserver wireguard date-tag, or the exact latest PostgreSQL 16 patch
+  (sources conflicted 16.10 vs 16.15). **The gitea `1.24.3 → v28.1.0` gap is almost certainly an
+  upstream versioning-scheme change, not 26 majors — do not act on that framing.**
+- **I did not verify Renovate's `kustomize` manager still processes `images:`/`newTag`** now that
+  `docker` is enabled, which decides whether `external-snapshotter` and `intel-gpu-plugin` are
+  managed at all.
+- **I did not raise any HelmRelease version ceiling**, so the 9 minor-locking ranges still cap
+  Renovate at patches. Bumping `kube-prometheus-stack` in particular will fail the gate until
+  `ci/versions.env`'s exact promtool/Prometheus `3.14.0` lockstep moves with it.
+- **L2/L3 conformance did not run** (`vars.CLUSTER_AVAILABLE` unset), so nothing here is proven
+  against a cluster by CI either.

decisions/2026-10-17-graceful-cluster-shutdown.md

@@ -0,0 +1,106 @@
+---
+date: 2026-10-17
+branch: feat/graceful-cluster-shutdown
+title: "Graceful node shutdown was never configured, and scaling a StatefulSet deleted its PVC"
+---
+
+## What changed
+
+All three nodes had to be powered off to move them onto a UPS. There was **no procedure for it**:
+`infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev cluster, so
+the whole sequence was improvised live. Two defects came out of doing it.
+
+**1. Kubelet graceful node shutdown was unconfigured on every prod node.** No `shutdownGracePeriod`
+anywhere in `infra/ansible/`, so `shutdown -h` stopped k3s, containerd went with it, and every pod
+was SIGKILLed where it stood — 9 CloudNativePG instances mid-checkpoint, the UniFi VM, and 25
+attached Longhorn volumes. `roles/k3s_server` now writes `kubelet-arg` into
+`/etc/rancher/k3s/config.yaml` with `shutdown-grace-period=90s` and
+`shutdown-grace-period-critical-pods=20s`.
+
+The half that is easy to miss and silently defeats the other half: **graceful node shutdown is
+enforced by logind, not by the kubelet.** The kubelet holds an inhibitor *delay* lock and systemd
+waits at most `InhibitDelayMaxSec` — **default 5 seconds** — before proceeding regardless. A grace
+period above that cap is truncated with no error, so the feature looks configured and still
+hard-kills pods. The role now ships a `logind.conf.d` drop-in setting it to 120, and **asserts**
+`k3s_logind_inhibit_delay_max_sec >= k3s_kubelet_shutdown_grace_period` rather than documenting the
+relationship, because the failure is invisible.
+
+**2. Scaling a StatefulSet to 0 DELETED its PVC and Longhorn volume.** `monitoring/loki` carries
+`persistentVolumeClaimRetentionPolicy: {whenDeleted: Delete, whenScaled: Delete}`, so the quiesce's
+`scale --replicas=0` destroyed `storage-loki-0` outright. Loki stores with `type: filesystem`, so
+that PVC was the only copy of its log data. This is the defect in this change set that actually
+destroyed something, and it was mine.
+
+New `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`) encodes the sequence that
+worked, and **refuses to scale any StatefulSet whose `whenScaled` is `Delete`** — it leaves it
+running, accepts that its volume hard-detaches at poweroff, and prints which volumes to expect
+`degraded`. A rebuild is a vastly better outcome than a deletion. Workload discovery is computed
+from the live cluster (every Deployment/StatefulSet whose pods mount a PVC, Longhorn or TrueNAS NFS)
+rather than hardcoded, so a newly onboarded service is covered without anyone remembering to add it.
+
+`runbooks/cluster-maintenance-shutdown.md` (new, registered in `runbooks/kustomization.yaml`) is the
+operator page: shutdown order, the TrueNAS-after-nodes constraint, power-on order, the
+un-hibernate-Postgres step Flux will not do for you, and a "what looks broken but is fine" section
+separating the `4/19`-Kustomizations-Ready cascade, the degraded-and-rebuilding volume and the
+self-recovering `keycloak-operator` from the one thing that genuinely needs a human (a stuck
+`oauth2-proxy` replica).
+
+## What I verified
+
+* **Graceful shutdown was genuinely absent**, not merely undocumented: `grep -rn
+  'shutdownGracePeriod|graceful-node-shutdown'` over `infra/ansible/` returns nothing, and no
+  `logind` or `InhibitDelay` setting exists anywhere in the tree.
+* **The whole shutdown/restart cycle was executed end to end on prod 2026-10-17** and the cluster
+  came back: 3/3 nodes Ready with etcd `healthz` ok, 19/19 Flux Kustomizations Ready,
+  `platform-config`'s `lastAppliedRevision` equal to the GitRepository artifact revision, all 3
+  CNPG clusters `healthy` with primaries elected, the UniFi VM `Running`, PiHole resolving
+  (`home.keithjackson.dev` → `192.168.2.20` via `192.168.2.23`), and the Gateway serving
+  `200/200/401` for apex/gitea/plex.
+* **The PVC-deletion mechanism is confirmed, not inferred:** `kubectl get sts loki -o
+  jsonpath={.spec.persistentVolumeClaimRetentionPolicy}` returns
+  `{"whenDeleted":"Delete","whenScaled":"Delete"}`, the PVC was `NotFound` after the cycle, and the
+  Longhorn volume count had gone 25 → 24.
+* **The damage was repaired and verified:** `scale sts loki --replicas=1` (restoring the replica
+  count the chart declares) recreated `storage-loki-0` as a fresh 50Gi Bound PVC, `loki-0` reached
+  `2/2 Running`, and the volume count returned to 25.
+* **The guard works against the live cluster**, tested both ways: `sts_deletes_pvc_on_scale
+  monitoring loki` returns true (would be skipped) and `pihole pihole` returns false (safe to
+  scale).
+* **The script's read-only paths run clean on prod:** `status` reports correctly, and the discovery
+  function returns the 13 workloads currently holding a PVC.
+* `bash -n` clean; the two edited Ansible YAML files parse.
+
+## What I did NOT verify
+
+* **The kubelet change has NEVER BEEN APPLIED OR TESTED.** This is the significant gap. I did not
+  run the role, did not restart k3s, and have **not** observed a node actually drain on
+  `shutdown -h`. Everything about it is reasoning from kubelet/systemd semantics plus the measured
+  absence of the setting. Applying it needs a **rolling** k3s restart (`serial: 1` — kube-vip keeps
+  the API up, etcd holds at 2/3), which is an operator action (CLAUDE.md §5). **Verify by draining
+  one node for real:** apply, then `shutdown -h now` on a single node and watch whether pods
+  terminate gracefully instead of vanishing.
+* **Whether 90s/20s are the right numbers.** Chosen to comfortably cover a Postgres fast shutdown
+  and a Longhorn flush; not measured against either. The budget is spent on every reboot, so if a
+  node takes noticeably longer to go down, this is why.
+* **Whether k3s accepts `kubelet-arg` in `config.yaml` as written.** The file already carries
+  `kube-apiserver-arg` as a list and `kubelet-arg` is the documented sibling, but I did not render
+  or apply it, so a typo would surface only at k3s restart — and a bad `config.yaml` is exactly the
+  kind of thing that makes k3s fail to come up. **Review this one before applying.**
+* **The `assert` task's `regex_replace` on the grace period** assumes a plain `90s`-style value; a
+  value expressed as `1m30s` would parse to the nonsense integer `130`. Fine for the committed
+  defaults, wrong if someone writes a compound duration.
+* **`resume`'s Flux-reconcile wait has never run**, and neither has `quiesce` as a script — the
+  2026-10-17 cycle was done with the equivalent commands by hand, so the script encodes a sequence
+  that worked but has not itself been executed end to end. Only `status` and the discovery and guard
+  functions have run.
+* **Loki's lost log window is gone for good.** Up to 72h of logs (its `retention_period`), destroyed
+  by my scale-down. Nothing recovers it; Velero's schedules cover the namespace but the PVC was
+  deleted, not corrupted, and I did not attempt a restore because the data self-expires anyway.
+* **No other `whenScaled: Delete` StatefulSet was audited.** The guard now catches them generically,
+  but I did not enumerate which other workloads carry the policy, so I cannot say loki was the only
+  one at risk. The runbook gives the one-liner to list them.
+* **TrueNAS and the router are entirely outside this change.** TrueNAS is not in the Ansible
+  inventory and refuses SSH, so its shutdown stays a web-UI step with no automation and no check.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1
+  offline subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

decisions/2026-10-17-keycloak-passkeys.md

@@ -0,0 +1,311 @@
+---
+date: 2026-10-17
+branch: feat/keycloak-passkeys
+title: Passkeys declared for the homelab realm — and the realm import will not apply them
+---
+
+## What changed
+
+`platform/keycloak/keycloak-realm-import.yaml` now declares passkey (WebAuthn Passwordless)
+login for the `homelab` realm, in two parts:
+
+1. **The WebAuthn Passwordless policy.** Ten `webAuthnPolicyPasswordless*` fields on the realm:
+   RP ID `${BASE_DOMAIN}`, RP entity name `Homelab SSO`, signature algorithms `[ES256, RS256]`,
+   `requireResidentKey: "Yes"`, `userVerificationRequirement: required`,
+   `authenticatorAttachment: "not specified"`, `attestationConveyancePreference: none`,
+   `acceptableAaguids: []`, `createTimeout: 300`, `avoidSameAuthenticatorRegister: true`.
+   Deliberately the **passwordless** family only; the two-factor `webAuthnPolicy*` family is left
+   at Keycloak's defaults because nothing in this realm uses `webauthn-authenticator`.
+
+2. **The browser flow.** `browser-mfa` keeps its alias and stays bound as `browserFlow`; its
+   interior is restructured so a passkey is an **ALTERNATIVE** beside the password form:
+
+   ```
+   browser-mfa                            (top level, unchanged)
+     browser-mfa-forms                    ALTERNATIVE
+       auth-username-form                   REQUIRED      <- was auth-username-password-form
+       browser-mfa-passkey-or-password      REQUIRED      <- NEW sub-flow
+         webauthn-authenticator-passwordless  ALTERNATIVE <- NEW: the passkey
+         browser-mfa-password-and-otp         ALTERNATIVE <- NEW sub-flow
+           auth-password-form                   REQUIRED  <- was half of the combined form
+           browser-mfa-conditional-otp          CONDITIONAL (unchanged, relocated intact)
+   ```
+
+The design decisions, with their reasoning, are written into the manifest comments in this repo's
+usual density. The ones that matter:
+
+* **Additive, not replacing. Enrolment optional, not forced.** This IdP gates Grafana, Gitea,
+  Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the \*arr stack *and* `kubectl` via the
+  `kubernetes` OIDC client. A mandatory passkey flow that misfires is a total lock-out from the
+  platform and the cluster API at once, leaving only the Keycloak bootstrap admin and the
+  break-glass cert kubeconfig. The gain from forcing it (one human stops typing a password) does
+  not price that risk.
+* **No `requiredActions` block was added, on purpose.** Declaring one would have been the obvious
+  way to add `webauthn-register-passwordless`, and it would have been actively dangerous — see
+  "What I verified".
+* **RP ID is `${BASE_DOMAIN}`, not `keycloak.${BASE_DOMAIN}`.** A passkey's RP ID is fixed at
+  enrolment and can never be re-keyed, so it must not be bound to a name that can move. This realm
+  already declares an `auth.${BASE_DOMAIN}` redirect URI, and this repo already owns a reconciler
+  built because a hostname move silently failed to reach the realm. Residual risk, unavoidable:
+  changing `BASE_DOMAIN` itself invalidates every enrolled passkey.
+* **A passkey login is not also asked for TOTP.** `browser-mfa-conditional-otp` hangs off the
+  password branch only. The justification is `userVerificationRequirement: required` — the
+  ceremony already proves possession plus a verified human, phishing-resistantly and origin-bound.
+  Stacking a replayable shared secret on top of that buys friction. The manifest names the exact
+  one-line edit to reverse this decision.
+* **Login becomes two screens** (username, then password) for every user, passkey or not, because
+  the server must know who is logging in before it can decide whether a passkey exists. Accepted
+  cost; it is the enabling change.
+* **Usernameless login deliberately not used**, although Keycloak 26.0.6 supports it — see "What I
+  verified" for the measured reason it would have made every login open on a passkey prompt.
+
+`platform/keycloak/README.md` gains a `## Passkeys` section: the decision table, the fact that
+this does not reach the live realm, the two operator paths that would apply it, and the four-step
+browser verification nobody has run.
+
+No secret, key, credential or AAGUID was minted or committed. No file under `clusters/**`,
+`conformance/**`, `ci/**`, `platform/kyverno/**`, `**/rbac.yaml` or `**/*.sops.yaml` was touched,
+so this should not trip `ci/approval-policy.sh`'s sensitive-path review requirement.
+
+## What I verified
+
+**The live realm import has already completed and will never re-run.** Measured on prod,
+2026-10-17:
+
+```
+$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
+{"conditions":[{"message":"","status":"True","type":"Done"},
+               {"message":"","status":"False","type":"Started"},
+               {"message":"","status":"False","type":"HasErrors"}]}
+```
+
+`Done=True` / `Started=False`, exactly the one-shot state `platform/keycloak/README.md` already
+documents for client secrets and client URLs. **So the change in this PR has zero effect on how
+anyone logs in today**, and the client-secret/client-URL reconciler does not and cannot close the
+gap: its hard endpoint allow-list makes `/authentication` unreachable by design, and realm-level
+policy fields are not client fields.
+
+**The CRD accepts every field I used.** Measured against the live CRD (not the upstream docs):
+`kubectl --context k3s-prod get crd keycloakrealmimports.k8s.keycloak.org` lists all twelve
+`webAuthnPolicyPasswordless*` properties with the types I wrote (`RequireResidentKey`,
+`RpEntityName`, `RpId`, `UserVerificationRequirement`, `AttestationConveyancePreference`,
+`AuthenticatorAttachment` as `string`; `SignatureAlgorithms`, `AcceptableAaguids`, `ExtraOrigins`
+as `array/string`; `CreateTimeout` as `integer`; `AvoidSameAuthenticatorRegister` as `boolean`),
+plus `authenticationFlows`, `authenticatorConfig` and `requiredActions`. The `spec.realm` schema
+has **no** `x-kubernetes-preserve-unknown-fields`, so a misspelled field would have been pruned
+silently rather than rejected — which is why this was checked rather than assumed.
+
+**Running Keycloak is 26.0.6.** `quay.io/keycloak/keycloak:26.0.6` on `keycloak-0`/`keycloak-1`,
+operator `quay.io/keycloak/keycloak-operator:26.0.6`. Every source claim below is read from the
+`26.0.6` tag, not from memory or from docs.
+
+**Every provider ID is quoted from upstream source, not guessed.**
+
+| String I used | Where it is defined at tag 26.0.6 |
+|---|---|
+| `webauthn-authenticator-passwordless` | `WebAuthnPasswordlessAuthenticatorFactory.PROVIDER_ID`, display "WebAuthn Passwordless Authenticator" |
+| `auth-username-form` | `UsernameFormFactory.PROVIDER_ID`, display "Username Form" |
+| `auth-password-form` | `PasswordFormFactory.PROVIDER_ID`, display "Password Form" |
+| `webauthn-register-passwordless` | `WebAuthnPasswordlessRegisterFactory.PROVIDER_ID` (referenced in comments only) |
+
+**`requiredActions` is REPLACE, not merge — measured, and it is why I did not declare one.**
+`model/storage-private/.../DefaultExportImportManager.java` lines 327-338:
+
+```java
+if (rep.getRequiredActions() != null) {
+    for (RequiredActionProviderRepresentation action : rep.getRequiredActions()) { ... }
+    DefaultRequiredActions.addDeleteAccountAction(newRealm);
+} else {
+    DefaultRequiredActions.addActions(newRealm);
+}
+```
+
+Declaring one required action therefore skips the entire default set. `RealmManager` is no
+backstop: `setupRequiredActions` is `if (realm.getRequiredActionProvidersStream().count() == 0)`,
+and the count is already non-zero by then (`RepresentationToModel.importRealm` runs at
+`RealmManager.java:585`, the backstop at `:612`). A realm built that way would have had the passkey
+action plus `delete_account` and nothing else — **no `CONFIGURE_TOTP`**, which would leave
+`auth-otp-form` demanding a TOTP that no platform-admin could ever enrol: a hard lock-out
+delivered by a change whose purpose was to *add* a login method.
+
+**And declaring it would have bought nothing.** `DefaultRequiredActions.addWebAuthnPasswordlessRegisterAction`
+registers `webauthn-register-passwordless` with `setEnabled(true)`, `setDefaultAction(false)`,
+priority 80, gated on `Profile.Feature.WEB_AUTHN` — which is `Type.DEFAULT` in
+`common/.../Profile.java:74`, i.e. on unless explicitly disabled. `enabled=true` /
+`defaultAction=false` *is* the additive-and-optional posture this change wants.
+
+**`authenticationFlows` has the opposite semantics, which is why declaring flows is safe.**
+`importAuthenticationFlows` adds the declared flows, then backfills each unbound binding
+(browser, registration, direct grant, reset credentials, client auth, first broker login, docker
+auth) from the built-in defaults individually (`DefaultExportImportManager.java` ~1320-1410).
+
+**The graceful degradation to password is measured from the flow engine, not assumed.** For a user
+with no passkey: `WebAuthnPasswordlessAuthenticator.requiresUser()` returns `false`, so
+`DefaultAuthenticationFlow.processSingleFlowExecutionModel` takes the `else` branch (`daf:456-462`),
+whose four conditions all hold — a user is set (from `auth-username-form`), `configuredFor` is
+false, `WebAuthnPasswordlessAuthenticatorFactory.isUserSetupAllowed()` is `false`, and
+`WebAuthnAuthenticator implements Authenticator, CredentialValidator<...>` — so it raises
+`CREDENTIAL_SETUP_REQUIRED`. The ALTERNATIVE loop (`daf:289-302`) catches exactly that, marks the
+execution `ATTEMPTED` and proceeds to the next alternative, with the upstream comment *"the user,
+since he has alternatives, should be able to go to another alternative and continue"*. Hence: no
+passkey, no passkey screen, straight to the password form.
+
+**Why usernameless was rejected, measured.** The `configuredFor` guard sits inside the
+`requiresUser()` branch. With `requiresUser()` false and no user yet in context, that guard is
+skipped and the engine just calls `authenticate(context)` — so a passkey execution placed before
+the username form is offered to *everyone*, including users who will never own one.
+
+**The policy sentinels are read from the shipped code, and one of my initial assumptions was
+wrong.** `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED = "not specified"` is the
+realm-representation sentinel — but `WebAuthnConstants.OPTION_NOT_SPECIFIED = ""` (empty string) is
+the WebAuthn-API-level one. I checked both because they disagree. The login theme's
+`webauthnRegister.js` at 26.0.6 settles the behaviour:
+
+```js
+if (input.requireResidentKey !== 'not specified') {
+    if (input.requireResidentKey === 'Yes') { authenticatorSelection.requireResidentKey = true; }
+    else { authenticatorSelection.requireResidentKey = false; }
+}
+...
+if (input.createTimeout !== 0) { publicKey.timeout = input.createTimeout * 1000; }
+rp: {id: input.rpId, name: input.rpEntityName}
+```
+
+So `"Yes"` is a literal string compare (any other non-sentinel value means *false*), `"not
+specified"` is compared literally for four fields, and `createTimeout` is **seconds** with `0`
+meaning "send no timeout". It also shows 26.0.6 emits only the legacy `requireResidentKey` member,
+never `residentKey: "required"` — the client maps that per WebAuthn L2, but the policy cannot
+express `residentKey: "preferred"` at all.
+
+`[ES256, RS256]` is exactly `WebAuthnPolicy.DEFAULT_POLICY` /
+`Constants.DEFAULT_WEBAUTHN_POLICY_SIGNATURE_ALGORITHMS` — declared explicitly, not changed.
+`getWebAuthnPolicyPasswordless` was read field by field to confirm `[]` for `acceptableAaguids`
+means "no restriction" and that `RpEntityName` must be non-empty or the **update** path
+(`DefaultExportImportManager.java:842`) drops the whole policy silently.
+
+**The manifest parses and renders.** `yaml.safe_load` round-trips the file and the flow graph reads
+back exactly as designed (five flows, the executions and priorities above, `browserFlow:
+browser-mfa`, `requiredActions` absent). `python3 ci/render-keycloak-client-urls.py --check` →
+`client-urls.yaml is in sync`, rc 0 (no client URL changed, so no regeneration was needed).
+`yamllint -c ci/yamllint.yaml platform/keycloak/keycloak-realm-import.yaml` → rc 0; no line exceeds
+the 160-column ceiling.
+
+**`browser-mfa` keeps its alias deliberately.** `conformance/tests/l2-behavioral/keycloak-config-as-code`
+asserts `kubectl get keycloakrealmimport homelab -o jsonpath={.spec.realm.browserFlow} | grep -q
+'browser-mfa'`. Renaming the top-level flow would have failed that test.
+
+**Envsubst safety.** `platform-config`'s Flux Kustomization declares
+`postBuild.substituteFrom: cluster-vars` (`clusters/base/platform-config.yaml:26-29`), so this file
+is substituted. The only brace token I added is `${BASE_DOMAIN}`, which is declared at
+`clusters/base/cluster-vars.yaml:42` (`home.keithjackson.dev`). Every comment I wrote avoids braces
+entirely — the RP-ID discussion says `keycloak.<BASE_DOMAIN>` with angle brackets precisely so an
+unparseable token cannot fail the whole Kustomization.
+
+**Gate result: rc 3, zero failures, `gitea-settings` the only stage the gate counts as not-run** —
+the documented honest best local outcome. Run in the pinned toolbox from this worktree with all
+three §3 workarounds (`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main
+checkout's `.git` and the sibling spec repo with `SPEC_REPO=/home-network-k3s`):
+
+```
+[ ok ] stage spec-immutability: pass      <- NOT skipped; both shim mounts worked, R-TEST-06 covered
+[ ok ] stage decisions-log: pass          <- archive frozen, 48 entries well-formed
+[ ok ] stage lint: pass
+[ ok ] stage l0: pass                     <- incl. flux-postbuild-substitution + keycloak-client-secret-reconciler
+[ ok ] stage render-golden: pass
+[ ok ] stage l1: pass
+[ ok ] stage approval-policy: pass        <- no sensitive path touched, so no non-author review needed
+[skip] stage gitea-settings: SKIPPED      <- needs SETTINGS_API_TOKEN, CI-only; expected
+[skip] stage conformance: SKIPPED         <- cluster-gated L2/L3; expected
+MERGE GATE: UNDETERMINED — Stages that did not run: gitea-settings
+```
+
+No lint finding and no check failure names either file I changed. Touched requirements resolved to
+`R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06`; coverage 51/51, 0 gaps.
+
+**A note on how that number was obtained, because the first attempt was worthless.** My first gate
+run returned the same rc 3 and the same seven passes — against a tree that did not contain this
+change at all. The gate's `tracked_files()` reads **git**, and my edits were still uncommitted, so
+every stage gated `origin/main`. The tell was in the log and I nearly missed it: the
+touched-requirement subset reported UNDETERMINED with *"check out the feature branch you mean to
+gate (not the base branch)"*, and `decisions-log` counted **47** entries rather than 48. **A green
+gate run on an uncommitted worktree is a gate run on the base branch.** The numbers above are from
+the second run, after committing and rebasing.
+
+**The base moved under me mid-run.** `origin/main` advanced from `9a41a8e` to `1a0b208` (PR #70,
+oauth2-proxy OIDC startup race) while I was working. This branch is rebased onto `1a0b208`, so the
+gate above ran on the base CI will actually merge into. The rebase was clean — PR #70 touched
+`platform/oauth2-proxy/helmrelease.yaml` and its own decisions entry, disjoint from these files.
+
+## What I did NOT verify
+
+**This change is unapplied. It changes nothing about the live realm, and I did not attempt to make
+it.** `Done=True` / `Started=False` was measured, not inferred, and I ran no mutating `kubectl`,
+no `flux reconcile`, no admin-API call and no `kcadm`. Whether the operator's eventual
+admin-console reconstruction of this flow matches what is declared here is, by construction,
+unverifiable from git.
+
+**I never enrolled a passkey and never pointed a browser at any of this.** Nothing here has been
+exercised by a real WebAuthn ceremony on any device. Specifically unverified:
+
+* **That the RP ID works.** `home.keithjackson.dev` is a registrable-domain suffix of
+  `keycloak.home.keithjackson.dev`, so it *should* satisfy the browser's check — but I verified
+  that by reading the WebAuthn rule, not by watching a browser accept it. A wrong RP ID fails
+  **silently, in the browser, before Keycloak is involved**, with no server-side log line. If
+  enrolment produces no OS prompt and no error, this is the first suspect.
+* **That `"not specified"` is the right literal for `authenticatorAttachment` in a realm import.**
+  I traced it to `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED` and to the theme's string
+  compare, but I have not seen a realm import accept it. Omitting the field would default to the
+  same value; if an import ever rejects the literal, drop the line.
+* **That `createTimeout: 300` behaves.** Seconds-vs-milliseconds is read from the theme JS. The
+  claim that 300 is long enough for the cross-device QR/hybrid flow, and that Chrome clamps rather
+  than rejects it, is reasoning about browser behaviour, not a measurement.
+* **Whether a passkey user can still reach the password form.** I argued it from
+  `createAuthenticationSelectionList` producing a "Try another way" option, and I noted that for a
+  user holding both credentials the *first* screen is chosen by the user's own credential ordering
+  rather than by my priorities. Neither was observed.
+* **The two-screen login regression for ordinary users.** `auth-username-form` +
+  `auth-password-form` replacing `auth-username-password-form` is a visible change for every user
+  including non-passkey family accounts, and nobody has logged in through it.
+* **That conditional OTP for platform-admins still fires on the password branch.** The sub-flow and
+  its `conditional-otp-platform-admin` config are byte-identical and merely relocated one level
+  deeper, but "identical YAML" is not "still enforced at runtime". This is the highest-value
+  regression check and it is untested. The previous breakage of this exact step
+  (`condUserRole` vs a non-existent property key) 403'd every platform-admin login.
+* **That a platform-admin authenticating by passkey is not challenged for TOTP.** That is the
+  intended behaviour of the shape I wrote; it is unobserved, and it is also a deliberate
+  relaxation of R-IDP-04 as previously implemented, which the operator may want to reverse.
+
+**Whether the realm import reconciles into an existing realm: measured as NO for the completed
+one-shot, but I did not test a re-import.** I did not delete and recreate the
+`KeycloakRealmImport`, so I cannot say what a re-run would do to the existing `homelab` realm —
+whether it would merge the new policy and flows, duplicate flow aliases, or fail. The 2026-09-06
+total SSO outage documented in `platform/keycloak/README.md` is the reason I did not experiment.
+**Treat the re-import path in the README as untested advice, not a runbook.**
+
+**Nothing was proved about the live realm's current state.** I did not read the realm's existing
+required actions, its existing WebAuthn policies, or its live flow executions — doing so needs the
+admin credential and I did not decrypt or use it. So I cannot confirm that the live realm's
+required-action set actually contains `webauthn-register-passwordless` today; I can only show that
+Keycloak 26.0.6's defaults register it and that this repo's import never declared any required
+actions to suppress them. **That inference has not been checked against the realm.**
+
+**L2/L3 conformance did not run** (no cluster from the gate container; `vars.CLUSTER_AVAILABLE` is
+unset in CI anyway), so the existing `keycloak-config-as-code` behavioural test did not re-execute
+against this change. I reasoned about its assertion by reading it.
+
+**No new conformance test was added.** Nothing in `conformance/` asserts the passkey policy or the
+new flow shape, so a future edit can silently remove either and no gate stage will notice. That is
+a real gap this PR leaves open; closing it would touch `conformance/**`, which is a sensitive path
+needing non-author review.
+
+**The quoted gate run predates this file's own gate-result paragraph.** It gated commit `207f5db`;
+the gate numbers and the rebase note above were then written into this entry, so the pushed tree
+differs from the gated one by this handoff's prose only — no manifest, no check, no generated file.
+`decisions-log` is the only stage that reads this file, and it validates structure (frontmatter,
+the three headings, non-stub bodies), which is unchanged. I did not re-run the full gate for a
+prose edit; CI will gate the pushed commit.
+
+**`gitea-settings` did not run locally** — it needs `SETTINGS_API_TOKEN`, which exists only in CI.
+I did not re-measure branch protection, the runner, or `required_approvals` against the Gitea API
+this session; the claims in `CLAUDE.md` §2 are taken on trust here and are explicitly flagged there
+as needing re-measurement.

decisions/2026-10-17-oauth2-proxy-oidc-startup-race.md

@@ -0,0 +1,99 @@
+---
+date: 2026-10-17
+branch: fix/oauth2-proxy-oidc-startup-race
+title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
+---
+
+## What changed
+
+Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
+discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
+serving yet, so the shared Gateway answers 503 and the proxy dies:
+
+```
+[provider.go:55] Performing OIDC Discovery...
+[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
+             failed to discover OIDC configuration: unexpected status "503": no healthy upstream
+```
+
+It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
+replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
+retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
+is already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+
+`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
+polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
+accrue CrashLoopBackOff backoff and retries promptly, which is the point.
+
+**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
+case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
+for a pod wedged in `Init` forever, which is not an improvement.
+
+## Why this shape and not the alternatives
+
+* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
+  example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
+  pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
+  at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
+  "reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
+* **No new base image.** The chart already runs an `alpine` init container for its built-in
+  `initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
+  `3.22` (verified to exist upstream) rather than the chart's own `latest`.
+* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
+  this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
+  the chart already has a hook for.
+* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
+  more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
+  oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
+  the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
+  the operator out of all of them, so the reversible, additive option wins.
+
+The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
+with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
+and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
+silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
+the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.
+
+## What I verified
+
+* **The failure is measured, not theorised.** The log above is from the live pod
+  (`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
+  with `--previous` after the cold boot. The surviving replica's log shows the same startup
+  succeeding once Keycloak was up, which is what identifies this as a race rather than a config
+  fault.
+* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
+  clean and both replicas have been healthy since.
+* **The chart key is real**, at the pinned version range: `helm search repo` resolves
+  `>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
+  `extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
+* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
+  assumed from memory.
+* The edited HelmRelease parses as YAML.
+
+## What I did NOT verify
+
+* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
+  seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
+  replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
+  seven apps, so **do it when you can watch it**, not unattended.
+* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
+  oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
+  from the measured failure, not demonstrated against it.
+* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
+  `ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
+  failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
+  degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
+  anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
+  init container's log on first apply**: it should print "Keycloak discovery reachable" within
+  seconds on a warm cluster, not the give-up message.
+* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
+  re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
+  good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
+  (`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
+  Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
+  this should hold — unverified.
+* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
+  not a measured bound.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
+  subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

decisions/2026-10-18-coredns-ha.md

@@ -0,0 +1,108 @@
+---
+date: 2026-10-18
+branch: fix/coredns-ha
+title: "coredns ran at one replica, so any node drain was a cluster-wide DNS outage"
+---
+
+## What changed
+
+**MEASURED on prod 2026-10-18: `kube-system/coredns` had `replicas: 1`.** In-cluster DNS was a
+single pod on a single node, so draining or rebooting that node was a **cluster-wide in-cluster DNS
+outage** until the pod rescheduled and became ready — every Service lookup, every CNPG replica
+finding its primary, every operator addressing the API by name. It was invisible because `1/1
+Ready` looks perfectly healthy.
+
+It also became urgent rather than latent: the scheduled reboot automation on
+`feat/automated-upgrades` drains and reboots one node at a time, unattended, overnight — turning a
+single point of failure into a recurring one.
+
+New `platform/coredns/hpa.yaml`: a `HorizontalPodAutoscaler` on the coredns Deployment with
+`minReplicas: 2`, `maxReplicas: 3`, registered in the existing `platform/coredns/kustomization.yaml`
+alongside the split-horizon ConfigMap already there.
+
+**Why an HPA and not `replicas: 2` — this is the whole decision.** k3s owns the coredns Deployment
+and we cannot durably edit it: it is applied by k3s's own deploy controller from a bundled manifest
+(`AddOn/coredns`, source `/data/k3s/server/manifests/coredns.yaml`, `managedFields` managers
+`deploy@k3s-prod-1` and `k3s`, plus an `objectset.rio.cattle.io/hash` label). k3s re-stages its
+bundled manifests on startup, so a `replicas: 2` written there — or a `kubectl scale` — is reverted
+on the next k3s restart or upgrade. Two alternatives were weighed and rejected:
+
+* **`coredns.yaml.skip` + our own templated copy** (the pattern `roles/kube_vip` already uses for a
+  file in `server/manifests`). Deterministic and never flaps, but it takes ownership of coredns away
+  from k3s permanently — k3s upgrades would stop updating it and the version would need refreshing
+  by hand forever. Too high a price for a replica count.
+* **A Flux-managed patch of the Deployment.** Rejected outright: Flux and k3s's deploy controller
+  would both own one object and fight continuously — the exact shape of the `default-deny`
+  NetworkPolicy drift loop this cluster already suffered (~2500 events/10min).
+
+An HPA avoids both: a separate object k3s does not own, driving the `/scale` subresource rather than
+the Deployment spec, so there is no shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA
+sees it below `minReplicas` and scales back within one reconcile.
+
+It is a **floor-holder, not an autoscaler**: the CPU metric exists only because an `autoscaling/v2`
+HPA needs one to be valid. `scaleDown.stabilizationWindowSeconds: 600` because DNS is the worst
+thing in the cluster to flap; `scaleUp` is immediate, since that is the path that restores redundancy
+after a k3s restart.
+
+**Immediate action taken outside this PR:** `kubectl -n kube-system scale deploy coredns
+--replicas=2` was run at the operator's explicit request, so the cluster has two replicas **now**
+rather than after merge. That hand scale is what this PR makes durable.
+
+## What I verified
+
+* **The defect:** `kubectl -n kube-system get deploy coredns` → `replicas=1 available=1`.
+* **k3s ownership:** the `AddOn/coredns` object exists with source
+  `/data/k3s/server/manifests/coredns.yaml`; the Deployment has no `ownerReferences` but carries
+  `objectset.rio.cattle.io/hash` and `managedFields` managers `deploy@k3s-prod-1` / `k3s`. So it is
+  k3s-managed, not Flux-managed — independently corroborated by another agent, which confirmed
+  `platform/coredns/` contains **only** the `coredns-custom` ConfigMap and does not touch the
+  Deployment or its replica count.
+* **The hand scale works and holds:** coredns went to `2/2 Running` on two different nodes, and
+  in-cluster DNS kept answering throughout (`nslookup kubernetes.default.svc.cluster.local
+  10.43.0.10` → `10.43.0.1`). Still `spec.replicas=2 ready=2` after ~2 minutes, with the AddOn
+  checksum **unchanged** — so k3s's deploy controller does not revert on a drift timer; it
+  re-applies on manifest change and on restart.
+* **The HPA's preconditions hold:** coredns ships `requests: {cpu: 100m, memory: 70Mi}` (so a
+  Resource/Utilization metric is meaningful), and metrics-server is installed as a k3s addon and
+  verified serving — `kubectl top pod -l k8s-app=kube-dns` returns **5m and 4m** against that 100m
+  request, i.e. ~5% utilisation, confirming the 80% target will never be approached and the HPA will
+  sit at `minReplicas`.
+* **The kustomization edit is additive:** `git diff --stat` is `1 file changed, 4 insertions(+)`,
+  and `- coredns` was already registered in `platform/kustomization.yaml`, so no change was needed
+  there.
+* All three touched/added YAML files parse.
+
+## What I did NOT verify
+
+* **The HPA has never been applied.** I did not reconcile it, so I have **not** observed it create,
+  reach `ScalingActive=True`, resolve its metric against real coredns data, or hold `minReplicas`.
+  Applying is an operator action (CLAUDE.md §5). The failure mode to watch for is
+  `ScalingActive=False` with `FailedGetResourceMetric` — the HPA would then be present and inert,
+  which looks identical to working because the hand scale already left 2 replicas in place. **Check
+  `kubectl -n kube-system describe hpa coredns` after merge**, not just the replica count.
+* **The central claim that k3s reverts `replicas` on restart is REASONED, NOT OBSERVED.** It rests
+  on k3s re-staging bundled manifests at startup plus the measured ownership evidence above. I did
+  **not** restart k3s to watch the revert happen — doing so is a rolling control-plane restart, an
+  operator action. If k3s in fact preserves a modified replica count, this HPA is unnecessary
+  (though harmless).
+* **I did not confirm the HPA wins the race against k3s's deploy controller after a restart.** Both
+  write the scale; the expectation is the HPA re-asserts within a reconcile, leaving a brief window
+  at 1 replica. Neither the window's length nor the outcome has been measured.
+* **I nearly shipped a destructive change.** My first write of
+  `platform/coredns/kustomization.yaml` replaced the file wholesale, listing only `hpa.yaml` — which
+  would have pruned the existing `coredns-custom` ConfigMap and **broken in-cluster split-horizon
+  DNS (R-DNS-01/03)**. It was caught by a guard assertion before commit and restored from `HEAD`, and
+  the diff is now additive. Recorded because the near-miss is the useful part: `platform/coredns/`
+  already existed and I assumed it did not. **No conformance check would obviously have caught the
+  removal** — it is valid YAML and a legitimate-looking prune.
+* **Whether 2 is enough.** Three nodes, and coredns' anti-affinity only *prefers* spreading, so both
+  replicas could in principle land on one node and a single drain would still take DNS down. I did
+  not add a `topologySpreadConstraint` or a PodDisruptionBudget — both arguably belong here and
+  neither is in this change.
+* **No alert fires if coredns drops to 1 replica**, so a future silent regression (k3s upgrade
+  changing the manifest, the HPA going inert) would not be reported. That gap is not closed here.
+* **No conformance test asserts this**, so a future edit could remove the HPA with no gate stage
+  noticing.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
+  subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

decisions/2026-10-18-ha-check-catastrophic-regex.md

@@ -0,0 +1,76 @@
+---
+date: 2026-10-18
+branch: fix/ha-check-catastrophic-regex
+title: "One regex made the merge gate take 28 minutes instead of 2"
+---
+
+## What changed
+
+The merge gate started taking **~28 minutes** instead of its usual **<2 minutes**, on every change
+touching `infra/**`. One regex was responsible for essentially all of it.
+
+`conformance/tests/l0-static/ha-control-plane-as-code/check.sh` contained:
+
+```python
+join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
+                      K3S_TEXT, re.S)
+```
+
+With `re.S`, every `.` also matches newlines, so `.*?` and the nested `(?:.*\n){0,10}?` can each
+consume the entire document. That is **nested unbounded repetition** — ten levels of a quantifier
+that can each swallow everything — and a failed or late match backtracks exponentially.
+
+Replaced `.` with `[^\n]*` and dropped `re.S`, so the `{0,10}` is a genuine ten-line window with
+nothing to backtrack into. Same intent, same verdict, no DOTALL.
+
+**Why it surfaced now, and it was mine.** `K3S_TEXT` is every `*.yml` in `roles/k3s_server`
+concatenated. The regex was latent for as long as it existed; **PR #71 added ~100 lines to that role
+(the kubelet graceful-shutdown config), which grew the input past the point where exponential
+backtracking stopped being survivable.** Nothing about the check changed — my change grew its input.
+
+**Why only `infra/**` changes were slow.** `ci/touched-requirements.sh:268` maps `infra/*` to
+`R-ENV-01`, and the resulting L0 run is `--include-scope whole-repo --requirements R-ENV-01`, which
+pulls in this whole-repo check. Doc-only and `platform/**` changes never selected it, which is why
+#69 (1m55s) and #72 (1m53s) stayed fast while #71 (27m44s), the #73 merge (27m46s) and #75 all ran
+~28 minutes.
+
+## What I verified
+
+* **The hot spot, by measurement rather than reading.** `docker top` on the live CI job container for
+  run #299 showed `python3 -` with **19:54 of CPU time** under
+  `ha-control-plane-as-code/check.sh`, inside
+  `conformance/run.sh --layer l0 --include-scope whole-repo --requirements R-ENV-01`.
+* **Reproduced locally**: the check did not finish within a 100-second budget on an unmodified
+  checkout.
+* **Localised to the exact line** by extracting the inline Python and running it under
+  `faulthandler.dump_traceback_later(25)`, which named `re/__init__.py:177 in search` called from
+  the `join_when` line — not inferred from reading the patterns.
+* **The pattern is unique in the repo**: `grep -rlnE "\{0,[0-9]+\}\?" conformance/tests/l0-static/*/check.sh`
+  returns only this file, so there is no sibling with the same hazard.
+* **The fix is fast AND verdict-preserving**, tested against the real `K3S_TEXT` (22,179 chars, 403
+  lines): the new regex matches (`YES`) in **0.2 ms**. Sanity-checked that the thing it looks for
+  genuinely exists — one `--server https://` and three `k3s_server_mode == 'secondary'` occurrences.
+* **The whole check now runs in 0.199s, rc 0, 115 assertions**, with the same success message as
+  before.
+
+## What I did NOT verify
+
+* **I did not re-time the full gate in CI after the fix.** The expectation is that `infra/**` changes
+  return to the ~2-minute range, but that is arithmetic from the 19m54s figure, not a measured
+  post-fix run. The PR's own CI run is the first real datapoint — read it.
+* **I did not audit the other 23 L0 checks for different performance pathologies.** This one was
+  found because it was actively burning CPU in a live job; a check that is merely slow (not
+  exponential) would not have stood out, and `ci/gate.sh` has **no per-stage timing** at all
+  (`grep -nE "SECONDS|date \+|elapsed"` finds nothing), so there is no data on where the remaining
+  time goes.
+* **Nothing prevents this class of bug returning.** No check asserts gate stage runtime, and no lint
+  forbids nested quantifiers under `re.S`. A timing budget per stage, or a `re` lint, would close it;
+  neither is written here.
+* **I did not verify my claim about *why* only `infra/**` was affected beyond the path mapping and
+  the observed durations.** The correlation is strong (5 runs, two clear clusters) and the live
+  process list confirms the `--requirements R-ENV-01` scope, but I did not read
+  `conformance/run.sh`'s scope-selection logic end to end.
+* The regex rewrite assumes the `when:` clause is indented with spaces or tabs only (`[ \t]*`). A
+  file using some other whitespace before `when:` would now fail to match where the old DOTALL
+  version would have succeeded. Unlikely in YAML, and the current input matches, but it is a
+  narrowing.

decisions/2026-10-18-keycloak-passkey-reconciler.md

@@ -0,0 +1,334 @@
+---
+date: 2026-10-18
+branch: feat/keycloak-passkey-reconciler
+title: "Make the passkey realm config converge via GitOps — policy, enrolment action and the browser flow, the flow blue/green so the bound login path is never mutated"
+---
+
+## What changed
+
+**Why this is a new branch and not an update to PR #72.** The brief for this work said to extend
+PR #72 (`feat/keycloak-passkeys`) in place and not open a new PR. That premise was already stale
+when I started: **PR #72 merged at 2026-10-18T05:44:04Z** (Gitea API: `state: closed`,
+`merged: true`, `merge_commit_sha: 4d489f2`), and `git log origin/main` carries
+`4d489f2 Merge pull request '…' (#72)`. A merged PR cannot be updated, and CLAUDE.md §1 leaves
+exactly one path to `main`. So this is a fresh branch from `origin/main` in its own worktree, and a
+new PR. For the same reason this is a **new** `decisions/` entry rather than an edit to
+`decisions/2026-10-17-keycloak-passkeys.md`, which is now merged history describing a different unit
+of work; it is the entry to read first for the passkey *design* decisions, which are not repeated
+here.
+
+**The problem.** PR #72 declared the WebAuthn passwordless policy and a restructured `browser-mfa`
+flow in `platform/keycloak/keycloak-realm-import.yaml`. That declaration **could not take effect**.
+A `KeycloakRealmImport` is a one-shot; prod's reads `Done=True / Started=False / HasErrors=False`,
+so the operator will never re-run it and every realm-level field or flow declared after the first
+import is inert — in git, applied by Flux, reported Ready, reaching the realm never. Merging it
+changed nothing about how anyone logs in, and nothing anywhere said so. Same failure class as the
+2026-09-06 client-secret incident, third field over.
+
+**What now makes it converge.** A **sibling** reconciler, modelled on
+`platform/keycloak/client-secret-reconciler.yaml`:
+
+- `platform/keycloak/passkey-reconciler.yaml` — ConfigMap (Python, stdlib only, **zero dollar
+  signs**) + CronJob on `*/15 * * * *`, `automountServiceAccountToken: false`, admin credential by
+  `secretKeyRef` only. Three passes, **all of which converge**:
+  1. **WebAuthn passwordless policy.** Strict read-modify-write of `GET`/`PUT /admin/realms/homelab`,
+     changing only the `webAuthnPolicyPasswordless*` keys that differ, readback-verified.
+  2. **`webauthn-register-passwordless` required action.** Strict read-modify-write of one
+     **literally-named** alias; converges `enabled: true` / `defaultAction: false` only.
+  3. **The browser authentication flow — BLUE/GREEN.** Build the declared tree under a new, unused
+     alias; verify it by readback; flip `browserFlow` in **one atomic PUT**.
+- `ci/render-keycloak-passkey-policy.py` + the **generated**
+  `platform/keycloak/passkey-desired-state.yaml` — the desired state, rendered from the realm
+  import, `--check`ed by the merge gate.
+- `conformance/tests/l0-static/keycloak-passkey-reconciler/` — `check.sh`, `chainsaw-test.yaml`
+  (`scope: whole-repo`), `mutations.py`.
+- `platform-config/kustomization.yaml` — both new resources, in the **same** Kustomization as the
+  realm import.
+- `platform/keycloak/README.md` — the layout table, the section heading, and the "does NOT reach the
+  live realm" section rewritten: all three declared things now converge, with the blue/green
+  mechanics, the rollback command and the `--no-bind` fallback written out.
+
+### The flow design was reconsidered, and the first version of it was wrong
+
+**This is worth recording because the superseded reasoning was not silly, it was just wrongly
+scoped.** The first design of pass 3 was **audit-only**: compare the live browser flow against git,
+report divergence loudly, and never write it — leaving the restructure as a one-time admin-console
+action. The reasoning was that converging a five-flow / ten-execution tree needs create-subflow,
+create-execution, set-requirement and bind-config as *separate* untransacted admin-API calls, so a
+run dying between them leaves a **half-built flow bound as the login path** for every SSO-gated app
+*and* kubectl — and this repo has had a total SSO outage from a realm change (2026-09-06), with the
+conditional-OTP sub-flow (which the declared structure moves one level deeper) once 403ing every
+platform-admin.
+
+**That risk analysis was sound but it assumed the wrong shape.** It is true only if you mutate the
+*bound* flow. The operator's instruction was *"if this can be done through GitOps, it should"*, and
+audit-only did not meet it — a manual admin-console step is exactly what the one-shot import already
+forces. So:
+
+- All the multi-call construction happens on a flow **nothing is using** (the declared alias plus a
+  hash of the desired shape, applied to *every* alias in the tree — reusing `browser-mfa-forms`
+  would mean writing *into* the bound tree). A failure at any point leaves an orphaned, unbound flow
+  and **zero user-visible effect**. The untransacted-calls problem dissolves.
+- The built flow is **read back and verified** — nesting levels, ordering, authenticators,
+  requirements, and the conditional-OTP config *contents* — **before** anything is bound. The
+  regression the operator feared is now caught *before* it can bite rather than after.
+- The flip is **one atomic PUT** of the realm setting `browserFlow`.
+- **Rollback is one write.** The previous alias is printed in the run log, and there is **no
+  `DELETE` anywhere in the allow-list**, so the old flow survives as a working rollback target.
+
+**The new safety property, enforced structurally rather than by intention.** Blue/green needs
+flow-create and execution-add endpoints, which widens the allow-list — and the allow-list alone
+cannot express "but not the flow that is bound", because its write entries are alias-parameterised.
+So a second guard sits at the same choke point in `_http()`: the live `browserFlow` alias and its
+**entire subtree** of sub-flow aliases and execution ids are read first, and every write is screened
+against those sets before a socket opens. Protecting the whole subtree matters — adding an execution
+to `browser-mfa-forms` changes live login behaviour just as surely as touching `browser-mfa`. It is
+**fail-closed**: the sets start each run as "not determined", and while they are, *every* flow write
+is refused; if the bound flow's executions cannot be read they stay undetermined and the flow pass
+writes nothing.
+
+**Idempotence became load-bearing rather than merely nice.** A rebuild-and-reflip every 15 minutes
+would churn the login path on a schedule, so the bound flow is compared *structurally* and a match
+produces **zero writes and no new flow**. Sub-flow and config *aliases* are deliberately not
+compared: a flow this program built carries a hash suffix, so comparing labels would make it
+permanently unrecognisable as converged. A crashed run's completed-but-unbound flow is **reused**; a
+half-built one is left alone and the next candidate alias is used, with the candidate count
+**bounded** so a persistently failing build fails loudly instead of creating unbounded sprawl.
+
+**Where the rollback target is recorded, and why not as an annotation.** The brief suggested
+recording the previous `browserFlow` alias as an annotation on the CronJob or in a ConfigMap. The
+pod **cannot**: it runs with `automountServiceAccountToken: false` and has no Kubernetes API access
+at all, which is the same property that made the mounted-generated-ConfigMap design preferable to
+reading the CR (and is why this change adds no `**/rbac.yaml`). Giving it write access to its own
+annotations would trade a narrow, gate-verified read for a Kubernetes API writer, which is a worse
+bargain than the one it would buy.
+
+So the record is in two places that cost nothing: the **run log** prints
+`ROLLBACK is one write: set browserFlow back to <alias>. The old flow was NOT deleted.`, and — more
+durably — **the realm itself**, because nothing is ever deleted. Every superseded flow is still
+listed under `Authentication -> Flows`, so the rollback target is discoverable from the admin console
+even if every Job log has rotated away. The README carries the rollback procedure.
+
+**I achieved full convergence including the bind.** `--no-bind` exists as the deliberate fallback
+(build and verify, leave the flip as a one-write human act) but it is not the default and was not
+needed.
+
+**No `**/rbac.yaml` was touched.** The brief floated having the reconciler read the
+`KeycloakRealmImport` CR from the Kubernetes API, which would need a Role, a RoleBinding and a
+ServiceAccount token. I rejected that for the narrower, already-proven pattern: a desired state
+**generated** from the import and **mounted**, so the pod keeps `automountServiceAccountToken:
+false` and no Kubernetes API access at all. Skew is prevented by the gate running the renderer with
+`--check`, not by a second declaration being trusted.
+
+**Sensitive paths touched** — checked against `infra/gitea/approval-policy.json` rather than from
+memory, which caught one I had initially missed. **Three**:
+
+| Path | Why it is in the policy | What I changed |
+|---|---|---|
+| `ci/**` | the pipeline itself | added `ci/render-keycloak-passkey-policy.py` |
+| `conformance/**` | changing the tests changes what green *means* | added the L0 test directory |
+| `platform-config/kustomization.yaml` | Flux wiring / which manifests are actually applied | added the two new resources |
+
+**Not** touched: `**/rbac.yaml`, `clusters/**`, `platform/kyverno/**`, `.gitea/**`, `bootstrap/**`,
+`infra/gitea/**`, `.sops.yaml`, any `*.sops.yaml`, `CLAUDE.md`.
+
+So this PR **requires an approving review from someone who is not its author**, and
+`ci/approval-policy.sh` is fail-closed, so it will block until that exists. I did **not** post
+`/approve-override` — CLAUDE.md §1 names doing that on your own agent PR as exactly the
+self-approval the stage exists to prevent. The operator needs to either review it or post the
+override.
+
+## What I verified
+
+**Measured against the live cluster** (read-only `kubectl --context k3s-prod`):
+
+- `keycloakrealmimport homelab`: `Done=True Started=False HasErrors=False` — the import is a
+  completed one-shot. **This is the premise of the whole change and it is measured, not assumed.**
+- Keycloak image actually running: `quay.io/keycloak/keycloak:26.0.6`.
+- `kustomization/platform-config` `.status.lastAppliedRevision` ==
+  `gitrepository/flux-system` artifact revision == `main@sha1:83d3568…`, which contains the PR #72
+  merge. So the inert declaration is genuinely applied and genuinely inert.
+
+**`PUT /admin/realms/{realm}` does not touch users, groups, clients or roles in 26.0.6** —
+established from source at tag `26.0.6`, not from documentation or memory:
+
+- `RealmAdminResource.updateRealm` is the only handler. Its javadoc: *"Any user, roles or client
+  information in the representation will be ignored. This will only update top-level attributes of
+  the realm."*
+- It delegates to `RepresentationToModel.updateRealm` →
+  `DefaultExportImportManager.updateRealm` (`model/storage-private/.../datastore/`), **lines
+  725–894**. I brace-matched that body and enumerated every `rep.getX()` call in it: it contains
+  **no** `getUsers()`, `getClients()`, `getGroups()`, `getRoles()`, `getRequiredActions()`,
+  `getAuthenticationFlows()` or `getAuthenticatorConfig()`. Those objects are **unreachable** from
+  the endpoint, not merely ignored by convention.
+- Two findings that changed the design, both from the same read:
+  - `updateRealm` **does** read `rep.getBrowserFlow()`, so the endpoint can rebind the realm's login
+    path. The write therefore carries `browserFlow` straight back from the GET and a hard assertion
+    refuses the PUT if it would differ.
+  - The passwordless policy is applied **all-or-nothing and only when `RpEntityName` is non-empty**
+    (line 842), and `getWebAuthnPolicyPasswordless(rep)` substitutes a **Keycloak default** for every
+    null field (line 1253ff). A partial PUT would therefore silently reset every key it omitted, and
+    one missing `RpEntityName` would drop the whole policy while returning 204. This is why the write
+    is a strict read-modify-write of the full live representation.
+- `AuthenticationManagementResource.updateRequiredAction` (line 1205) is **per-alias** and likewise
+  **replaces** the provider from the representation (name, alias, priority, enabled, defaultAction,
+  config), and 404s when the alias is unregistered. Hence read-modify-write there too.
+- `DefaultRequiredActions` registers `webauthn-register-passwordless` with `enabled=true`,
+  `defaultAction=false`, `priority=80` — confirming the realm import's claim that declaring it would
+  restate a default while deleting ten siblings.
+- **The realm GET really does return what the read-modify-write needs.** This is load-bearing: if
+  the GET omitted the passwordless keys, the PUT body would be missing them and 26.0.6 would reset
+  them to defaults. `ModelToRepresentation.toRepresentation` (lines 471–482) sets **all ten**
+  `webAuthnPolicyPasswordless*` fields plus `ExtraOrigins`, and line 496 sets `browserFlow`. I
+  checked for an `internal`/`export` guard around that block — there is none in lines 380–500, so
+  those fields are emitted unconditionally, not only on the export path.
+
+**The five flow-construction endpoints, read from 26.0.6 source before being used:**
+
+- `POST /authentication/flows` — `AuthenticationManagementResource.createFlow`, rejects a duplicate
+  alias. Carries **no alias in its path** (the alias is in the body), which is why it needed its own
+  body-based screen; without that it escaped the protected-subtree guard entirely. My selftest
+  caught that, not my reading.
+- `POST /flows/{alias}/executions/flow` — `addExecutionFlow`, body `{alias, type, description}`.
+  Creates the sub-flow **at requirement DISABLED**, and refuses if the parent `isBuiltIn()`.
+- `POST /flows/{alias}/executions/execution` — `addExecutionToFlow`, body `{provider}`. Also
+  DISABLED unless the factory has exactly one requirement choice. Both assign `getNextPriority()`,
+  so **insertion order is the resulting order** — which is why the builder inserts depth-first in
+  declared order and then verifies by readback.
+- `PUT /flows/{alias}/executions` — `updateExecutions`. **A trap I would have walked into:** it
+  compares `rep.getPriority()` against the stored priority, and a JSON body omitting `priority`
+  deserialises it as the int **0** — which would silently **reshuffle the whole execution order**
+  while only a requirement change was intended. Every requirement write therefore carries the
+  execution's current priority back. There is a mutation for it and a selftest assertion on the
+  resulting order.
+- `POST /executions/{id}/config` — `newExecutionConfig`, creates an authenticator config and binds
+  it. Config aliases are suffixed like flow aliases, because 26.0.6 keeps them unique per realm and
+  the old flow already uses the declared alias.
+
+**Offline proofs, all green:**
+
+- The reconciler's `--selftest`, **extracted from the shipped ConfigMap** and run: **216 assertions
+  pass**. Its stub Keycloak models the 26.0.6 behaviours that bite — the RpEntityName gate, the
+  all-or-nothing policy rebuild, the required-action replace, the `updateExecutions` priority trap,
+  and flow construction where everything is created DISABLED — so these are tested against the real
+  semantics rather than a forgiving fake.
+- **A build killed at four different points (after 1, 3, 6 and 9 construction writes) never binds
+  and never touches the live flow.** Each case asserts the realm is still bound to the old alias,
+  that no write targeted the bound subtree, and that every live flow survives. This is the assertion
+  that makes blue/green a property rather than a claim.
+- **Verify-before-bind is exercised, not assumed.** Two stub sabotages — a server that accepts the
+  requirement writes and applies the wrong value, and one that writes the conditional-OTP role under
+  `condition.evaluated.user.role` — both end with the flow **built but NOT bound** and the old flow
+  still live.
+- `mutations.py`: **all 30 injected regressions caught**, including the six that are the blue/green
+  safety argument itself (build into the bound flow; never call the screen; reset the protected sets
+  to empty; bind before verify; report a failed build as success; reuse an existing flow without
+  verifying it).
+- Mutation testing again found real gaps in my own work, which I fixed rather than weakening the
+  mutations. Across the two designs: no readback coverage on the realm write, no coverage of an
+  undeclared policy key surviving, a mis-indented anchor, and — in this design — **no coverage at
+  all of the bound flow being unreadable**, which is the fail-closed path. It also showed me that
+  **two of my proposed mutations were not real regressions**: the module-level initial value of the
+  protected sets is dead code (`run()` resets them first), and removing the early return after a
+  build failure is covered by verify-before-bind anyway. I re-anchored both onto the properties that
+  *are* load-bearing rather than pretending the originals had been caught.
+- The new L0 `check.sh`: rc 0, all assertions. It asserts the renderer `--check`; that the realm
+  import and the desired state are rendered by the **same** Flux Kustomization; that the renderer's
+  and reconciler's writable key tuples are identical (parsed out of both sources with `ast`); that
+  `defaultAction` is `false`; that the realm import declares **no** `requiredActions` block
+  (previously only a comment saying "DO NOT ADD ONE"); that the **sibling** reconciler's allow-list
+  was not widened; that **nothing is deletable** and the execution-/config-id-scoped write endpoints
+  stay off the list (they carry no flow alias, so they could not be screened); and — the
+  load-bearing one — that the **protected-subtree guard exists in the source, is wired into
+  `_http()`, resets to "undetermined" every run, screens the body-only create-flow endpoint, and
+  that the policy write still refuses to rebind `browserFlow`**.
+- `ci/render-keycloak-passkey-policy.py --write` output round-trips: the generated JSON is accepted
+  by the reconciler's `declared()`, its 10 policy keys equal the reconciler's `POLICY_FIELDS`, and
+  it flattens to the exact 11-step tree the realm import declares, with
+  `conditional-otp-platform-admin` bound at level 4.
+- `${BASE_DOMAIN}` is carried into the generated ConfigMap **verbatim**, and both files sit in
+  `platform-config`, so one substitution pass decides the RP ID.
+- Zero dollar signs in the reconciler script (the Flux envsubst trap), verified on the committed
+  manifest.
+
+**The conditional-OTP regression is detectable — and now also preventable.** The comparison is
+key-by-key on the authenticator config contents, and because it runs **before** the bind, a flow
+carrying the bug never becomes the live login path. `--selftest` covers it three ways: the historical
+`condition.evaluated.user.role` key injected into a *built* flow (built, verified, **not bound**,
+old flow still live); the same bug already live in the *bound* flow (reported, then **fixed** by
+building and binding a correct flow); and the config disappearing entirely. `mutations.py` deletes
+the comparison and the undeclared-key report and requires the selftest to notice both. The suite
+also covers a requirement flipped from `ALTERNATIVE` to `REQUIRED`, the un-restructured legacy flow,
+a correct-but-unbound flow, and a cyclic declaration.
+
+**Gate:** see the gate section of the PR / the run quoted in my report — rc and per-stage results
+are recorded there rather than paraphrased here.
+
+## What I did NOT verify
+
+- **The live realm's actual passkey state is entirely UNVERIFIED.** I never authenticated to the
+  Keycloak admin API. Doing so needs the `keycloak-admin-credentials` password, which CLAUDE.md §6
+  forbids decrypting or printing, so I did not read the realm's current
+  `webAuthnPolicyPasswordless*` values, its required-action list, or its `browser-mfa` executions. I
+  therefore **do not know** how much drift the first real run will report, nor that the live flow is
+  in the "legacy shape" my selftest models — only that the import never re-ran, which makes drift
+  the overwhelmingly likely state. The first run's log is the measurement, and nobody has seen one.
+- **Nothing has ever run against a real Keycloak.** Every behavioural proof is against an
+  in-process stub I wrote, and a stub I wrote cannot falsify my own reading of the source. This
+  matters more for the blue/green flow builder than for anything else in the change, because it is
+  **five endpoints I have never exercised**. Specifically unproven in reality:
+  - That `POST /flows/{alias}/executions/flow` and `.../executions/execution` accept the bodies I
+    send and assign priorities in insertion order. If they do not, the build produces a tree that
+    fails verification — which is the safe failure (nothing is bound, login unaffected), but it
+    means **the flow would never converge** and the Job would fail every 15 minutes.
+  - That `PUT /admin/realms/homelab` accepts a full round-tripped representation. The analogous
+    *client* endpoint rejects one with `400 "Cannot parse the JSON"` (measured 2026-09-06), and the
+    sibling reconciler's whole minimal-PUT rule exists because of it. The source says the realm
+    endpoint differs; no measurement here confirms it. **If it does not, both the policy pass and
+    the atomic bind fail closed** — loudly, writing nothing.
+  - The exact JSON shapes compared: `AcceptableAaguids` as `[]` rather than absent or null,
+    `CreateTimeout` as a JSON int, `RequireResidentKey` as the literal string `"Yes"`. A shape I did
+    not anticipate would show up as **perpetual drift** — a write every 15 minutes that never
+    converges — rather than as an error.
+  - That `authenticationConfig` on an execution-info response is the config **id** (I read it from
+    the representation class, but never saw a live response).
+- **I did not verify the alias-collision behaviour against a real realm.** The recovery logic
+  assumes `POST /flows` returns a non-2xx on a duplicate alias (the source checks
+  `getFlowByAlias != null` and raises `ErrorResponse.exists`), and that a config alias collision
+  likewise fails. If either silently succeeded, the recovery path would behave differently than
+  modelled.
+- **Flow sprawl is bounded but real.** Every genuinely new desired shape creates a new flow and
+  **nothing is ever deleted** — by design, because the old flow is the rollback target. Over many
+  shape changes the realm accumulates unbound flows. There is no cleanup, automated or documented
+  beyond "an operator deletes them". I consider that the right trade (deletion is the one operation
+  that could destroy a rollback target) but it is a cost, not a non-issue.
+- **No L2 behavioural test was added.** The reconciler has an `--audit` mode built for one and the
+  sibling has `tests/l2-behavioral/keycloak-client-secrets-match-git`, but I added no equivalent.
+  And `vars.CLUSTER_AVAILABLE` is unset, so L2/L3 would not run in CI anyway. **CI proves the
+  offline subset only; it proves nothing about the live realm.**
+- **No alerting.** Unlike the first design there is no longer any failure that is "expected by
+  design", so a PrometheusRule on `kube_job_status_failed` would now be meaningful — and I did not
+  add one. A silently failing CronJob (image pull failure, rotated admin password, changed Secret
+  key) is visible only to someone who looks at Jobs. This is a real gap, not a resolved one.
+- **That passkey login works.** No passkey has been enrolled and no browser has been pointed at
+  this. Nothing offline can prove a WebAuthn ceremony completes, that the `ALTERNATIVE` fork
+  degrades to the password form via `CREDENTIAL_SETUP_REQUIRED`, or that a platform-admin is still
+  asked for TOTP on the password branch. The README's four-step manual check remains the only thing
+  that would establish it, and it remains undone.
+- **That the RP ID is right for the origin.** `${BASE_DOMAIN}` is a legal registrable-domain suffix
+  of `keycloak.${BASE_DOMAIN}` on paper, but a wrong RP ID fails **in the browser**, before Keycloak
+  is involved, with no server-side log line. The reconciler will happily converge the realm on a
+  wrong RP ID; it checks that git and the realm agree, never that git is correct.
+- **The first real run will change how everyone logs in.** Once this merges and Flux applies it, the
+  next tick builds a flow and — if it verifies — **flips the realm's browser flow**. The UX change
+  is deliberate and documented (username screen, then password), but it is the first time it will
+  have happened, unattended, on a schedule. The rollback is one write and the old flow survives;
+  that is the mitigation, and it has not been rehearsed either.
+- **CI's own result.** My gate run is local. `gitea-settings` cannot run locally, and
+  `ci/approval-policy.sh` reports green N/A offline while this branch touches three sensitive paths,
+  so **the PR is guaranteed to need a non-author approving review** and I have not seen CI report. I
+  did not and will not post `/approve-override` on my own PR.
+- I re-measured branch protection this session (`protected: true`, `user_can_push: false`,
+  `enable_status_check: true`, context `gate / gate (pull_request)`, `required_approvals: 0`), but
+  the branches endpoint does **not** expose `block_admin_merge_override`, so that row of CLAUDE.md
+  §2 is **not re-confirmed** by me.

decisions/2026-10-18-pihole-local-ttl.md

@@ -0,0 +1,82 @@
+---
+date: 2026-10-18
+branch: fix/pihole-local-ttl
+title: "The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes"
+---
+
+## What changed
+
+The operator reported DNS "flapping between the cluster ingress and the router (public IP)".
+PiHole was not at fault — both replicas answer correctly and consistently. The bug is an
+**asymmetry in cacheability** that we created.
+
+Measured on prod 2026-10-18:
+
+```
+dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+```
+
+dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The SOHO router forwards
+LAN queries to PiHole and **caches** the result — the decrementing TTL above is the proof it caches
+rather than forwarding live. So:
+
+* the **correct** answer is **uncacheable** (TTL 0): the router must re-query PiHole on *every*
+  lookup, which is maximum exposure to any momentary gap;
+* the **wrong** answer is **cacheable for 300s**: the instant the router falls back to its upstream
+  (standard forwarder behaviour when the forward target does not answer) it latches the public WAN
+  address and serves it to **every LAN client** for five minutes.
+
+So **one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP** —
+then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed.
+That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip.
+
+Fix: `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines` in `catalog/pihole.yaml`, so the
+private answer is cacheable and a short gap is invisible to clients. `apps/wave-0/pihole.yaml`
+regenerated with the canonical render command.
+
+## What I verified
+
+* **PiHole is NOT the fault.** Both replicas, queried directly by pod IP, answer `192.168.2.20`;
+  the VIP returned it 6/6; this workstation 8/8. DHCP hands out only `192.168.2.23`.
+* **The router caches rather than forwards live** — three successive queries returned TTL 153, 148,
+  143 (decrementing), which a pure forwarder could not do.
+* **The TTL asymmetry**, as quoted above: 0 from PiHole, 300 from public DNS.
+* **The router does answer DNS and returns the public chain** (`wan.home.keithjackson.dev` → the WAN
+  address), so a fallback always has a wrong answer available.
+* **`local-ttl` is the right directive**: dnsmasq applies it to replies made from configuration,
+  `/etc/hosts` and DHCP leases, which covers all five `address=` names with one directive. It is a
+  standalone option, not a sub-field of `address=`, so prepending it to the `;`-separated
+  `FTLCONF_misc_dnsmasq_lines` list is syntactically correct.
+* **The render is clean and scoped**: `catalog/render.py --all catalog -o apps --wave-layout
+  --no-image-automation` (the canonical command `ci/render-golden.sh` asserts against, and
+  deliberately **not** `--emit-waves`) changed exactly two files — `catalog/pihole.yaml` and
+  `apps/wave-0/pihole.yaml` — and the rendered diff is the single `local-ttl=60;` prefix.
+
+## What I did NOT verify
+
+* **UNAPPLIED.** Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL,
+  nor the router caching the private answer, nor the flap stopping. **Verify after merge with
+  `dig home.keithjackson.dev @192.168.2.23` (expect `TTL=60`, not 0) and then the same query at the
+  router (expect `192.168.2.20` with a decrementing TTL, not `wan.…`).**
+* **I did not confirm that PiHole/FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The
+  directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header
+  records that FTL has previously written a bogus line into `dnsmasq.conf` and reported "0 invalid"
+  — so a malformed passthrough fails *silently*. If the TTL is still 0 after merge, that is the
+  cause, not this reasoning. Check the pod's `/etc/pihole/dnsmasq.conf`.
+* **60s is chosen, not derived.** Long enough for a pod restart and a lease re-election (both
+  seconds); it does **nothing** for an outage longer than the TTL, where the cache expires and the
+  public answer latches again. That is a real availability problem, not a caching one.
+* **This does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services
+  (`pihole-dns-tcp`, `pihole-dns-udp`) with **two independently-elected L2 announcement leases**, so
+  two nodes can ARP for the same address at once — they sat on different nodes earlier the same day
+  (tcp on prod-2, udp on prod-1). The same pattern applies to crafty (`.22`) and unifi (`.21`).
+  Merging each pair into one mixed-protocol Service would remove the class; the operator has parked
+  that decision, and it is deliberately **not** in this change.
+* **Client-side bypasses are unexamined.** A device with its own resolver, or a browser with
+  DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help
+  there. Only this one workstation's resolver chain was inspected.
+* **No alert fires if the split-horizon answer regresses to the public address.** Nothing probes
+  `home.keithjackson.dev` from a LAN vantage point and compares it against `${GATEWAY_VIP}`, so the
+  next recurrence would again be found by a human noticing. That gap is not closed here.

decisions/2026-10-18-renovate-cron-window.md

@@ -0,0 +1,78 @@
+---
+date: 2026-10-18
+branch: fix/renovate-cron-window
+title: "The Renovate cron fired outside Renovate's own schedule, so it would never have opened a PR"
+---
+
+## What changed
+
+`.gitea/workflows/renovate.yaml` shipped (PR #73) with `cron: "15 3 * * *"`. **That schedule could
+never have produced a single PR.**
+
+Gitea Actions cron is **UTC**. `renovate.json` carries `"schedule": ["after 3am and before 6am"]`
+with `"timezone": "America/New_York"`, and that inner schedule is **authoritative** — Renovate
+re-checks it and declines to open PRs outside it. So:
+
+```
+cron 03:15 UTC  ->  23:15 EDT, the PREVIOUS EVENING in New York
+window          ->  03:00-06:00 local
+inside?         ->  NO (3h45m before the window opens)
+```
+
+The original comment claimed the cron was "slightly ahead of the repo window ... so a run is always
+in flight when the window opens". It is 3h45m *behind*, on the previous day. Every nightly run would
+have started, logged "not within schedule", opened **zero** PRs and exited **successfully** — a
+green tick forever, indistinguishable from "nothing needed updating". That is the same
+"failure-mode-is-a-green-tick" shape this repo keeps hitting, and it would have hidden the ~30
+pending updates PR #73 was written to surface.
+
+Changed to `cron: "15 8 * * *"`, chosen to survive DST rather than to be tidy. The window's UTC
+position moves:
+
+| | local window | in UTC |
+|---|---|---|
+| EDT (UTC-4, Mar–Nov) | 03:00–06:00 | 07:00–10:00 |
+| EST (UTC-5, Nov–Mar) | 03:00–06:00 | 08:00–11:00 |
+
+`08:15 UTC` is inside **both** (04:15 EDT / 03:15 EST), so no edit is needed at either switchover.
+A cron cannot follow DST, so this is the only way to get it right once.
+
+## What I verified
+
+* **The defect, by arithmetic rather than assertion:** `datetime(2026,10,18,03,15, UTC)` converted to
+  `America/New_York` via `zoneinfo` is `2026-10-17 23:15 EDT`; `3 <= hour < 6` is **False**.
+* **The fix, at both DST extremes:** `08:15 UTC` → `04:15 EDT` in July and `03:15 EST` in December,
+  both inside `03:00–06:00`. Computed, not reasoned.
+* **The inner schedule really is authoritative and really is set:** `renovate.json` on `main` carries
+  `timezone = America/New_York` and `schedule = ['after 3am and before 6am']`.
+* **`RENOVATE_TOKEN` now exists** as a repository Actions secret (listed by name via the Gitea API
+  alongside `SETTINGS_API_TOKEN`), so the workflow's fail-hard preflight will pass. It did not exist
+  when PR #73 was authored.
+* The edited workflow parses as YAML.
+
+## What I did NOT verify
+
+* **Renovate has still never executed.** This fixes *when* it runs, not *whether it works*. No run
+  has happened, so nothing here proves it extracts dependencies, authenticates to Gitea, or opens a
+  usable PR.
+* **`enabledManagers` may silently disable the custom managers — UNRESOLVED and the likelier
+  remaining bug.** `renovate.json` lists `enabledManagers: [..., 'regex', ...]` and defines
+  `customManagers` (correct modern key; `regexManagers` is absent, which is right). But modern
+  Renovate names the custom manager **`custom.regex`** in `enabledManagers`, not bare `regex`. If
+  the bare name is invalid or ignored, every inline `# renovate:` marker in the tree does nothing —
+  including the `external-snapshotter` `newTag` pin PR #73 flagged as probably inert. **I did not
+  change it**, because guessing at a config key that could fail validation and take the whole run
+  down is worse than leaving a question that one dry run answers definitively. The `dry_run: true`
+  dispatch log settles it: look for a per-manager dependency-extraction count, and for any config
+  validation warning naming `regex`.
+* **Whether a dry run inside vs outside the window differs usefully.** Renovate honours its schedule
+  even in dry-run, so a manual dispatch outside 03:00–06:00 local will also log "not within
+  schedule". That makes the first dispatch's output less informative than it looks; to see what
+  Renovate *would* do, either dispatch inside the local window or override the schedule for that
+  run. Not tested either way.
+* **No alert fires if the schedule stops firing.** PR #73 recorded this gap and it is still open: a
+  Renovate workflow that silently stops running (disabled Actions, expired token, deleted secret)
+  produces no signal at all. Closing it needs a check on last-successful-run age, which nothing
+  does.
+* No conformance test asserts the cron falls inside `renovate.json`'s window, so this exact bug can
+  return. A static check comparing the two is feasible and is not written.

decisions/2026-10-18-renovate-invalid-config.md

@@ -0,0 +1,78 @@
+---
+date: 2026-10-18
+branch: fix/renovate-invalid-config
+title: "renovate.json was invalid, so the first manual run errored out"
+---
+
+## What changed
+
+The operator ran the Renovate workflow manually and it **failed**. The cause is not the workflow:
+**`renovate.json` was never a valid Renovate config.**
+
+It carried five pseudo-comment keys — `_comment`, `_kubernetes_comment`, `_unmanageable_comment`,
+`_ignorePaths_comment`, `_docker_comment` — because JSON has no comments. Renovate **rejects unknown
+top-level options**, so the entire config failed validation. Proven with the official validator
+rather than inferred:
+
+```
+$ podman run --rm -v "$PWD/renovate.json:/tmp/renovate.json:ro,Z" \
+    ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json
+ERROR: Found errors in configuration
+  Invalid configuration option: _comment
+  Invalid configuration option: _docker_comment
+  Invalid configuration option: _ignorePaths_comment
+  Invalid configuration option: _kubernetes_comment
+  Invalid configuration option: _unmanageable_comment
+```
+
+This is the second independent reason Renovate had never worked, on top of PR #77's cron firing
+outside `renovate.json`'s own schedule window. Either alone was sufficient to produce "nothing ever
+happens"; #73 shipped both.
+
+**Converted `renovate.json` → `renovate.json5`.** JSON5 permits real `//` comments, and Renovate
+discovers `renovate.json5` natively — nothing in `.gitea/` or `ci/` pins the old filename (only
+prose inside comments), and no `RENOVATE_CONFIG_FILE` is set. Every word of the original rationale
+is preserved as comments positioned next to what it explains, which a flattened `description`
+string would have lost.
+
+A header on the new file records the failure, the validator command, and the instruction to run it
+before committing — this class of bug is invisible to the repo's own gate.
+
+## What I verified
+
+* **The exact failure**, from the official validator, quoted above. Not read off a log, not inferred
+  from the workflow.
+* **The fix validates**: the same validator against `renovate.json5` returns
+  `INFO: Config validated successfully against 1 file(s)`.
+* **Nothing was dropped in the conversion**, checked programmatically against `git show
+  HEAD:renovate.json`: 18 non-comment keys before, 18 after, **no missing keys, no extra keys, and
+  no key whose value changed**. `docker: {enabled: true}` confirmed still present — the validator
+  prints a normalisation diff that mentions it, which is its own migration suggestion, not a
+  removal.
+* **74 comment lines survive** in the new file, so the rationale is intact.
+* **`RENOVATE_TOKEN` exists** as a repository Actions secret (listed by name via the Gitea API), so
+  the workflow's fail-hard preflight is satisfied.
+* **A suspicion was DISPROVED:** `enabledManagers` containing bare `regex` was thought to need to be
+  `custom.regex`, which would have silently disabled every inline `# renovate:` marker. The
+  validator accepts `regex`, so that concern was unfounded and nothing here changes it.
+* Nothing functionally references `renovate.json` by name — `grep -rn "renovate\.json" .gitea/ ci/
+  conformance/` returns only comment prose.
+
+## What I did NOT verify
+
+* **Renovate has still never completed a run.** This removes one of two blockers; PR #77 removes the
+  other. Neither is merged, so **no successful Renovate run has been observed** and I cannot say the
+  bot works — only that these two specific defects are fixed.
+* **I did not re-run the workflow after the fix.** The honest test is a `workflow_dispatch` with
+  `dry_run: true` once both this and #77 are merged. Expect ~30 dependencies behind, rate-limited to
+  `prHourlyLimit: 2` / `prConcurrentLimit: 5`.
+* **Whether the managers actually extract anything.** A valid config that matches no files is still
+  a no-op. The dry-run log's per-manager dependency counts are the only thing that proves the
+  `managerFilePatterns` are right, and I have not seen them.
+* **Whether `renovate.json5` is honoured by the pinned Renovate version in the workflow.** JSON5
+  discovery is long-standing and the validator accepted the file, but the workflow's image tag was
+  not cross-checked against the version that added it.
+* **Nothing in the gate catches an invalid Renovate config.** This bug would have recurred silently.
+  A one-line L0 check shelling out to `renovate-config-validator` would close it; it is not written
+  here, and `conformance/**` is a sensitive path.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

decisions/2026-10-29-image-pipeline.md

@@ -0,0 +1,374 @@
+---
+date: 2026-10-29
+branch: feat/image-pipeline
+title: "Images are built, verified and published by CI on rootless BuildKit; Renovate carries the digests into git"
+---
+
+## What changed
+
+Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and
+the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's
+package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and
+both `build-image.sh` scripts said "publishing is an operator action".
+
+### The builder: rootless BuildKit — DECISION MADE (operator, on PR #81)
+
+**The problem it answers, measured on this PR's first CI run (images run 325, failed in 6 s):** a
+CI job on this runner gets **no Docker daemon at all**, for two reasons, both in the deployed
+versions' source:
+- act_runner v0.2.11 `internal/app/cmd/daemon.go` sets `container.docker_host` to `-` ("never
+  mount into jobs") whenever `DOCKER_HOST` is `tcp://`, which it is here (the DinD sidecar);
+- gitea/act v0.261.2 `pkg/container/docker_run.go` `sanitizeConfig` drops **every** bind,
+  socket included, when `valid_volumes` is empty, which it is (the runner has no config file).
+
+Giving jobs the DinD daemon would have made every branch push root on the node, because that
+daemon is `privileged: true`.
+
+**The decision:** no job may reach a privileged daemon, and images are built by a **rootless
+BuildKit** daemon over mTLS. The runner (`apps/wave-1/gitea-actions.yaml`, DOCKER_HOST,
+valid_volumes) is **untouched**.
+
+- **`platform/buildkit/`** — a new hand-managed component in its own namespace `buildkit`. The
+  runner manifest is renderer-generated (`catalog/render.py` EXTRA_UNITS), and the builder has no
+  business sitting next to a privileged container. Flux reconciles it as the new Kustomization
+  `buildkit` (`clusters/base/buildkit.yaml`, `dependsOn: platform`).
+  - **Deployment:** `moby/buildkit:v0.34.1-rootless@sha256:f68752a5…0c4f89`, pinned by index
+    digest. The securityContext, every line measured (see *What I verified*):
+    - `privileged: false`;
+    - uid/gid 1000, `runAsNonRoot`;
+    - `capabilities: drop ALL, add SETUID, SETGID`;
+    - `allowPrivilegeEscalation: true`;
+    - `seccompProfile: Unconfined` and `appArmorProfile: Unconfined`;
+    - `readOnlyRootFilesystem: true`, with five emptyDirs.
+
+    Other settings:
+    - `--oci-worker-no-process-sandbox`;
+    - no ServiceAccount token;
+    - memory limit 4Gi, which covers build steps too;
+    - cache emptyDir with `sizeLimit: 6Gi`, and GC capped at 4 GB in `buildkitd.toml`
+      (ConfigMap).
+  - **mTLS via cert-manager** (`certificates.yaml`): a self-signed CA used only here, the server
+    cert `buildkitd-tls`, and the client cert `buildkit-client-ci`. buildkitd verifies every
+    client against that CA.
+  - **Service:** ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`.
+  - **NetworkPolicies:**
+    - default-deny;
+    - ingress on 1234 **only from the act-runner pod** (`gitea-actions`,
+      `app.kubernetes.io/name=act-runner`);
+    - egress: DNS, world :443/:80 for downloads, and the `ingress` entity (the Gateway) for the
+      registry push.
+  - `versions.env`: the CI-side client, `buildx` v0.38.1 with the sha256 from its
+    `checksums.txt`, plus the address and certificate name CI dials.
+- **How DinD job containers appear to Cilium (measured, read-only).**
+  `iptables -t nat -S POSTROUTING` in the dind container shows
+  `-A POSTROUTING -s 172.18.0.0/16 ! -o br-… -j MASQUERADE` (and the same for docker0). So a job's
+  connection leaves with the **act-runner pod's IP, and therefore that pod's Cilium identity**.
+  That makes "only the act-runner pod" expressible and correct, but it cannot tell one job from
+  another, or a job from act_runner itself. Per-job authentication is the client certificate.
+- **Admission: no Kyverno exception needed.**
+  - Every ClusterPolicy actually applied (`platform-config/kustomization.yaml`; live:
+    `conformance-trigger-job-shape`, `label-cdi-worker-pods`, `pm-agent-job-shape`,
+    `require-auth-mode-label`, `require-default-deny-baseline`, `require-egress-class-label`,
+    `restrict-host-namespaces`, `secret-provenance`, `validate-default-deny-shape`) matches only
+    namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`, or only its own
+    namespace in the job-shape policies' case. `buildkit` carries no such label.
+  - There are no namespaced Kyverno Policies and no cluster-wide PSA default (both measured).
+  - The namespace is deliberately left without a PSA label, because PSS `baseline` forbids
+    `Unconfined` seccomp and AppArmor.
+  - `validate-default-deny-shape` does apply to the namespace's default-deny NetworkPolicy, and
+    its shape (both directions) satisfies it.
+- **Unprivileged user namespaces on the nodes (measured, read-only, via Ansible `sysctl`).** On
+  all three Debian 13.7 nodes, kernel `7.1.8+deb13-amd64`:
+  - `kernel.unprivileged_userns_clone = 1`;
+  - `user.max_user_namespaces = 127206`;
+  - no `kernel.apparmor_restrict_unprivileged_userns` (an Ubuntu-only key);
+  - AppArmor enabled (`Y`).
+
+  So this is a satisfied prerequisite, not an open one.
+- **The TLS conformance check needed a narrow, self-proving exception** (`conformance/**`, so this
+  changes what green means; flagged for the reviewer).
+  - The first gate run failed L0 `tls-lifecycle-automated` clause (5): "every Certificate's
+    issuer must be ACME". That is broader than the spec it serves:
+    - R-TLS-01 and docs/08 concern *externally reachable* services;
+    - ADR-0006 says "mTLS between workloads is a separate concern … not covered here";
+    - a public CA cannot issue for a `*.svc.cluster.local` name or a client-auth certificate.
+  - The new clause (5b) exempts a Certificate annotated
+    `homelab.keithjackson.dev/tls-scope: workload-mtls` **only if it proves it**:
+    - no Gateway listener serves its Secret;
+    - every DNS name and the commonName are a bare label or `*.svc[.cluster.local]`;
+    - it has no IP, URI or email SANs;
+    - its issuer is a namespaced `selfSigned`/`ca` Issuer in its own namespace;
+    - a `ca` issuer's Secret is produced by a likewise-exempt tracked Certificate.
+  - Exemptions are printed, and clause (9), the renewal arithmetic, still applies.
+  - **Negative-tested:** a public DNS name, and a Gateway serving `buildkitd-tls`, each fail the
+    check with a specific message.
+- **`infra/gitea/buildkit-client-secret.sh`** (operator action): copies `ca.crt`, `tls.crt` and
+  `tls.key` from the Secret `buildkit/buildkit-client-ci` into the Actions secrets `BUILDKIT_CA`,
+  `BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. Before writing anything it checks that:
+  - the cert verifies against the CA;
+  - the cert is a client-auth cert;
+  - the key matches the cert.
+
+  It never prints a value. `--check` reports whether the stored copy is older than the current
+  certificate.
+
+### Verification moved into the Containerfiles
+
+`docker run` cannot exist on BuildKit, so each Containerfile now names its shipped stage after the
+image (`toolbox`, `agent-runner`, `conformance-runner`) and ends in a **`verify` test stage**,
+`FROM <image> AS verify`. That stage runs every previous assertion at build time, as the image's
+non-root uid (65532 for agent-runner and conformance-runner), in `RUN --network=none`. It is never
+pushed.
+- **Every existing assertion is preserved**, and the toolbox gained one it never had: every tool
+  on PATH, and five tools report their pinned versions.
+- `probe-validation.sh` is `COPY`ed into the verify stage only, so it never ships.
+- **Bug fixed on the way:** in the old `docker run` block, `cmd | grep -q … && echo ok` and
+  `[ … ] && echo` could never fail the script under `set -e` + `pipefail`. The "entrypoints refuse
+  arguments", "not root", "no API key baked in" and "HOME writable" checks asserted nothing. They
+  are real assertions now.
+- **What a build step cannot reproduce, and how it is approximated** (also written in each stage's
+  header):
+  - **uid:** exact.
+  - **`--network none`:** exact, and each stage *proves* it. Its first check fails if DNS or a
+    TCP connection works. A negative control without `--network=none` fails that check.
+  - **`--read-only` rootfs:** not reproducible, because a RUN step always has a writable layer.
+    It is approximated by permission:
+    - `touch /should-fail` must fail;
+    - a `find / -xdev -writable ! -type l` inventory, run as the uid, must lie entirely under
+      `/workspace`, `/tmp` or `/var/tmp`. `! -type l` was needed: Fedora's `/usr/tmp` symlink
+      to `/var/tmp` tripped it.
+
+    The real control remains the pods' `readOnlyRootFilesystem`, which the admission policies
+    enforce.
+  - **`--tmpfs /workspace`:** approximated by the image's own `/workspace`, owned by 65532, which
+    is exactly what the pod's emptyDir replaces.
+
+  The opt-in `--verify-clone` still `run`s the image, so it remains local-only.
+- **`build-image.sh` (all three):**
+  - verify is `build --target verify` and the image is `build --target <image>`;
+  - a new `--print-build-args` prints the pins as `KEY=VALUE`, which is what CI consumes. The
+    arg list therefore exists once;
+  - the local podman path still works.
+- The L0 check `alert-rules-unit-tested` now looks for `"PROMTOOL_VERSION=$PROMTOOL_VERSION"` in
+  the toolbox's build-arg list.
+
+### The pipeline (`.gitea/workflows/images.yaml` + `scripts/images.sh`)
+
+- **The builder.** The driver downloads the pinned `buildx` and refuses a checksum mismatch. It
+  writes the client cert from the `BUILDKIT_*` secrets to a 0600 temp dir that is removed on exit,
+  creates a `remote` builder, and runs `buildx inspect --bootstrap`.
+- **It fails loudly, never skips.** It stops with a clear message if:
+  - the secrets are absent;
+  - buildkitd is unreachable. The message lists the causes in order: not deployed yet (Flux
+    applies it only after merge), stale certificate, NetworkPolicy.
+- **Pull request:** `buildx bake <image>-verify` for changed images. Nothing is pushed and no
+  registry credential is referenced. The `BUILDKIT_*` builder secrets *are* given to PR jobs,
+  because they must build.
+  - agent-runner is built FROM the toolbox built **in the same run** through a bake `target:`
+    context. That toolbox exists in no registry, so the toolbox is built as its base even when
+    its own inputs did not change.
+- **Push to main / dispatch on main:** for each image, the verify target first, and only then the
+  image itself with `push=true`. Tags are `sha-<12 hex>` (immutable) and `main` (moving).
+  - The digest is read back from the registry and compared with the client's metadata.
+  - agent-runner builds FROM the toolbox digest just pushed, or else from the pinned
+    `AGENT_BASE_IMAGE` (the zero placeholder bootstraps).
+  - `BUILDX_NO_DEFAULT_ATTESTATIONS=1` keeps one manifest per tag, which is what admission,
+    Renovate and the retention script assume.
+- **Unchanged from the first iteration:** change detection, publish idempotence,
+  `REGISTRY_PUSH_TOKEN` and its preflight, the Renovate markers, and `registry-retention.sh`. The
+  DinD prune step is gone, because nothing is built in DinD now.
+
+### Unchanged from the first iteration (still true)
+
+- **Digest references** carry `# renovate-digest: image=<repo> tag=main` markers:
+  - `platform/pm-agent/deployment.yaml` and `configmap.yaml`;
+  - `infra/agent/versions.env` `AGENT_BASE_IMAGE`, now the published toolbox by digest, zero
+    until the first publish;
+  - `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to zero.
+
+  The admission policies are untouched.
+- **`renovate.json5`**:
+  - the `custom.regex` marker manager;
+  - `enabledManagers` `regex` → `custom.regex`;
+  - the `kubernetes` manager disabled for these three repos;
+  - `ignorePaths` narrowed so `conformance/versions.env` is visible.
+
+  `renovate.yaml` pins `renovate-version: 44.167.0` and adds a docker `RENOVATE_HOST_RULES` entry
+  using `RENOVATE_TOKEN`, which now needs `read:package`.
+- **`infra/gitea/registry-retention.sh`** (dry run by default; operator action); the
+  `conformance-trigger-kubeconfig.sh` precondition fix; and the docs (pm-agent README, runbook,
+  plan, conformance README, devcontainer README, workflows README). The docs are updated again
+  for BuildKit, and there is a new `platform/buildkit/README.md`.
+
+### Decisions and pushback
+
+- **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.**
+  Checked against Gitea **v1.24.3** source:
+  - `services/auth/basic.go` `Basic.Verify` turns a task token into the pseudo-user
+    `gitea-actions` (ID −2) and sets no token scope;
+  - `routers/api/packages/container/container.go` `Authenticate` (the `/v2/token` exchange every
+    docker client performs) requires `read:package`/`write:package`/`all` scope, so it answers
+    401. The Actions token cannot even *pull*;
+  - `services/context/package.go` `determineAccessMode` has `// TODO: ActionUser permission check`
+    and no branch for that user. Even past login it would get at most read on a public org;
+  - `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3.
+
+  The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight
+  pattern. With BuildKit the credential reaches the daemon through the buildx session (a
+  0600 `config.json` in the job's temp dir), never through the cluster.
+- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate:
+  - `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` →
+    `custom.regex`;
+  - that migration runs on every repository run (`lib/workers/repository/init/merge.ts` →
+    `migrateAndValidate`), not only in the validator;
+  - `lib/modules/manager/index.ts` `getEnabledManagersList` accepts either spelling;
+  - `custom.regex` exists since 37.51.1 (renovatebot/renovate#24079).
+- **The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate
+  the workflow actually runs.**
+  - `renovatebot/[email protected]` defaults `renovate-version` to `39`
+    (`action.yml`, `src/docker.ts`), which is 39.264.0.
+  - The validator used in #79 was `ghcr.io/renovatebot/renovate` with no tag, i.e. `latest`,
+    44.167.0.
+  - **Measured:** `renovate:39.264.0 renovate-config-validator` on main's `renovate.json5` gives
+    `ERROR: Found errors in configuration` with five errors, all `managerFilePatterns`.
+    `fileMatch` was renamed in 40.2.0.
+  - So once #80 fixes the filename, every run would still die at `CONFIG_VALIDATION`.
+  - Fixed by pinning `renovate-version: 44.167.0`. This branch's config validates clean on it.
+    The only `--strict` complaint is a pending migration of the pre-existing top-level
+    `"docker": {"enabled": true}` block, which I left alone.
+- **Unworkable as briefed: tracking the worker Job template's digest.**
+  - `infra/agent/worker-job-template.yaml` is `COPY`ed into agent-runner. A Renovate bump there
+    changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on
+    forever.
+  - `pm-dispatch` overwrites that field from `PM_WORKER_IMAGE` at runtime
+    (`pm-dispatch.py:649`), so nothing is lost by leaving it all-zero. It is commented as
+    deliberately untracked.
+  - For the same loop reason, `conformance/versions.env` is **not** an input of the
+    conformance-runner build, and a pipeline-only change does not republish on main.
+- **Changed from the brief: retention is a script, not a Gitea cleanup rule.**
+  - Gitea 1.24.3 has **no API** for cleanup rules (none in `routers/api/v1/api.go`; web
+    session+CSRF forms only, in `routers/web/web.go`). So a rule cannot be created, or
+    drift-asserted, as code. **`ci/gitea-settings.sh` was therefore not extended.** A cheaper
+    proxy (count versions per image) would need `read:package` on `SETTINGS_API_TOKEN`, which
+    it lacks.
+  - A built-in rule is also **unsafe here**. Gitea stores a manifest as a file of the TAG
+    version it was pushed under (`routers/api/packages/container/manifest.go`
+    `createPackageAndVersion`), and resolves a by-digest pull from those files
+    (`models/packages/container/search.go`). Deleting the tag a deployed digest was pushed under
+    breaks the `@sha256:` pull, and the rule's "newest N" knows nothing about what git pins. The
+    script keeps every digest git pins.
+- **Auto-merge.** Digest bumps of these images auto-merge on a green gate (the existing
+  `digest` rule). The exception is the conformance-runner bump: `conformance/versions.env` is a
+  sensitive path, so it waits for an administrator's `/approve-override` every time.
+
+## What I verified
+
+**Proved locally, against the exact pinned images:**
+
+- **Rootless buildkitd, configured as the manifest configures it.**
+  - Setup: the pinned `moby/buildkit:v0.34.1-rootless` digest, under rootless podman, with
+    `--cap-drop ALL --cap-add SETUID --cap-add SETGID`, seccomp unconfined and `--read-only`;
+    tmpfs at the same five paths; the ConfigMap's `buildkitd.toml`; mTLS on :1234.
+  - The image's `/etc/subuid` was overridden with `user:1:999` and `user:1001:64535`, because
+    rootless podman maps only 65,537 uids. rootlesskit's real `newuidmap` user-namespace path
+    still ran.
+  - It started, `buildctl debug workers` listed the OCI worker, and `debug workers -v` showed the
+    GC config applied (max 4.29 GB, reserved 1.07 GB).
+- **The securityContext, item by item.** Each requirement was measured by taking it away:
+  - **default seccomp:** daemon starts, every RUN step fails → `Unconfined` is required;
+  - **no `SETUID`/`SETGID`:** rootlesskit dies with
+    `newuidmap … operation not permitted` → both capabilities are required;
+  - **`--read-only` rootfs:** works once the five emptyDirs exist. Its only failure was the
+    probe client's `~/.docker`, which is now an emptyDir;
+  - **minimal caps + Unconfined seccomp:** works, and a RUN step executes.
+- **The full CI path, in the actual CI job image.** Setup: `ghcr.io/catthehacker/ubuntu:act-22.04`
+  running `.gitea/workflows/scripts/images.sh` in verify mode. It downloaded and sha256-checked
+  buildx v0.38.1, connected over mTLS, and bake-built all three `verify` targets from an empty
+  cache. rc 0, 1m32s total:
+  - `TOOLBOX VERIFIED` (42 s);
+  - `AGENT-RUNNER VERIFIED` (26 s): selftest OK, probe "20 refusal(s), 0 failure(s)", and built
+    FROM the in-run toolbox through the `target:` context, with the log showing
+    `[agent-runner-verify toolbox 1/6]` and `base=toolbox-in-this-run`;
+  - `CONFORMANCE-RUNNER VERIFIED` (21 s): all seven refusals;
+  - every stage printed `confirmed: no network`.
+- **mTLS rejects what it should.** Tested with a real BuildKit API call (`buildctl debug workers`):
+  - no client cert → `remote error: tls: certificate required`;
+  - a client cert from another CA → rejected the same way;
+  - the valid cert → accepted.
+
+  Through the driver:
+  - another CA's client cert → rc 1;
+  - the wrong CA for the server → rc 1;
+  - an unreachable host → rc 1 in about 20 s, with the explanatory message;
+  - missing secrets → rc 1, naming the three secrets.
+- **The local podman path**, `build-image.sh` for all three (rc 0).
+  - Negative control: a verify stage expecting a different promtool version printed
+    `DRIFT promtool want 9.9.9` and failed the build.
+  - Negative control: the `--network=none` check fails when the flag is removed.
+- **`buildkit-client-secret.sh`'s sanity checks**, run offline on local certs. They tell apart:
+  - a good client cert;
+  - a server cert (not client-auth);
+  - a mismatched key;
+  - a cert from another CA.
+
+**Measured live, read-only:**
+- node user namespaces, Debian 13.7 and AppArmor (via an Ansible `sysctl`/`cat`);
+- DinD's MASQUERADE rules;
+- the live ClusterPolicies, namespaced Policies, PSA labels, and cert-manager v1.21.3 with its
+  ClusterIssuers;
+- the act-runner pod labels.
+
+**Renovate (unchanged since the first iteration):**
+- the validator with 44.167.0: exit 0;
+- `extract` finds all four markers;
+- the `kubernetes` duplicates are `skipReason: disabled`;
+- 39.264.0 rejects `main`'s config.
+
+**Local gate, CI results:** see the PR body. They are recorded there because this entry is part of
+the tree being gated.
+
+## What I did NOT verify
+
+- **platform/buildkit has never run in the cluster.** It can't until it merges: agents may not
+  `kubectl apply`, and Flux reconciles only `main`. So this PR's own `images` job **cannot build
+  and is expected to fail at the builder preflight**, which is the designed loud failure. The
+  following are therefore all unproven in-cluster:
+  - kubelet admitting the pod;
+  - containerd honouring `appArmorProfile: Unconfined`;
+  - the five emptyDirs;
+  - cert-manager issuing the three certs;
+  - the CiliumNetworkPolicies.
+- **AppArmor `Unconfined` is reasoned, not measured.** My test host runs SELinux, so the AppArmor
+  requirement comes from upstream's example plus the nodes having AppArmor enabled. If the pod
+  were to work under `RuntimeDefault` AppArmor, that line could be tightened; I did not test it.
+- **The local userns mapping is narrower than the cluster's.** Locally it was 65,535 ids across
+  two ranges; in the cluster it is the image's own `100000:65536` against real host subuids. That
+  is the normal case, but it is the cluster's first test.
+- **The NetworkPolicies are untested.** In particular:
+  - that `toEntities: [ingress]` admits the push to the Gateway VIP. The comment explains the
+    reasoning, but it is unmeasured;
+  - that world :443/:80 covers every download the builds make. A dnf mirror on another port
+    would fail a build visibly.
+- **No push, anywhere.** The whole publish path is unexercised:
+  - `REGISTRY_PUSH_TOKEN` reaching buildkitd through the buildx session;
+  - `push=true`;
+  - the metadata digest compared with the registry read-back;
+  - agent-runner FROM `toolbox@sha256:`;
+  - the immutable-tag guard;
+  - idempotence.
+
+  So are the 15 s Envoy route-timeout risk on blob finalisation, and the Gitea PVC (10Gi, shared
+  with repositories) absorbing ~2 GB per toolbox publish until retention runs.
+- **Node disk under real builds.** The cache emptyDir (6Gi limit, 4 GB GC cap) sits on nodes that
+  had 7–9 GB free. It is untested under concurrent pressure.
+- **Certificate rotation is designed, not exercised.** The server cert renews 30 days early but
+  buildkitd only reads it at start; the client copy needs the script re-run.
+- **`--oci-worker-no-process-sandbox` is an accepted risk, not a mitigated one.** A build step can
+  ptrace the daemon and read its TLS key and cache. Anyone with the client cert, which means every
+  PR job, can run build steps.
+- **Renovate has still never completed a run.** It is blocked on #80, on `RENOVATE_TOKEN` gaining
+  `read:package`, and on this branch. The `[email protected]` + 44.x image pairing is
+  untested.
+- **Pods cannot pull these images until PR #61** (the kubelet pull secret) merges.
+- **`registry-retention.sh --apply` has never run**; its keep logic has only run against an empty
+  registry.

decisions/2026-10-29-keycloak-passkey-reconcile.md

@@ -0,0 +1,172 @@
+---
+date: 2026-10-29
+branch: feat/keycloak-passkey-reconcile
+title: "Passkey reconciler review fixes: bind only on git's say-so (rollback sticks), protect the real 26.0.6 sub-flows, screen the requirement PUT by body id, alert on failure"
+---
+
+## What changed
+
+This branch carries the unmerged passkey reconciler (commit `36fcb7b`, cherry-picked with its
+authorship, recorded in `decisions/2026-10-18-keycloak-passkey-reconciler.md`) plus the fixes for a
+review of it. Read that entry for the design; this one records what the review found and what
+changed.
+
+**1. HIGH: the bind was automatic, and rollback did not stick.** Within one tick of merge the
+reconciler would build and BIND `browser-mfa-<hash>`, a login flow no human had tried. Worse, the
+README's rollback (rebind `browser-mfa`) was undone every 15 minutes: the next run found the
+verified candidate again and rebound it, and a hotfix inside the bound flow was "corrected" by
+building and binding an `-r2`.
+
+- New git-declared switch: `bind-browser-flow` in `platform/keycloak/passkey-desired-state.yaml`,
+  a second key of the same ConfigMap, so it reaches the pod through the existing mount. **Default
+  `"false"`.** It is the one hand-edited line in a generated file: the renderer carries it through
+  on `--write`, and `--check` (run by the L0 gate) refuses anything but the quoted strings `"true"`
+  / `"false"`. An unquoted `true` is a YAML boolean, which a ConfigMap rejects at apply time and
+  which would have failed the whole `platform-config` Kustomization, realm import included.
+- `"false"`: pass 3 builds and verifies the candidate (or reuses a verified one), logs
+  `WOULD BIND <alias>`, exits 0 with `KEYCLOAK_PASSKEY_RECONCILED_OK_PENDING_BIND`, and never PUTs
+  a changed `browserFlow`. That holds after an earlier bind too, so the operator's rebind sticks.
+  If the bound flow *is* the candidate alias and has been hand-edited, it is left alone (no `-r2`
+  is built beside it).
+- `"true"`: binds as before, and keeps asserting it. That is GitOps working; the README says so,
+  and says not to hotfix the flow while it is `"true"`.
+- The CronJob passes no flag. `--no-bind` still exists for a hand-run Job and can only narrow
+  git's value. The L0 check asserts the CronJob carries no `--` argument.
+- Rollback is now: a PR setting `"false"` (first), then one rebind write. README documents the
+  order and why it matters, plus how to test the candidate on one client (a per-client
+  `account-console` browser-flow override) before the go-live PR.
+
+**2. HIGH: the protected-subtree guard read the wrong field for sub-flows.** Read from source at tag
+26.0.6 (`services/src/main/java/org/keycloak/services/resources/admin/AuthenticationManagementResource.java`,
+fetched from GitHub this session): `recurseExecutions` sets `displayName = flowRef.getAlias()` and
+`flowId` on a sub-flow row and never calls `setAlias`. `alias` is set only on authenticator rows,
+where it holds the bound **config's** alias. So the protected set was `{bound}` alone, and a write
+into `browser-mfa-forms` would not have been refused. The old stub returned `alias` on sub-flow
+rows, which is why all 216 assertions passed anyway.
+
+- New `subflow_alias()` reads `displayName` and cites the source. I did not resolve by `flowId`:
+  `getFlows()` (same file) filters to `isTopLevel()`, so a nested sub-flow's id resolves to nothing
+  through the flows list. A sub-flow row with no `displayName` makes the listing unreadable, which
+  leaves the flow pass fail-closed.
+- The stub now returns rows shaped like 26.0.6's: sub-flow rows carry `displayName`, `flowId` and no
+  `alias`/`providerId`. Authenticator rows carry a display *type* as `displayName` (not the
+  provider id) and the config alias as `alias`. Nulls are omitted.
+- New assertions: after a real run, `browser-mfa-forms` and `browser-mfa-conditional-otp` are in
+  the protected set, and an execution add or requirement write into them is refused. A missing
+  `displayName` fails closed.
+
+**3. MEDIUM: `PUT /flows/{alias}/executions` was screened on the wrong field.** `updateExecutions`
+resolves `realm.getAuthenticationExecutionById(rep.getId())` without checking membership in
+`{alias}`. The same handler also renames the flow named by `rep.getFlowId()` when the body carries
+`authenticationFlow` + `displayName`. Now:
+
+- `body["id"]` is screened against the protected execution ids.
+- It must also be in `_BUILD_EXECUTION_IDS`: the ids read back from the flow this call just
+  created, open only inside `build_flow`.
+- The config `POST /executions/{id}/config` is held to the same build scope.
+- The body may carry only `id`, `requirement` and `priority`.
+- Selftest cases cover each of these.
+
+**4. Stale docs on sensitive paths** now describe what ships, including the switch:
+`platform-config/kustomization.yaml`, the chainsaw test header, the desired-state header and its
+renderer template, the renderer's own header and `--check` message, and the README.
+
+**5.**
+- `backoffLimit: 0`: every failure is deterministic within a tick, and a retried build burns
+  another of the 5 candidate aliases.
+- New alert `KeycloakPasskeyReconcilerFailing` in `platform/monitoring/alerts.yaml` (warning,
+  `for: 15m`, `kube_job_status_failed` on `job_name=~"keycloak-passkey-reconciler-.*"`), the same
+  shape as the sibling's. It has promtool tests in
+  `platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml`, in both directions plus
+  "the sibling's failure is not this alert" and "one transient failure stays quiet". The alert
+  ratchet in `alert-rules-unit-tested` would have refused it untested. A new runbook,
+  `runbooks/keycloak-passkey-reconciler.md`, is projected by the runbooks generator.
+- The stale "Renovate does not manage plain `image:` tags" comment is corrected: `renovate.json5`
+  enables the `kubernetes` manager over `platform/**`. The image is pinned to
+  `python:3.13.15-alpine3.23@sha256:6438599575cca0d1df94aeee0d2ae088d4d8846eab554b2ee7784a3a6df0d516`,
+  the multi-arch index digest read from Docker Hub today. The tag is mutable: the sibling
+  reconciler's pod on prod reports `imageID ...@sha256:75f27d68...` under the same tag, which is
+  neither today's index nor today's amd64 manifest.
+
+**6. Hygiene.** The ~160-line header that appeared in both the YAML and the script is now kept once,
+in the script, where it ships next to the code and its selftest. A short header above the
+ConfigMap points to it. Nothing else was restructured, apart from the build's second half moving
+into `_finish_build()` so the build scope can be opened and closed around it.
+
+Sensitive paths touched: `ci/**` (renderer), `conformance/**` (L0 check, mutations, chainsaw),
+`platform-config/kustomization.yaml` (comment only). This needs the operator's
+`/approve-override` or a non-author review. I did not post one.
+
+## What I verified
+
+- **Local toolbox gate, rc 3** (pinned toolbox via `dev.sh`, `DEV_KUBECONFIG=/dev/null`, ENGINE
+  shim bind-mounting the main `.git`, run on the committed branch). Stages: `spec-immutability`
+  pass (the spec repo resolved as a sibling), `decisions-log` pass, `lint` pass, `l0` pass
+  (including `keycloak-passkey-reconciler`, `alert-rules-unit-tested` and
+  `alert-runbooks-resolvable`), `render-golden` pass, `l1` pass, `approval-policy` pass (N/A
+  offline; it is guaranteed to fail in CI). **`gitea-settings` did not run** (no
+  `SETTINGS_API_TOKEN` locally) and is the only stage that did not run. L2/L3 `conformance` is
+  skipped by design (no cluster). rc 3 is **not** green.
+- `platform/monitoring/tests/run.sh platform/monitoring/tests/keycloak-passkey-reconciler.test.yaml`
+  in the toolbox: `promtool check rules` SUCCESS (54 rules), `promtool test rules` SUCCESS,
+  promtool 3.14.0.
+
+- **Selftest: 259 assertions pass** (was 216), extracted from the shipped ConfigMap by the L0
+  check.
+- **`mutations.py`: 40 of 40 injected regressions caught** (was 30). The ten new ones each undo
+  one fix:
+  - sub-flows read by `alias` again;
+  - an unnameable sub-flow tolerated;
+  - the requirement PUT screened on its path only;
+  - execution writes not held to the build tree;
+  - extra body keys allowed;
+  - the bind switch ignored in `run()`;
+  - the bind switch ignored in `reconcile_flow()`;
+  - an absent switch reading as true;
+  - an invalid switch guessed at;
+  - `-r2` built beside a hand-edited bound flow.
+  "Sub-flows by `alias`" is caught less cleanly (the selftest aborts on an unreadable listing
+  rather than failing an assertion). It is still caught.
+- The renderer accepts `"true"` and `"false"` and refuses `true` (unquoted) and `"yes"`. I
+  checked this by hand-editing the committed file and running `--check`.
+- The 26.0.6 source for `recurseExecutions`, `updateExecutions` and `getFlows` was read directly
+  (lines ~225-231, 658-724 and 747-806 of the file at that tag).
+- Read-only `kubectl --context k3s-prod`: Keycloak pods run `quay.io/keycloak/keycloak:26.0.6`
+  (2 replicas). The only CronJob in `keycloak` is `keycloak-client-secret-reconciler`, so the
+  passkey reconciler is **not** live.
+
+## What I did NOT verify
+
+- **Nothing has run against a real Keycloak.** The stub's row shapes now follow the 26.0.6
+  source, but they are still my reading of that source, not a captured response. In particular,
+  I have never seen a live executions listing to confirm `displayName` on a sub-flow row. If it
+  differs, the failure is fail-closed (no build), not fail-open.
+- **The candidate flow has never been tested in a browser.** That is now a precondition of the
+  go-live PR (README, "Testing the candidate flow before binding it"). Nobody has run those steps.
+  I also have not checked that the per-client `account-console` override behaves as described on
+  26.0.6.
+- **"Merging changes nothing users see" is true of login, but passes 1 and 2 still write the
+  realm.** They set the passwordless policy and enable the enrolment action. That action is
+  enabled by default in 26.0.6, but I did not read the live realm, because no admin API calls were
+  allowed. Pass 3 also creates the unbound candidate flow (realm DB writes, invisible to login).
+- **Race in pass 1, not fixed:** the policy PUT is a read-modify-write that carries `browserFlow`
+  back from its own GET. An operator rebind landing between that GET and PUT would be reverted.
+  The window is one HTTP round trip and only exists when the policy has drifted.
+- **The alert has never seen a real series.** The CronJob has never run, so
+  `kube_job_status_failed{job_name=~"keycloak-passkey-reconciler-.*"}` does not exist yet. Its
+  shape is copied from the sibling's rule, which does have live series.
+- **The digest is of today's index.** I did not pull it or run the image. Renovate's handling of a
+  `tag@digest` pin under the `kubernetes` manager is from its documentation, not from a Renovate
+  run here.
+- **CI's own result** is in the PR, not here. `approval-policy` is expected to fail until the
+  operator overrides.
+
+### Follow-ups (out of scope here, deliberately not done)
+
+- A dedicated least-privilege service-account client for the reconciler, instead of the
+  master-realm break-glass admin it shares with the client-secret reconciler.
+- A NetworkPolicy for the `keycloak` namespace.
+- Whether a passkey login should skip TOTP for platform-admins. This is the PR #72 design
+  decision, currently "no TOTP after a passkey".
+- A `KeycloakPasskeyReconcilerStale` rule (no successful run in N hours), like the sibling's.
+  Only the failure alert was in scope.

decisions/2026-10-29-renovate-config-filename.md

@@ -0,0 +1,57 @@
+---
+date: 2026-10-29
+branch: fix/renovate-config-filename
+title: Renovate has failed every run since #79 — the workflow still named renovate.json
+---
+
+## What changed
+
+**`.gitea/workflows/renovate.yaml`** — `configurationFile: renovate.json` → `renovate.json5`.
+
+PR #79 (`fix/renovate-invalid-config`) renamed the config to `renovate.json5` and did not update
+the one line in the workflow that names it. Measured 2026-10-29 from the Gitea Actions log of run
+322 (task 513, schedule, 08:15 UTC):
+
+    Error: ENOENT: no such file or directory, stat '/workspace/Home-Network/home-network-k3s-impl/renovate.json'
+
+Runs 315–322 (2026-10-22 → 10-29, every one listed) are all `failure` on the `schedule` event,
+each failing in about one second. Renovate never started. `RENOVATE_TOKEN` **does** exist as a
+repository Actions secret (listed by name via the API; value not read), so the token preflight
+passes and this filename is the only blocker observed.
+
+**`ci/lint.sh`** — a new assertion after yamllint: every non-comment `configurationFile:` key in
+`.gitea/workflows/*.y*ml` must name a path that exists. This is valid YAML, so yamllint cannot see
+it, and the gate stayed green through 11 days of failures.
+
+**Second blocker, same PR:** the workflow did not pin a Renovate version, so
+`renovatebot/[email protected]` ran its default, Renovate **39** (`action.yml`:
+`renovate-version` `default: '39'`, read at that tag). `renovate.json5` uses `managerFilePatterns`,
+which needs >= 40.2.0. Under 39.264.0 the validator rejects the config with five errors. This was
+found and reproduced by the image-pipeline worker (PR #81). #79 had validated against `latest`. Pinned
+`renovate-version: 44.167.0`, the version the config validates clean against per that worker.
+
+## What I verified
+
+- The root cause, from the run log quoted above, and that `origin/main` has `renovate.json5` and
+  no `renovate.json` (`git ls-tree`).
+- The new guard, both ways, run standalone: on this branch it prints
+  `workflow configurationFile references resolve`, rc 0. With the line reverted to `renovate.json`
+  it prints `configurationFile 'renovate.json' does not exist in the repo`, rc 1. My first version
+  also matched its own explanatory comment in the workflow and failed. That is why the grep is
+  anchored to non-comment lines.
+- Toolbox gate: result and rc are reported in the PR, not predicted here.
+
+## What I did NOT verify
+
+- **The 39.x rejection and the 44.167.0 pass were not reproduced by me.** I confirmed only the
+  action's default (`'39'`). The validation results are the PR #81 worker's.
+- **Renovate has still never completed a run.** This removes the observed blocker. Others may
+  follow it. The test is a `workflow_dispatch` with `dry_run: true` after merge, and its log is
+  the first evidence that Renovate authenticates, extracts dependencies, and honours `.json5`.
+- **The `enabledManagers: ["regex"]` vs `custom.regex` question** (#73, #77) is still open. The
+  same dry-run log answers it: look for the regex/custom manager's dependency-extraction count.
+- **No alert for "Renovate keeps failing".** The guard prevents this specific cause from recurring.
+  Nothing notices a scheduled workflow that is red every day. That gap stays open.
+- **Sensitive path.** `.gitea/**` and `ci/**` are both in `approval-policy.json`. CI will block
+  on `approval-policy` until the operator posts `/approve-override` as a review. The local gate
+  reports that stage as N/A.

decisions/2026-10-30-gitea-admit-buildkit.md

@@ -0,0 +1,81 @@
+---
+date: 2026-10-30
+branch: fix/gitea-admit-buildkit
+title: The registry push 403 is Cilium refusing buildkitd's EGRESS at the Gateway — fixed in platform/buildkit, not in Gitea's ingress
+---
+
+## What changed
+
+**`platform/buildkit/networkpolicy.yaml`** — `buildkitd-allow-egress` gains one rule:
+`toEndpoints {io.kubernetes.pod.namespace: gitea, app.kubernetes.io/name: gitea}` on TCP 3000,
+with a comment carrying the measurement. `platform/buildkit/README.md`'s policy row says the same.
+
+**What did NOT change, deliberately:** `catalog/gitea.yaml` `inClusterClients` and
+`apps/wave-0/gitea.yaml`. The brief proposed admitting `buildkit` there. The evidence below says
+that is the wrong policy: Gitea's ingress is not what refuses the push, and adding it would only
+open the direct ClusterIP path from build code to Gitea, which nothing uses.
+
+### Why: the brief's hypothesis was half right
+
+Images run 337 failed its push with `HEAD .../blobs/sha256:5855... 403 Forbidden: Access denied`.
+The PM's hypothesis was that this was Cilium's Envoy denying the request, not Gitea. **That part
+holds.** The hypothesis also said the denial came from Gitea's *ingress* policy, which does not
+admit the `buildkit` namespace. **That part is wrong.** The denial is enforced on buildkitd's own
+*egress*. Since Cilium 1.16.8/1.17.2 (CVE-2025-30162; prod runs `cilium:v1.20.1`), the Gateway's
+L7 load balancer re-evaluates the source pod's egress policy against the identity of the backend
+it resolves to. buildkitd's egress allowed `world` :443/:80 and the `ingress` entity. That gets
+the TCP connection to Envoy, but no rule matches Gitea's backend identity, so Envoy answers
+`403 Access denied` without forwarding the request. This is the failure commit `68690e5` fixed for
+every SSO app's hop to Keycloak, and the fix has the same shape. Keycloak's ingress names no app
+namespaces either.
+
+## What I verified
+
+All measurements are read-only, against `k3s-prod`, on 2026-10-30 around 05:27–05:37 UTC.
+
+- **Gitea never saw the failed push.** Its router log for the previous 12h had 316 `/v2/` lines
+  and **0** that were `toolbox/blobs`, PUT, POST or PATCH. It also had **0** from buildkitd's pod
+  IP (`10.0.0.7`). In the same window it logged the CI job's own `HEAD .../toolbox/manifests/
+  sha-0dd57fbaf002 → 404` from the act-runner pod (`10.0.0.38`). Requests that came through the
+  Gateway reach Gitea and get logged. This one never arrived.
+- **A reproduction with no credentials.** An unauthenticated `GET https://gitea.home.keithjackson.dev/v2/`
+  (Gateway VIP `192.168.2.20`) gave:
+  - from the buildkitd pod: **`403 Forbidden`**, with no matching line in Gitea's log;
+  - from the act-runner pod's `dind` container: **`401 Unauthorized`**, which is Gitea's normal
+    registry challenge, logged by Gitea as `GET /v2/ for 10.0.0.38:0, 401`.
+
+  No token was involved, so this cannot be a package-permission problem. The bot account and the
+  `REGISTRY_PUSH_TOKEN` were not the cause, and the operator does not need to change them.
+- **Hubble names the enforcement point.** On `cilium-55jv8`, buildkitd's node:
+  `buildkit/buildkitd-859d67c8f8-x44nh (ingress) -> gateway/cilium-gateway-shared:443 http-request
+  DROPPED (HTTP/1.1 GET https://gitea.home.keithjackson.dev/v2/)`, followed by `http-response
+  FORWARDED (HTTP/1.1 403 0ms)`. In the JSON form, **`traffic_direction: EGRESS`**. Envoy wrote
+  the 403 itself in 0 ms.
+- **Gitea's ingress does not need the source namespace.** The act-runner pod's namespace
+  (`gitea-actions`) is **not** in Gitea's ingress allows. Its NetworkPolicy
+  `default-deny-ingress-allow-egress` has an `{}` egress rule, so its egress is unrestricted. Its
+  requests through the Gateway still reach Gitea. `cilium-dbg bpf policy get` on Gitea's endpoint
+  (1876 on `cilium-9rk5x`) shows all Gateway traffic admitted as `reserved:ingress 3000/TCP`
+  (391 MB, 1.19 M packets) and no row for `gitea-actions`. The source-identity check therefore
+  happens on the source's egress, not on the destination's ingress.
+- **The policy gap is visible in the BPF map.** `bpf policy get 83` (buildkitd) has these egress
+  rows and nothing else: `reserved:world` 80/443, `reserved:ingress ANY`, and kube-dns 53. There
+  is no Gitea identity.
+- The edited YAML parses (three documents, names unchanged).
+
+## What I did NOT verify
+
+- **That this rule turns the 403 into a 401/200. UNVERIFIED.** That needs the policy applied, which
+  is a Flux/operator action. My confidence is high but not proven, and it rests on two things. The
+  Hubble verdict is EGRESS. Commit `68690e5` made the identical change for Keycloak and recorded
+  "Verified live: headlamp-ns -> keycloak discovery 403 -> 200".
+- **The port restriction.** The rule names `3000`, Gitea's container port, which is the upstream
+  port Envoy connects to. The Keycloak precedent has no port. If the 403 survives, the first thing
+  to test is whether L7LB enforcement uses the frontend :443 instead. I expect it does not.
+- **Run 337's own flows.** The Hubble ring buffer had already rotated past them, and buildkitd had
+  been restarted since (the pod was 18 minutes old). The reproduction above uses the same path, the
+  same pod spec and the same policy, but it is not the original request.
+- Envoy's access logs in the `cilium-envoy` pods. Hubble's L7 flow already shows the drop, so I
+  did not read them.
+- The push itself, the Renovate digest flow, and whether the bot team's `Packages=Write` is enough
+  for a first push that creates a new package. These only become testable after this lands.

decisions/2026-10-30-pin-pihole-jackett-plex.md

@@ -0,0 +1,96 @@
+---
+date: 2026-10-30
+branch: feat/pin-pihole-jackett-plex
+title: Pin pihole, jackett and plex to the digests already running, and make Renovate able to follow them
+---
+
+## What changed
+
+- **`catalog/{pihole,jackett,plex}.yaml` `source.image`** pinned to the tag whose digest was
+  already running, so the *image* does not change:
+  - `pihole/pihole` (no tag) -> `pihole/pihole:2026.10.0`
+  - `lscr.io/linuxserver/jackett:latest` -> `lscr.io/linuxserver/jackett:v0.24.2865-ls7`
+  - `plexinc/pms-docker:latest` -> `plexinc/pms-docker:1.43.4.10903-e5521bd8c`
+- **`apps/wave-0/{pihole,jackett,plex}.yaml`** re-rendered (`render.py --all catalog -o apps
+  --wave-layout --no-image-automation`, in the toolbox). The diff is exactly the three app-container
+  `image:` lines; nothing else moved.
+- **`catalog/jackett.yaml`** `update.range` comment: "stay on the Jackett 0.22.x stable line" was
+  false (0.24 runs, and `>=0.22.0` has no ceiling). Range unchanged; comment corrected.
+- **`renovate.json5`**, four packageRules plus a comment block:
+  - jackett `versioning: regex:^v(major).(minor).(patch)-ls(build)$` (Renovate's linuxserver example).
+  - jackett **`overridePackageName: ghcr.io/linuxserver/jackett`**: not in the brief, and needed.
+    Renovate's docker datasource reads at most `dockerMaxPages` (20, a global-only option) tag
+    pages from any host except a short list that includes ghcr.io (1000 pages). lscr.io is not on
+    that list. Jackett has **32,093** tags, oldest first, ~1000 per page; the running tag is #31,980.
+    Before this rule the dry run extracted jackett at `v0.24.2865-ls7` and **proposed nothing**,
+    with no warning. The override changes only the lookup; the manifests keep `lscr.io/...`.
+    Every jackett rule matches on `depName`, because the override rewrites `packageName`.
+  - jackett `schedule: after 3am and before 6am on sunday`, i.e. weekly. Measured: 13 ls builds in
+    12 days (ls7 built 2026-10-17, ls20 built 2026-10-29). Each is a `patch` and auto-merges, so on
+    the daily window it would mean a PR, a gate run and a jackett restart (Recreate) every day.
+    Weekly picks the newest tag, so it skips nothing. `catalog/jackett.yaml update.cadence` still
+    says daily. That field fed the retired Flux image-automation, and the rule's description
+    says the rule is now the cadence that takes effect.
+  - plex `versioning: regex:^(major).(minor).(patch).(build)-[0-9a-f]+$`.
+  - pihole: **no rule**, as the operator decided. CalVer parses under default docker versioning.
+    Patches and minors auto-merge, and a year rollover is a major (dashboard approval). This is
+    written in a comment.
+  - Corrected the "WHAT RENOVATE STILL CANNOT FIX" comment. It listed these three as floating;
+    they publish immutable version tags.
+  - No `allowedVersions` mirror was added. None of the three catalog ranges has a ceiling
+    (`>=2024.0.0`, `>=0.22.0`, `>=1.40.0`), so #86's mirror has nothing to encode for them.
+- **Runbooks:** `image-policy-stalled.md` no longer says jackett and pms-docker are
+  `latest`-only. In `plex-quicksync.md`, "the floating-image caveat" became "the image-version
+  caveat", because the deployment no longer runs `:latest` with an implicit `Always`, and the symptom
+  is now "after a pms-docker bump". `plex-claim.md`'s two "running `:latest`" mentions are
+  now in the past tense.
+
+## What I verified
+
+- **Live digests** (read-only `kubectl --context k3s-prod`, pod `.status.containerStatuses[].imageID`):
+  pihole-0 and pihole-1 `sha256:d34134716fc3…`, jackett `sha256:4fcf3cc22dd3…`, plex
+  `sha256:e0ab27395614…`.
+- **Each chosen tag resolves to the same digest** (anonymous registry HEAD, OCI index + manifest-list
+  Accept): `pihole/pihole:2026.10.0` = `sha256:d34134716fc3…`; `ghcr.io/linuxserver/jackett:v0.24.2865-ls7`
+  = `sha256:4fcf3cc22dd3…`; `plexinc/pms-docker:1.43.4.10903-e5521bd8c` = `sha256:e0ab27395614…`.
+  (For contrast, `pms-docker:latest` is now `sha256:6d778d41…` and `jackett:latest` `sha256:ee6ae915…`.
+  The pods would have drifted on their next restart.)
+- `renovate-config-validator` (ghcr.io/renovatebot/renovate:44.167.0) reports "Config validated
+  successfully". The only warning is the existing `docker` block migration.
+- **Renovate dry run** (`RENOVATE_PLATFORM=local`, `--dry-run=full`, 44.167.0, anonymous, includePaths
+  limited to the six files). It extracted all three from both `catalog/` (custom regex) and
+  `apps/wave-0/` (kubernetes) under the same depName, so each lands in one branch:
+  - plex `1.43.4.10903-e5521bd8c` -> `1.43.5.11079-b3b13e506` (patch, `renovate/plexinc-pms-docker-1.x`)
+  - jackett `v0.24.2865-ls7` -> `v0.24.2961-ls20` (patch, `renovate/lscr.io-linuxserver-jackett-0.x`).
+    This needs the ghcr override; without it there was no proposal.
+  - pihole `2026.10.0` -> none. That is correct, because 2026.10.0 is the newest numbered tag on Docker Hub.
+- **Automerge, schedule and dashboard flags** come from Renovate's own `applyPackageRules`
+  (datasource-merge, then update-type-merge, the same order as `flatten.js`), run on the resolved
+  config. The local platform does not run the branch worker:
+  - plex patch: automerge **true**, daily window.
+  - jackett patch: automerge **true**, schedule **sunday** window.
+  - pihole (synthetic updates) `2026.10.1` patch and `2026.11.0` minor: automerge **true**. `2027.01.0`
+    major: automerge **false**, `dependencyDashboardApproval` **true**, +needs-review/breaking.
+- **Versioning filters** (Renovate's versioning module): the jackett regex rejects `latest`, bare
+  `0.24.2961`, `amd64-…` and `development-…`. The plex regex rejects `latest`, `public`, `beta`,
+  `plexpass` and `…-amd64`/`…-arm64`. Docker versioning ranks `2026.09.0 < 2026.10.0`.
+- Touched paths (`catalog/`, `apps/`, `renovate.json5`, `runbooks/`, `decisions/`) are not in
+  `infra/gitea/approval-policy.json` `sensitive_paths`.
+
+## What I did NOT verify
+
+- **That the merge is restart-free. It is not.** The pod-template `image` string changes, so Flux
+  rolls all three: plex and jackett are `Recreate`, so each has a brief outage, and pihole is a
+  2-replica StatefulSet that rolls one pod at a time. The digest is identical, so the image's
+  behaviour should not change, but the kubelet resolves the new tag name and may pull it. The
+  rollout itself was not observed.
+- The implicit `imagePullPolicy` moves from `Always` (`:latest` or no tag) to `IfNotPresent`. This is
+  intended, and was not observed live.
+- A real (non-dry) Renovate run against Gitea, and whether the Sunday window actually opens a jackett
+  PR. The schedule was only evaluated, not exercised.
+- Other `lscr.io/linuxserver/*` images (radarr, sonarr, qbittorrent) may hit the same 20-page
+  truncation once their tag counts grow. Not measured here, and not changed.
+- Plex not self-updating on a version tag was checked only against the running pod (read-only
+  `kubectl exec cat`). `/etc/cont-init.d/50-plex-update` does `exit 0` when `version` from
+  `/version.txt` equals the installed `plexmediaserver` package. Both are
+  `1.43.4.10903-e5521bd8c`. I did not boot a pod on a *new* tag to watch that short-circuit.

decisions/2026-10-30-pm-agent-live.md

@@ -0,0 +1,101 @@
+---
+date: 2026-10-30
+branch: feat/pm-agent-registry-pull-secret
+title: PM agent — #61 rebased onto the CI image pipeline, first published digest pinned, and the remaining path to live audited
+---
+
+## What changed
+
+PR #61 (`feat/pm-agent-registry-pull-secret`, the kubelet pull credential and admission rule
+7) was rebased onto `origin/main` after #81 (CI image pipeline), #82 and #83. Conflicts in
+`docs/plans/pm-agent-issue-to-pr.md`, `platform/pm-agent/README.md` and
+`runbooks/pm-agent.md` were resolved keeping both sides: #61's "pull credential now wired"
+and #81's "CI publishes, Renovate bumps". No manual build-and-push text came back.
+
+On top of the rebase, as a second commit:
+
+- **The first published digest is pinned**: `deployment.yaml` and `PM_WORKER_IMAGE` in
+  `configmap.yaml` now name
+  `gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42`
+  (images run 340, tag `sha-523cd9118f95`, which is also `main`). The `# renovate-digest:`
+  markers are unchanged and still sit on the line directly above each value, so later bumps
+  stay Renovate's. Pinning by hand once removes the dependency on Renovate's first-ever run.
+- **The first run is dry, and scoped to a label**: `PM_DRY_RUN: "true"`,
+  `PM_REQUIRE_LABEL: "agent"` (the plan's follow-up 3); `config-generation` bumped to `2`.
+  Dry run is also a *correctness* requirement, not only caution: the pinned digest was
+  built before this PR added `imagePullSecrets` to the worker Job template, which is baked
+  into the image. Admission now requires that entry, so a worker Job built from the run-340
+  image would be denied. Merging this PR touches `infra/agent/**`, so `images.yaml`
+  re-publishes agent-runner; Renovate's bump to that digest is what makes workers
+  admissible. Only then flip `PM_DRY_RUN` to `false`.
+- **The token scope list was wrong; it is fixed.** Gitea 1.24.3's `routers/api/v1/api.go`
+  closes the `/user` route group with `tokenRequiresScopes(AccessTokenScopeCategoryUser)`,
+  and the dispatcher's first call is `GET /user`. The documented scopes are now
+  `read:user`, `write:repository`, `write:issue`, `read:package` everywhere (both
+  `.example`s, README, runbook row).
+- **Org-package permission is documented separately from the scope.** Per
+  `services/context/package.go` `determineAccessMode` (v1.24.3), a non-admin's access to an
+  org-owned package is the highest Packages-unit mode across its org teams, else read if the
+  org is visible to it. Home-Network is `public`, so it works today by visibility alone; the
+  registry `.example` now has the operator create a `pm-agent-pull` team (no repositories,
+  Packages = Read) so it does not depend on that.
+- **`sops` is toolbox-only**: both `.example`s now say so and give the in-place
+  `./infra/devcontainer/dev.sh run sops <file>` edit (the same procedure
+  `runbooks/vpn-remote-access.md` documents), which writes no plaintext file.
+- The runbook's Secret-key check used `-o jsonpath='{.data}'`, which prints values before the
+  pipe trims them; it is now a `go-template` that emits key names only. Two new runbook
+  rows: the `read:user` 403, and the admission denial from a pre-#61 digest.
+
+## What I verified
+
+- **Registry digest**, read-only from the registry API (bearer token from `/v2/token`,
+  `HEAD /v2/home-network/agent-runner/manifests/<tag>` with OCI and Docker accept headers):
+  `main` and `sha-523cd9118f95` both answer `200`,
+  `Docker-Content-Digest: sha256:c4eba5b3…6eb42`, `application/vnd.oci.image.manifest.v1+json`.
+  Tags list: `[main, sha-523cd9118f95]`. The admission pattern
+  `gitea.home.keithjackson.dev/home-network/agent-runner@sha256:*` matches the pinned value.
+- **#81's claim that the template's digest needs no tracking holds**: `pm-dispatch.build_job`
+  sets `containers[0].image = cfg.worker_image` (from `PM_WORKER_IMAGE`) on every Job.
+  Renovate's `renovate-digest` custom manager only matches `platform/pm-agent/*.yaml` and
+  `versions.env`, so the template is not touched by it.
+- **Node path to the registry, read-only over SSH, all three prod nodes**: `/etc/hosts` line 9
+  is `192.168.2.20 keycloak.home.keithjackson.dev gitea.home.keithjackson.dev`, nsswitch is
+  `hosts: files dns`, and `curl https://gitea.home.keithjackson.dev/v2/` from the node answers
+  `401` with `ssl_verify_result=0` and `remote_ip=192.168.2.20`. That is the host resolver,
+  which is what containerd (a node process) uses; a pod's view would not have shown it.
+- **Network path from the namespace**: dispatcher and worker both use
+  `PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:3000` (ConfigMap, Job template, and
+  `pm-worker`'s hard allowlist, which refuses anything else); the git remote and the API base
+  are both derived from it. So nothing goes through the Gateway and #83's
+  `toEndpoints`-through-Gateway issue does not apply. Egress: `toEndpoints` gitea pods `:3000`
+  (both CNPs), `kube-apiserver` (dispatcher only), `world:443` (the model). Ingress on the
+  Gitea side: `gitea-allow-pm-agent-ingress` exists live and is `VALID True`.
+- **Live state (read-only, `--context k3s-prod`)**: namespace `pm-agent` exists; Deployment
+  `0/1`, pod `ImagePullBackOff` for 24d on the all-zero digest; Secret `pm-agent-credentials`
+  (2 keys) exists, `pm-agent-registry` does not yet (it ships in this PR); Flux Kustomization
+  `pm-agent` Ready at `main@dfc432c`; ClusterPolicy `pm-agent-job-shape` Ready.
+- **Gitea state (read-only API)**: 1.24.3; org Home-Network `public`; no user `pm-agent`
+  exists; teams are `Owners` and `registry-push`; collaborators are `jakman85` (owner) and
+  `renovate-bot` (write); the repository has **no labels** and **no open issues**.
+- **Placeholders, without decrypting**: the registry ciphertext decodes to 155 bytes, exactly
+  the length of the `.example`'s placeholder JSON. The credentials values decode to 57 and 39
+  bytes — neither a 40-char Gitea token nor a ~100-char Claude OAuth token. Both files'
+  `lastmodified` is the commit that created them as placeholders.
+- `kyverno test`, `PM_SELFTEST=1 pm-dispatch` and the gate: numbers in the PR description and
+  the final handoff report for this run.
+
+## What I did NOT verify
+
+- **Any authenticated pull, poll or worker.** No `pm-agent` account or token exists, so the
+  scope set has not been exercised end to end. The `read:user` requirement is read from
+  Gitea's v1.24.3 source, not from a live 403.
+- **That the credentials ciphertext is a placeholder.** Lengths are strong evidence, not
+  proof; I did not decrypt.
+- **That Renovate will open the post-merge bump.** Its token was being re-minted at the time
+  of writing and it has never completed a run here. Renovate's `kubernetes` manager also
+  scans `infra/agent/*.yaml` and `platform/**`; what it does with a digest-only reference
+  with no tag (the template) is unmeasured.
+- **The dispatcher's `claude --print` call to `api.anthropic.com`** through `world:443`, and
+  whether the run-340 image's dispatcher behaves as this tree's code does (it predates #61's
+  selftest line, nothing else in `pm-dispatch.py`).
+- **The admission policy against a real Job.** Still never admitted or refused one live.

decisions/2026-10-30-renovate-auth.md

@@ -0,0 +1,108 @@
+---
+date: 2026-10-30
+branch: fix/renovate-auth
+title: Renovate's token needs five scopes, not three; the workflow documented the wrong list, and now probes it
+---
+
+## What changed
+
+**The fix is operator-side.** `RENOVATE_TOKEN` lacks `read:organization`, and it will next lack
+`write:issue`. The operator must re-mint it with five scopes:
+
+- `read:user`
+- `read:organization`
+- `write:repository`
+- `write:issue`
+- `read:package`
+
+The git half fixes the two things that sent the operator round this loop twice.
+
+**`.gitea/workflows/renovate.yaml`**:
+- **The header's scope list is corrected**, with the call each scope serves. The old list named
+  only `write:repository`, `read:user` and `read:package`, and the operator minted exactly that.
+  The preflight's "TO FIX" text is corrected the same way.
+- **A new step, "Preflight - the token has every scope Renovate needs".**
+  - It probes read-only GETs: `/user` for read:user, `/orgs/Home-Network` for read:organization,
+    `/repos/{repo}` for read:repository, `/repos/{repo}/labels` for read:issue, and registry
+    `/v2/token` for read:package.
+  - It reads the token's own scope list out of Gitea's 403 from `/admin/orgs` (names only), and
+    requires `write:repository` and `write:issue` in it.
+  - It says what a failure means: 401 is the value; a Gitea 403 is a scope, and the probe prints
+    Gitea's message; an "Access denied" 403 is a Cilium/Envoy denial.
+  - The token is fed to curl on stdin (`--config -`), never in argv.
+
+**`ci/lint.sh`**: a new guard rejects any non-comment `uses:` in `.gitea/workflows` that needs a
+Docker daemon: `docker://`, `renovatebot/github-action` and Docker's daemon-bound
+`docker/*-action`s. That is the class of bug fixed in #84.
+
+### Root cause, with evidence
+
+Run 350 (task 567, 06:49 UTC) **got further than 348 and 349.** Gitea's router log, filtered to
+the act-runner pod `10.0.0.38`:
+
+    06:49:31 GET /api/v1/user                                   200 OK
+    06:49:31 GET /api/v1/version                                200 OK
+    06:49:31 GET /api/v1/repos/Home-Network/home-network-k3s-impl  200 OK
+    06:49:31 GET /api/v1/orgs/Home-Network                      403 Forbidden
+
+There were no `/v2/` requests: Renovate stopped before extraction. From the job log, verbatim:
+
+    ERROR: Repository has unknown error (repository=Home-Network/home-network-k3s-impl)
+      "url": "https://gitea.home.keithjackson.dev/api/v1/orgs/Home-Network",
+      "message": "token does not have at least one of required scope(s), required=[read:organization], token scope=read:package,write:repository,read:user"
+      "x-envoy-upstream-service-time": "27", "server": "envoy"
+    INFO: Repository finished ... "result": "unknown-error", "exitCode": 8
+
+**This is not Cilium.** The body is Gitea's JSON scope message, Gitea logged the request, and
+`x-envoy-upstream-service-time` shows that Envoy forwarded it upstream.
+
+**The secret WAS updated.** Runs 348 and 349 got `GET /user 403` from `tokenRequiresScopes`. Run 350
+got `/user 200`, and its token scope list now includes `read:user`.
+
+**Why `read:organization` is needed.** Renovate 44.167.0, `lib/modules/platform/gitea/index.ts`
+`initRepo`, calls `helper.isOrg` → `GET /orgs/{owner}` and rethrows on error. Gitea 1.24.3,
+`routers/api/v1/api.go`, puts `/orgs/{org}` behind
+`tokenRequiresScopes(AccessTokenScopeCategoryOrganization)`.
+
+**Why `write:issue` is needed next.** In the same `api.go`, `/repos/{o}/{r}/issues…`, `/labels`
+and `/milestones` sit in a group with `tokenRequiresScopes(AccessTokenScopeCategoryIssue)`
+(lines 1466–1582). Renovate's gitea-helper uses `/issues`, `/issues/{n}/labels`, `/labels` and
+`/issues/{n}/comments`, for the dependency dashboard (`:dependencyDashboard` is extended) and for
+PR labels. A dry run only reads them; a scheduled run writes them.
+
+The other causes the coordinator listed are ruled out:
+- **Schedule window and dry-run.** The run never reached branch processing.
+- **Repository config.** The run never reached the clone (`"cloned": undefined`).
+- **The Summary step.** It is not involved; the failure is Renovate's own exit 8.
+
+## What I verified
+
+- I read the whole run 350 job log (204 lines), and the router lines above. Runs 348 and 349 are
+  as recorded in `decisions/2026-10-30-renovate-first-run.md`'s follow-up: `/user` got 403 from
+  `tokenRequiresScopes`.
+- **The probe step, in the pinned image.** I extracted the step body from the YAML and ran it in
+  `ghcr.io/renovatebot/renovate:44.167.0` (`curl` 8.5.0 and `jq` are present) with a deliberately
+  **bogus** token. Every probe reported 401 with the "token VALUE" diagnosis, the scope-list read
+  fell back to a warning, and the step exited 1.
+- **The scope-list parser.** Against run 350's exact message format it reports `write:issue`
+  MISSING. Against a five-scope list it reports both OK. Against `all` it reports both OK.
+- **The lint guard.** It flags `renovatebot/github-action` in the pre-#84 `renovate.yaml`
+  (dfc432c, line 152). It reports nothing on the current workflows, and it ignores commented
+  lines. `bash -n ci/lint.sh` passes.
+- **Account facts, read-only.** `renovate-bot` is a collaborator with `write`, and the org is
+  public.
+
+## What I did NOT verify
+
+- **The probe with a real token. UNVERIFIED.** I never had the token. In particular, I have not
+  observed that `GET /api/v1/admin/orgs` gives renovate-bot's token a 403 with a `token scope=`
+  list. The source says the scope check runs before `reqSiteAdmin`. If the message format
+  differs, the step warns and does not fail.
+- **That five scopes are enough for a full non-dry run.** They cover every helper endpoint Renovate
+  44.167.0 calls, mapped to Gitea 1.24.3's route categories by reading source. Branch pushes go
+  over git-over-HTTPS with the token; I believe that needs write:repository, but I have not
+  observed it.
+- **Registry reads as renovate-bot.** `read:package` on a public org is expected to work. The new
+  probe is the first real test of it.
+- **Sensitive paths.** `.gitea/**` and `ci/**` are both sensitive. CI's `approval-policy` will block
+  until the operator posts `/approve-override` as a PR review.

decisions/2026-10-30-renovate-first-run.md

@@ -0,0 +1,113 @@
+---
+date: 2026-10-30
+branch: fix/renovate-first-run
+title: Renovate never started because its action needs a Docker daemon jobs here do not get; it now runs as the job container
+---
+
+## What changed
+
+**`.gitea/workflows/renovate.yaml`**: the job no longer uses `renovatebot/github-action`. The
+pinned Renovate image (`ghcr.io/renovatebot/renovate:44.167.0`, the same version the action was
+pinned to) is now the job's `container:`. A `run:` step calls `renovate-entrypoint.sh renovate`
+inside it. `actions/checkout` is gone, because Renovate clones the repo itself.
+`GITHUB_COM_TOKEN` is wired from a new **optional** secret, `RENOVATE_GITHUB_COM_TOKEN`. The step
+unsets it, and `RENOVATE_DRY_RUN`, when they evaluate to the empty string. The preflight also
+prints `renovate --version`, which proves the step ran inside the pinned image.
+
+**`renovate.json5`**: one comment now names `container.image` as the version pin, in place of
+`renovate-version`. The config itself is unchanged.
+
+### Why: the root cause
+
+Every run of the action died before Renovate started. That includes the operator's dispatch,
+run 343 (task 555, 2026-10-30 06:14 UTC), and the 11 scheduled runs before it (312–322). From
+run 343's log:
+
+    RENOVATE_TOKEN is present (value not printed).
+    [command]/usr/bin/docker run --env LOG_LEVEL ... --env RENOVATE_TOKEN=*** --env RENOVATE_CONFIG_FILE=/github-action/renovate.json5 ...
+    failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running: dial unix /var/run/docker.sock: connect: no such file or directory
+    Error: The process '/usr/bin/docker' failed with exit code 1
+
+The action is only a wrapper around `docker run`. On this runner a job deliberately gets no
+Docker daemon (`decisions/2026-10-29-image-pipeline.md`): act_runner maps `docker_host` to `-` when
+`DOCKER_HOST` is `tcp://`, and the DinD daemon is privileged. So this was a
+platform-compatibility failure. The run never reached authentication, token scope, config or the
+network. **The re-minted token is not implicated, and nothing in this log can say whether its
+scopes are right.**
+
+A job `container:` needs no daemon. act creates it through the runner's DinD the same way it
+creates the default `ubuntu-latest` image, then `exec`s each step inside it. I checked this
+against gitea/act v0.261.2:
+
+- The job container's entrypoint is `/bin/sleep` (`run_context.go`). The Renovate image has
+  `/usr/bin/sleep`.
+- The step command files are written `0o666` (`step.go`), so the image's uid 12021 can write
+  `$GITHUB_STEP_SUMMARY`.
+- `HOME` comes from the image env.
+
+## What I verified
+
+- **The root cause.** I read all of run 343's job log (162 lines, job 0; job 1 is 404). It has one
+  error, quoted above, and no Renovate output at all. Runs 312–322 are all `failure` from the
+  `schedule` event.
+- **The config validates against the pinned tag.** `podman run ... ghcr.io/renovatebot/renovate:44.167.0
+  renovate-config-validator /tmp/renovate.json5` printed `Config validated successfully`. It also
+  printed the known migration diff for the top-level `docker` block.
+- **A dry run with the fix extracts and looks up dependencies.** This was a local reproduction. I
+  made a clean snapshot of this branch's tracked files. I ran it in a podman container of the pinned
+  image with entrypoint `/bin/sleep`, as the image's own uid 12021 with `HOME=/home/ubuntu`.
+  Inside it I `exec`ed the workflow step's `run:` body, extracted verbatim from the YAML, with:
+  - `RENOVATE_PLATFORM=local`, `RENOVATE_DRY_RUN=full` and `GITHUB_COM_TOKEN=` (empty, to
+    exercise the unset);
+  - a docker host rule for `gitea.home.keithjackson.dev` using the `gitea-admin` credential from
+    `~/.git-credentials`, passed through env and never printed.
+
+  The run exited 0 and logged `Dependency extraction complete`:
+  - flux 15 deps / 13 files
+  - github-actions 24 / 4
+  - kubernetes 131 / 67
+  - kustomize 2 / 2
+  - regex (both custom managers) 7 / 6
+  - **total 179 deps / 92 files**
+
+  It found **42 updates**: 29 kubernetes, 6 flux, 2 kustomize, 1 github-actions and 4 regex
+  `digest`. 7 of them are `major`.
+- **The `# renovate-digest:` lookups against the Gitea registry succeed.** All three
+  `/v2/token?scope=repository:home-network/<img>:pull` requests succeeded, and so did
+  `HEAD /v2/home-network/<img>/manifests/main`. They proposed `digest` updates for
+  `conformance/versions.env`, `infra/agent/versions.env`, `platform/pm-agent/configmap.yaml` and
+  `platform/pm-agent/deployment.yaml`, all of which carry the zero placeholder digest today. The
+  `kubernetes` manager's copies of those images report `disabled`, which is the intended effect of
+  the packageRule.
+- **The new pin is still maintained by Renovate.** The github-actions manager extracts
+  `ghcr.io/renovatebot/renovate:44.167.0` from `renovate.yaml` as depType `container`, and proposes
+  44.198.0.
+- **The `regex` vs `custom.regex` question from #73/#77 is closed.** Both custom managers produced
+  dependencies.
+- **Missing GitHub token.** The local run logged `GitHub token is required for some dependencies`.
+  That accounts for 17 deps skipped with `github-token-required`, which is why
+  `RENOVATE_GITHUB_COM_TOKEN` is offered as an optional secret.
+
+## What I did NOT verify
+
+- **The workflow on the real runner. UNVERIFIED.** My reproduction ran with podman, not
+  act_runner + DinD, and used `--platform=local`. The act behaviour (sleep entrypoint, `exec` as the
+  image user, 0666 command files) comes from reading the gitea/act v0.261.2 source, not from a run.
+  The first real evidence will be a `workflow_dispatch` after merge.
+- **The Gitea platform path and `RENOVATE_TOKEN`'s scopes. UNVERIFIED.** I have no access token, only
+  the `gitea-admin` password, which Renovate's Gitea platform cannot use. So these were not
+  exercised: platform init (`GET /user`, which needs `read:user`), repo clone, and the registry
+  lookups using the **token** with `read:package` rather than a password.
+- **Behaviour inside the schedule window.** A dispatch outside 03:00–06:00 America/New_York
+  (07:00–10:00 UTC during EDT) will extract and look up, but will log "not within schedule"
+  instead of `DRY-RUN: Would create ...` lines. I did not change this.
+- **Renovate as uid 12021 doing artifact updates** (for example helm lockfiles) on a real,
+  non-dry run. `--dry-run=full` does not run the tools.
+- **The self-bump.** Renovate will propose `ghcr.io/renovatebot/renovate` 44.167.0 → 44.198.0 as a
+  minor, which the default rule auto-merges. `.gitea/**` is a sensitive path, so it will in fact
+  wait for an administrator override. Nothing re-runs `renovate-config-validator` on that bump,
+  and no gate check runs the validator.
+- **No alert for "Renovate red every day".** That gap is still open. This failure ran for 12 days
+  without anyone noticing.
+- **Sensitive path.** `.gitea/**` is in `infra/gitea/approval-policy.json`, so CI's
+  `approval-policy` stage will block until the operator posts `/approve-override` as a PR review.

decisions/2026-10-30-renovate-hold-auth-and-self.md

@@ -0,0 +1,52 @@
+---
+date: 2026-10-30
+branch: fix/renovate-hold-auth-and-self
+title: Operator decision — the oauth2-proxy sidecar and Renovate's own version never auto-merge
+---
+
+## What changed
+
+`renovate.json5` gains two packageRules. Both set `automerge: false` and `addLabels: ["needs-review"]`.
+They sit after the auto-merge rule, so they override it at every update type. Each rule's
+`description` carries the operator's reason.
+
+1. **`quay.io/oauth2-proxy/oauth2-proxy`.** This is the single depName emitted for:
+   - `catalog/render.py` `OAUTH2_PROXY_IMAGE` (custom regex manager);
+   - the five rendered `apps/wave-*` sidecars;
+   - `platform/crafty/deployment.yaml` (kubernetes manager).
+
+   **Reason:** since v7.13.0, oauth2-proxy validates refreshed OIDC sessions against the access
+   token, not the id_token. Every sidecar sets `--cookie-refresh`, and that path is untested against
+   this Keycloak: the platform oauth2-proxy runs v7.15.5 without `--cookie-refresh`.
+   `render-golden` and `--selftest` pass on the bump, but both check rendered text, not the binary.
+2. **`ghcr.io/renovatebot/renovate`.** This is the job `container.image` in
+   `.gitea/workflows/renovate.yaml`. The rule also lists `renovatebot/github-action`, so a
+   reintroduction would be held too. It is not used anywhere today; `ci/lint.sh` rejects it.
+
+   **Reason:** nothing re-validates `renovate.json5` against the new version before it takes
+   effect. The 39.x `CONFIG_VALIDATION` failure is the precedent.
+
+## What I verified
+
+- **The config validates on the pinned 44.167.0.** `renovate-config-validator` printed
+  `Config validated successfully`. The migration diff is only the pre-existing top-level `docker`
+  block.
+- **The flag change is exactly the intended one.** I used Renovate 44.167.0's own
+  `resolveConfigPresets` and `applyPackageRules` to evaluate b4ff287's config and this branch's
+  config against the same updates (the local dry run of the #86 config). Exactly **2 of 29**
+  branch rows differ:
+  - `ghcr.io-renovatebot-renovate-44.x`: `automerge=true labels=` → `automerge=false labels=needs-review`;
+  - `quay.io-oauth2-proxy-oauth2-proxy-7.x`: the same change, and every upgrade in the branch
+    resolves `false`, the `catalog/render.py` one included.
+
+  The other 27 rows are byte-identical.
+
+## What I did NOT verify
+
+- **The three GitHub-hosted branches** in run 357 (`pin-dependencies`, `kubevirt-cdi`,
+  `intel-intel-device-plugins-for-kubernetes-0.x`) are not in the harness table: my local run has
+  no github.com token. Neither rule's package names can match them.
+- **A live Renovate run with this config.** The next debug-level dispatch should show
+  `needs-review` on both branches.
+- **The oauth2-proxy refresh path itself** remains untested. Exercising it on one app is the
+  review the label asks for.

decisions/2026-10-30-renovate-run-hygiene.md

@@ -0,0 +1,128 @@
+---
+date: 2026-10-30
+branch: fix/renovate-run-hygiene
+title: Renovate's first completed dry runs, audited; it would have opened red PRs for every app and missed the databases. Config fixed
+---
+
+## What changed
+
+**`renovate.json5` only.** Each change carries a comment quoting the measurement behind it.
+
+1. **Three new regex managers, so updates edit the SOURCE.**
+   - `catalog/*.yaml` `source.image`.
+   - `catalog/render.py` `OAUTH2_PROXY_IMAGE`.
+   - CNPG `imageName:` under `platform/` and `apps/`.
+
+   The first two match the rendered line's depName, so the catalog edit and the generated
+   `apps/wave-*` edit land in **one** branch.
+2. **`allowedVersions` mirroring each ServiceDefinition's `update.range`:**
+   - gitea/gitea `<2.0.0`
+   - nextcloud `<32.0.0`
+   - qbittorrent `<6.0.0`
+   - radarr `<6.0.0`
+   - sonarr `<5.0.0`
+   - semaphore `<3.0.0`
+
+   Plus CNPG postgresql `<17.0.0`, because a Postgres major is a migration, not a bump.
+3. **The blast-radius never-automerge list gains** `gitea/gitea`, `gitea/act_runner` and
+   `ghcr.io/cloudnative-pg/postgresql`.
+4. **HelmRelease range widening** (`matchManagers: flux`, ranged currentValue) is never
+   auto-merged and is labelled `range-widen`.
+5. **Majors need a Dependency Dashboard tick** (`dependencyDashboardApproval`) before a PR opens.
+6. **`flux` manager docker deps are disabled.** These are duplicates of `kubernetes` manager
+   extractions.
+7. **`velero/velero-plugin-for-aws` uses `semver` versioning**, so its RC pin can reach a stable
+   release.
+8. **`python` minors are not auto-merged.**
+9. **Two false comments are corrected.**
+   - Ranges "cap Renovate at patches": false, Renovate widens them.
+   - `nextcloud:31-apache` and `docker:27-dind` "cannot be maintained": false, they were.
+
+PR #85 is merged, so nothing here overlaps an open branch.
+
+### The audit: dispatch runs 353 (07:12 UTC) and 354 (07:23 UTC), on 593d0fa
+
+Both runs ended `success`, Renovate `"result": "done"`, `exitCode 0`. Both were at `info` level, so
+I reproduced the per-dependency detail with a debug-level local dry run of the same commit (see
+below).
+
+The runs were inside the 03:00–06:00 America/New_York window, so branches were processed:
+- run 353: 37 "Would commit" branches;
+- run 354: 40, because the operator had added `RENOVATE_GITHUB_COM_TOKEN` 9 s before it.
+
+Gitea's router log for the act-runner pod during both runs shows 317 2xx responses. The only
+non-2xx were all expected:
+- `/v2/` 401: the registry challenge;
+- `/admin/orgs` 403: the #85 scope probe, by design;
+- 6× `HEAD …/manifests/sha256:000…` 404: the placeholder digests.
+
+| # | Class | Count | Cause | Severity | Fix |
+|---|---|---|---|---|---|
+| 1 | App-image PRs edit only generated `apps/wave-*` | 9 branches | A ServiceDefinition has no `apiVersion`, so the `kubernetes` manager extracts 0 deps from `catalog/`. The oauth2-proxy sidecar is a Python constant (`catalog/render.py:132`) | **Real problem.** Every such PR fails `ci/render-golden.sh` (reproduced: rc 1 "GOLDEN DRIFT"), stays red and holds a PR slot | git (1) |
+| 2 | CNPG `imageName` untracked | 3 Clusters | The `kubernetes` manager reads `image:` only. The one tracked postgres line was the grant Job (`nextcloud-db.yaml:303`, "keep in step with spec.imageName") | **Real problem.** A real run bumps the Job and leaves the databases behind | git (1) |
+| 3 | Bogus or out-of-policy majors | 5 branches: gitea 28.2.0, qbittorrent 20.04.1, radarr 6.x, nextcloud 35, postgresql 18.6 | Odd legacy tags, plus `update.range` in `catalog/` not being read by Renovate | Risk if run for real: unmergeable PRs, which hold slots | git (2) |
+| 4 | Auto-merge of the merge gate's own host | gitea 1.24.3→1.27.3, act_runner 0.2.11→0.6.1, both "minor" | Not in the blast-radius list. A 0.x minor is breaking by semver | **Risk if run for real** | git (3) |
+| 5 | Range widening auto-merged | external-dns, headlamp (minor); longhorn (blast radius); kube-prometheus-stack (major) | Renovate widens `>=a <b`. The config's comment claimed the opposite | **Risk if run for real:** silently lets Flux adopt new minors | git (4, 9) |
+| 6 | Majors hold PR slots | 10 majors in run 354 against `prConcurrentLimit: 5` | Majors never auto-merge, so they stay open | Risk: starves auto-mergeable PRs | git (5) |
+| 7 | `Cannot find replaceString … alpine:3.22` (INFO) | 1 per run | The same HelmRelease `image:` line is extracted by `flux` and `kubernetes` (also the velero plugin) | Cosmetic. The branch is still right | git (6) |
+| 8 | velero-plugin-for-aws stuck on `v1.14.3-rc.2` | 1 dep, 0 updates | Docker versioning treats `-rc.2` as a suffix, so stable `v1.14.3` and `v1.14.4` never qualify | **Real problem** (silent) | git (7) |
+| 9 | python 3.13→3.15 as auto-mergeable "minor" | 2 files | Language runtime minor | Risk if run for real | git (8) |
+| 10 | `WARN GitHub token is required` | run 353: 5 packages, 17 deps | No github.com token | Real (they were unmaintained) | **operator, done:** run 354 has no WARN and adds `pin-dependencies`, `kubevirt-cdi`, `actions-checkout-7.x`, `major-github-artifact-actions` |
+| 11 | Floating or untagged images (`invalid-value`) | 4: jackett, plex `latest`; pihole, keycloak no tag | No version to compare | Real, already documented | none in git. Pinning by hand is an operator/PM decision |
+| 12 | `runs-on: ubuntu-latest` `invalid-version` | 11 | The github-actions manager extracts runner labels | Cosmetic, debug-only | none |
+| 13 | `Docker Manifest is unknown` (debug) | 3 | HEAD on the all-zero placeholder digests | Expected. Clears when the own-image digest PRs merge | none |
+| 14 | `Docker: error fetching data from DockerHub` 403 (debug) | 5 | Hub refuses `page=11`. Renovate already has the tags | Cosmetic. Updates were still found | none |
+| 15 | `Config migration necessary` | 1 | Top-level `"docker": {"enabled": true}` | Cosmetic, pre-existing | none (as before) |
+
+**Checked and fine:**
+- The 3 own-image digest updates were found in both runs. All 4 `# renovate-digest:` markers parse.
+- No marker failed to parse.
+- No package file is under `conformance/runs` or `tests`, nor is `DECISIONS-LOG.md`, `decisions/` or
+  any `*.sops.yaml` (77 package files checked).
+- `kubevirt-cdi` grouping works.
+- The schedule window behaved as designed.
+- Vulnerability alerts: none in either log.
+
+## What I verified
+
+- I read both job logs in full (132 and 129 lines), and the router log for 10.0.0.38 during both
+  runs.
+- **`renovate-config-validator` on 44.167.0**: `Config validated successfully`. The migration diff
+  is only the pre-existing `docker` block.
+- **Local dry runs.** Pinned image, `--platform=local`, debug, origin/main snapshot. "Before" is
+  593d0fa and "after" is this config; the GitHub-hosted deps were skipped in both.
+  - **Extraction:** `regex` went from 6 files / 7 deps to **19 files / 20 deps**. Every other
+    manager is unchanged: flux 15, github-actions 24, kubernetes 131, kustomize 2.
+  - **Branches:** 33 before, **29 after**. Removed: gitea-28.x, qbittorrent-20.x, radarr-6.x,
+    nextcloud-35.x and postgresql-18.x. Added: velero `v1.14.3-rc.2 → v1.14.4` (patch).
+  - **Gitea, qbittorrent, radarr, sonarr, semaphore and seerr branches** now each carry
+    `catalog/<svc>.yaml` + `apps/wave-*/<svc>.yaml`.
+  - **The oauth2-proxy branch** carries `catalog/render.py` plus the 5 rendered apps and
+    `platform/crafty`.
+  - **The postgresql-16.x branch** carries all three Clusters.
+  - The 2 flux docker duplicates report `disabled`.
+- **render==golden holds for the new branch shape.**
+  - In the toolbox, I applied the gitea branch edit (catalog + apps) and the oauth2-proxy branch
+    edit (`render.py` constant + 5 apps) textually. `./ci/render-golden.sh` reported
+    `all 10 onboarded service golden(s) reproduce`, rc 0.
+  - Negative control, the old shape with only `apps/wave-0/gitea.yaml` bumped: rc 1,
+    `GOLDEN DRIFT`.
+  - Both edits were reverted afterwards.
+
+## What I did NOT verify
+
+- **The effect of the automerge, label and dashboard-approval rules (3, 4, 5, 8). UNVERIFIED in a
+  run.** `--platform=local` stops before branch processing, so I saw only which updates exist, not
+  their merged flags. The matchers are standard (`matchPackageNames`, `matchManagers`,
+  `matchCurrentValue`, `matchUpdateTypes`). The next dispatch's debug log shows them per branch.
+- **`allowedVersions` under docker versioning** used Renovate's "Falling back to npm semver
+  syntax" path (16 debug lines). That it filters is shown by the five removed branches.
+- **render==golden for the other app branches** (qbittorrent, radarr, sonarr, semaphore, seerr). I
+  proved it only for gitea and oauth2-proxy. They have the same shape: one `source.image` line ↔
+  one rendered `image:` line.
+- **The `allowedVersions` rules duplicate `catalog/*/update.range`** and can drift from it. Nothing
+  checks them against each other. A lint comparing the two is the obvious follow-up.
+- **Any real (non-dry) run.** None has happened. PR creation, labels, auto-merge through Gitea's
+  API and the Dependency Dashboard issue are all unobserved.
+- **Sensitive path?** `renovate.json5` is not in `infra/gitea/approval-policy.json`, as far as I
+  know. CI's `approval-policy` result on the PR is the measurement.

decisions/2026-10-30-renovate-track-keycloak-operator.md

@@ -0,0 +1,113 @@
+---
+date: 2026-10-30
+branch: feat/renovate-track-keycloak-operator
+title: Renovate tracks the Keycloak operator (the real Keycloak version), never auto-merged; the catalog's phantom image is pinned as documentation and disabled
+---
+
+## What changed
+
+- **`platform/keycloak/kustomization.yaml`**: each of the three `keycloak-k8s-resources/26.0.6`
+  release URLs (two CRDs and `kubernetes.yml`) now has a
+  `# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker on the line
+  above it. That is the format the existing `custom.regex` marker manager parses (the kubevirt
+  pattern). A comment explains that these URLs are the Keycloak version. The `Keycloak` CR has no
+  `spec.image`, so the operator's `RELATED_IMAGE_KEYCLOAK` picks the server image.
+- **`renovate.json5`** has two new packageRules:
+  - **`keycloak/keycloak-k8s-resources`**: `groupName: keycloak-operator`, `automerge: false`,
+    `addLabels: [needs-review, blast-radius]`. A comment explains why. The operator, the CRDs and
+    the server are released in lockstep. The IdP gates SSO for every app and for kubectl. And
+    `passkey-reconciler.yaml` encodes Keycloak 26.0.6 Admin API shapes that the static gate never
+    exercises. Majors still go through the existing dashboard-approval rule.
+  - **`quay.io/keycloak/keycloak`**: `enabled: false`. Its only occurrences are in files that
+    deploy nothing.
+
+  The "STILL CANNOT FIX" comment also gains a one-line correction for keycloak.
+- **`catalog/keycloak.yaml`**: `source.image` changed from `quay.io/keycloak/keycloak` (no tag) to
+  `quay.io/keycloak/keycloak:26.0.6`. A comment says the image is DOCUMENTATION-ONLY and that the
+  operator chooses the real one. `apps/wave-1/keycloak.yaml` was re-rendered, which changes one
+  line.
+- **`platform/keycloak/README.md`**:
+  - Corrects the false claim "Renovate maintains the version". Until now Renovate could not see
+    the version.
+  - Adds a new section, **"Upgrading Keycloak: the operator IS the version"**. It lists the 26.0.6
+    assumptions in the passkey reconciler to diff against a new tag, and the post-merge checks.
+
+**The phantom: pin and disable, not delete.** `apps/wave-1/keycloak.yaml` is not listed in
+`apps/wave-1/kustomization.yaml`, and render-golden excludes it (`RENDER_GOLDEN_NOT_ONBOARDED`).
+Deleting it would still not be a no-op:
+
+- It is the fixture for the keycloak cases in `apps/wave-1/tests/app-slis.test.yaml` (the
+  StatefulSet-based `keycloak-down` rule). Those cases are discovered by the
+  every-PrometheusRule-in-apps/wave-1 convention.
+- Removing it means moving that rule and its tests to `platform/keycloak`. That is a separate
+  observability change, not a Renovate one.
+
+Pinning plus `enabled: false` changes one line in a file nothing applies. It cannot affect the
+cluster, and render-golden stays green.
+
+## What I verified
+
+- **Live state, prod, read-only kubectl, 2026-10-30.**
+  - Keycloak CR `keycloak` has an empty `spec.image`.
+  - The operator Deployment runs `quay.io/keycloak/keycloak-operator:26.0.6`, imageID
+    `sha256:8e3a1a56346f…`, with `RELATED_IMAGE_KEYCLOAK=quay.io/keycloak/keycloak:26.0.6`.
+  - `keycloak-0` and `keycloak-1` both run `quay.io/keycloak/keycloak:26.0.6`, imageID
+    `sha256:a93d22e13b86…`.
+  - There is no PrometheusRule containing "keycloak" in any namespace. This confirms the phantom
+    deploys nothing, including its `keycloak-down` rule (see the next section).
+- **`renovate-config-validator`** on the pinned `ghcr.io/renovatebot/renovate:44.167.0`:
+  `Config validated successfully`. The only migration note is the pre-existing `docker` block.
+- **Local dry run** (`RENOVATE_PLATFORM=local`, `RENOVATE_DRY_RUN=full`, debug, the same pinned
+  image, this worktree, with a GitHub token passed by environment variable; the token string
+  occurs 0 times in the log):
+  - The regex manager went to 21 files / 24 deps.
+  - `keycloak/keycloak-k8s-resources` was extracted three times from
+    `platform/keycloak/kustomization.yaml`, each at `currentValue 26.0.6`, datasource
+    `github-tags`, versioning `semver`.
+  - Each extraction has one update, `26.8.1`, updateType `minor`, and all three share **one**
+    `branchName: renovate/keycloak-operator`.
+  - `quay.io/keycloak/keycloak` was extracted from `catalog/keycloak.yaml` (regex) and from
+    `apps/wave-1/keycloak.yaml` (kubernetes) at 26.0.6, with `skipReason: disabled` and no
+    updates.
+- **Rule effects.** `--platform=local` does not print each branch's merged flags. So I ran
+  Renovate's own `applyPackageRules` (from `dist/util/package-rules` in the same 44.167.0 image)
+  over this `renovate.json5` for each case:
+
+  | Update | Result |
+  |---|---|
+  | operator minor 26.0.6 → 26.8.1 | `automerge:false`, `groupName:keycloak-operator`, labels `dependencies, automated-update, needs-review, blast-radius` |
+  | operator patch | same as the minor |
+  | operator major | `automerge:false`, `dependencyDashboardApproval:true`, labels add `breaking` |
+  | `quay.io/keycloak/keycloak`, catalog and rendered | `enabled:false` |
+  | control: sonarr patch | `automerge:true`, so the rule did not disable auto-merge globally |
+- **render-golden, host:** `render.py --selftest` passed, and `all 10 onboarded service golden(s)
+  reproduce`. Before the pin, the committed phantom was byte-identical to the canonical render. I
+  re-rendered it after the pin.
+- **Toolbox gate** (commit 24db269, worktree with the ENGINE shim mounting `.git` and the spec
+  repo, `DEV_KUBECONFIG=/dev/null`): **rc 3**.
+  - The only stage that did not run is **`gitea-settings`**.
+  - spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all passed.
+  - L2/L3 `conformance` scaffold-skipped, as expected.
+  - **Not green.**
+- **Sensitive paths:** none touched (checked against `infra/gitea/approval-policy.json`), so no
+  `/approve-override` is needed.
+
+## What I did NOT verify
+
+- **A real Renovate run.** No PR was opened by Renovate. The labels and `automerge:false` are
+  proven by Renovate's own matcher, not by an opened PR.
+- **That 26.8.1 is safe to adopt.** I did not diff 26.0.6 against 26.8.1. That includes the
+  reconciler's assumptions: `recurseExecutions` displayName, `updateExecutions`, the
+  `RpEntityName` guard, `requiredActions` REPLACE, and config aliases. The README section lists
+  them for whoever reviews the first bump PR. It also says to take or confirm a fresh CNPG backup
+  first, because a server minor may migrate the schema.
+- **FINDING, not fixed: Keycloak has no deployed "IdP down" alert.** The `keycloak-down` SLI rule
+  exists only inside the undeployed `apps/wave-1/keycloak.yaml`. promtool tests it, but it is not
+  on the cluster: there is no keycloak PrometheusRule live. Separately, because that file exists,
+  `render.py --emit-waves` would treat keycloak as onboarded. It would then add a Deployment named
+  `keycloak` beside the operator's StatefulSet (the known `--emit-waves` hazard). The proper fix is
+  a separate PR: move the rule and its tests into `platform/keycloak`, then delete the phantom.
+- **The catalog tag can drift.** Nothing ties `catalog/keycloak.yaml`'s documentation tag to the
+  operator URLs. It must be updated by hand after an operator bump, and the README says so.
+- **Concurrent branches** `fix/renovate-hold-auth-and-self` and `feat/pin-pihole-jackett-plex`
+  also edit `renovate.json5`. Neither had merged when I checked. A rebase may be needed.

decisions/2026-10-31-pm-agent-digest-99.md

@@ -0,0 +1,41 @@
+---
+date: 2026-10-31
+branch: chore/pm-agent-digest-99
+title: Pin pm-agent dispatcher and worker to the agent-runner digest built from PR #99's merge
+---
+
+## What changed
+
+- `platform/pm-agent/deployment.yaml`: the dispatcher `image:` moves from
+  `agent-runner@sha256:df36d9b7…` (images run 383) to
+  `agent-runner@sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2`.
+  `config-generation` goes from "4" to "5" so the pod rolls.
+- `platform/pm-agent/configmap.yaml`: `PM_WORKER_IMAGE` gets the same digest. A comment line
+  records where it came from. The `# renovate-digest:` marker still sits directly above both
+  values.
+- No other values changed. `PM_DRY_RUN` stays "false". `PM_DECISION_MAX_TURNS` is not set, so
+  it keeps its in-code default of 1. No `*.sops.yaml` was touched.
+
+## What I verified
+
+- PR #99's merge commit is `466f63ec3c3df968a1dab25a8811dcd41961efde` (origin/main HEAD at
+  branch time).
+- The Gitea Actions tasks API shows the `images` workflow (images.yaml) on that commit as
+  **run 394** (task 643), with event `push` and status `success`.
+- I resolved the digest read-only from the registry by HEADing the manifests with OCI and
+  Docker Accept headers. Both `agent-runner:sha-466f63ec3c3d` and `agent-runner:main` returned
+  `200`, an `application/vnd.oci.image.manifest.v1+json` content type, and
+  `docker-content-digest: sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2`.
+  The two are identical.
+- Toolbox gate: the rc and skipped stages are in the PR description.
+
+## What I did NOT verify
+
+- That the dispatcher actually rolls to the new digest after merge, and that Flux applies it.
+  Flux's `lastAppliedRevision` and the pod's image were not checked. That is post-merge work.
+- That the new image surfaces the triage failure's real cause. It will show only on the next
+  failing triage, in the dispatcher log.
+- That admission (rule 7, `imagePullSecrets`) admits worker Jobs from this digest. The
+  template comes from `infra/agent/` at PR #99's merge, which still carries the
+  `imagePullSecrets` entry, but no Job was created to prove it.
+- I did not run the image's own selftest against this digest.

decisions/2026-10-31-pm-agent-go-live.md

@@ -0,0 +1,55 @@
+---
+date: 2026-10-31
+branch: chore/pm-agent-go-live
+title: Take pm-agent out of dry run (operator-approved); label scope and worker caps unchanged
+---
+
+## What changed
+
+- `platform/pm-agent/configmap.yaml`: `PM_DRY_RUN` `"true"` -> `"false"`. The operator approved
+  this after reading the dry-run triage of issue #97. `PM_REQUIRE_LABEL: agent`,
+  `PM_MAX_ACTIVE_WORKERS: 2`, `PM_MAX_NEW_WORKERS_PER_CYCLE: 1`, the worker image digest and every
+  other knob are unchanged. The two comments that said dry run was the only mode that worked
+  (because the run-340 digest lacked `imagePullSecrets`) now describe the run-383 digest that
+  PR #96 pinned.
+- `platform/pm-agent/deployment.yaml`: `config-generation` `"3"` -> `"4"`. Without the bump the
+  dispatcher keeps its startup config and stays in dry run.
+- `platform/pm-agent/README.md` and `docs/plans/pm-agent-issue-to-pr.md` (follow-up 3): the
+  "currently dry run" status text now says dry run is off. `runbooks/pm-agent.md` is unchanged.
+  Its "stop it: set `PM_DRY_RUN: "true"`" step and its admission-refusal troubleshooting row
+  are still accurate.
+
+## What I verified
+
+Read-only, `kubectl --context k3s-prod` through the toolbox, 2026-10-31 ~11:45 local, with Flux
+`pm-agent` and `platform-config` both at `main@1126740`:
+
+- ClusterPolicy `pm-agent-job-shape`: `Ready=True`, `validationFailureAction=Enforce`,
+  `background=false`. It has seven rules, including `worker-pull-secret-must-be-the-committed-one`.
+- RBAC: ServiceAccounts `pm-agent` and `pm-agent-worker`, Role `pm-agent` and RoleBinding
+  `pm-agent` (roleRef `pm-agent`, subject `ServiceAccount/pm-agent`) all exist. `auth can-i`
+  as `system:serviceaccount:pm-agent:pm-agent` returned `yes` for `create jobs.batch` and `no`
+  for `delete jobs.batch`.
+- Secrets (key names only): `pm-agent-registry` is `kubernetes.io/dockerconfigjson` with key
+  `.dockerconfigjson`. `pm-agent-credentials` is `Opaque` with keys `claude-code-oauth-token`
+  and `gitea-token`.
+- Egress: `default-deny` NetworkPolicy exists. CiliumNetworkPolicies `pm-agent-allow-egress`
+  and `pm-agent-worker-allow-egress` both report `Valid=True`.
+- Deployment `pm-agent` is 1/1 on `agent-runner@sha256:df36d9b7…` with
+  `config-generation: "3"` live. No Jobs exist in the namespace.
+- `platform/pm-agent/{configmap,deployment}.yaml`, README and docs do not match any
+  `sensitive_paths` glob in `infra/gitea/approval-policy.json`, so no review or override applies.
+
+## What I did NOT verify
+
+- **That a real worker Job is admitted and completes.** Nothing can show this until a labelled
+  issue gets a `start` decision after the merge. The L0 Kyverno suite shows the policy accepts
+  the committed template. Whether the API server admits the Job built from the df36d9b image's
+  baked template has not been tested live.
+- That the worker pod can pull its image with `pm-agent-registry`, reach Gitea and the model
+  through its egress policy, and push a branch and open a PR. None of this has run before.
+- The pod list also shows a stale `pm-agent-65db599649-v678g` (25d, `ContainerStatusUnknown`,
+  from an old ReplicaSet). It does not block anything, but I did not investigate it. Clearing
+  it is an operator action.
+- CI: see the PR for the gate result. Locally the best possible result is rc 3, with only
+  `gitea-settings` not run.