feat(keycloak): declare passkey (WebAuthn passwordless) login for the homelab realm

Adds the realm's WebAuthn Passwordless policy and restructures the browser-mfa
flow so a passkey is an ALTERNATIVE beside the password form. Additive and
optional by design: this IdP gates every SSO app plus kubectl, so a mandatory
passkey flow that misfires is a total lock-out. Enrolment is not forced.

Policy: RP ID is ${BASE_DOMAIN} (not the keycloak host) because a passkey's RP ID
is fixed at enrolment and must not be bound to a name that can move; resident key
required (a non-discoverable credential is a second factor, not a passkey); user
verification required (without it, possession alone is a complete login).

Flow: auth-username-password-form is split into auth-username-form +
auth-password-form, because the server must know who is logging in before it can
decide whether a passkey exists. Conditional OTP for platform-admins is relocated
intact onto the password branch only — a UV-required passkey is already two
phishing-resistant factors.

No requiredActions block is declared, deliberately: Keycloak 26.0.6 treats
requiredActions as REPLACE not merge, so declaring one would delete the default
set including CONFIGURE_TOTP and lock every platform-admin out of the OTP step
this flow still enforces. webauthn-register-passwordless is already registered
enabled-but-not-default by Keycloak's own defaults.

UNAPPLIED: the realm import is a one-shot resting at Done=True/Started=False, so
this reaches the live realm never. See the handoff entry and the README section.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitignore

@@ -37,6 +37,14 @@ coverage/
 infra/tofu/backend.hcl
 infra/tofu/.r2.env
 
+# Operator scratch, writable from inside the dev container (the repo is its only persistent
+# read-write mount). Holds things that must never be committed: a staged copy of the SOPS age or
+# WireGuard key, rendered VPN client profiles (infra/vpn-ingress/new-peer.sh writes here by
+# default). `infra/ansible/.gitignore` already does this for its own .cache/; this is the same
+# convention at the root. Do NOT rely on `*.key` alone to protect a staged key — that only covers
+# one filename shape, which is why this rule exists.
+.cache/
+
 # Claude Code local scratch: agent worktrees + session state. Nothing here is tracked;
 # ignoring it keeps `git add -A` from staging embedded worktree repos and keeps the lint
 # stage (which scans untracked files) from tripping on .claude/worktrees/.

apps/wave-0/plex.yaml

@@ -77,7 +77,7 @@ spec:
   storageClassName: longhorn
   resources:
     requests:
-      storage: 8Gi
+      storage: 16Gi
 ---
 # PVC plex-media (R-STATE)
 apiVersion: v1

catalog/plex.yaml

@@ -351,7 +351,7 @@ exposure: public              # edge port-forward + public DNS; clients + Seerr
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: config
-    size: 8Gi
+    size: 16Gi
     class: longhorn           # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
     mountPath: /config        # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
                               # renderer default /data/config. Without this the PVC sat unused and Plex wrote to the

catalog/qbittorrent.yaml

@@ -359,7 +359,7 @@ storage:
                                 # The live /config contents were copied onto each PVC before this landed,
                                 # so API keys and credentials other apps reference were preserved.
   # THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
-  # (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
+  # (192.168.2.30:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
   # /downloads, which is precisely where this app is ALREADY configured to write.
   #
   # WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
@@ -434,7 +434,7 @@ dependencies: []                # default-deny: reaches only DNS; arr reach it n
                                 # for weeks and neither has any CNP naming the NFS server or port 2049
                                 # (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
                                 # egress gateway off this traffic (its selector is a podSelector), and
-                                # 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
+                                # 192.168.2.30 is inside the 192.168.0.0/16 excludedCIDR besides — see
                                 # platform/vpn-egress/egress-steering.yaml, "BULK NFS".
 ingressRoutes:
   - host: qbittorrent.${BASE_DOMAIN}

clusters/base/cluster-vars.yaml

@@ -234,26 +234,28 @@ data:
   WG_CLIENT_CIDR: "10.90.0.0/24"             # remote VPN clients; .1 = server. NOT a LAN range
 
   # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
-  # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
-  # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
+  # THE address of the NAS whose ZFS datasets back every `class: bulk` volume: the media library
+  # (Storage/plex, 8.33 T — plex, sonarr, radarr, qbittorrent) and the legacy Nextcloud archive
   # (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
   # `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
   # declared in the ServiceDefinition, because the path is identical in every environment and only
   # the ADDRESS is not.
   #
-  # WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
-  # The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
-  #   dev  nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
-  #        NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
-  #        /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
-  #        per the convention of the LAN block above.
-  #   prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
-  #        22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
-  #        the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
-  #        DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
-  # So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
-  # blast radius of shipping it unset, which is NOT the usual "fails to apply".
-  BULK_NFS_SERVER: "192.168.1.9"
+  # IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read "the two clusters are on
+  # OPPOSITE SIDES of pfSense": the datasets lived on a Proxmox host whose only interface was
+  # vmbr0 192.168.1.9/24 on pfSense's LAN, which prod (192.168.2.0/24, pfSense's WAN) could reach
+  # only through a DNAT at 192.168.2.102. Both pools were physically moved onto a standalone
+  # TrueNAS at 192.168.2.30 — prod's OWN /24 — so there is no crossing to arrange and no candidate
+  # values to choose between. Measured 2026-10-07: 192.168.2.102:2049 no longer answers;
+  # 192.168.2.30:2049 and :111 do. There is now exactly ONE storage host and one correct value;
+  # this stays a knob only because the renderer reads it per-environment.
+  #
+  # *** CHANGING THIS VALUE IS NOT A PLAIN RECONCILE. *** A bound PersistentVolume's spec.nfs is
+  # immutable — measured by server dry-run 2026-10-07:
+  #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+  # so Flux's apply FAILS rather than updating, and the pods keep their old mount. The PVs must be
+  # deleted and recreated; the operator sequence is docs/BULK-NFS-STORAGE.md §10.
+  BULK_NFS_SERVER: "192.168.2.30"
 
   # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
   # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,

clusters/base/vpn-ingress.yaml

@@ -8,8 +8,8 @@
 #     NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves
 #     (DECISIONS-LOG: "Headlamp gating stall").
 #   * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the
-#     operator creates the out-of-band `wg-server-key` Secret (runbooks/vpn-remote-access.md
-#     step 1). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
+#     operator replaces the `REPLACE-ME` placeholder in platform/vpn-ingress/wg-server-key.sops.yaml
+#     with a real key (runbooks/vpn-remote-access.md step 2). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
 #     the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the
 #     cause (a missing manual step) invisible from any Flux condition.
 #
@@ -51,6 +51,9 @@ spec:
     substituteFrom:
       - kind: ConfigMap
         name: cluster-vars
-  # NO `decryption:` stanza is needed in the overlays: this path carries no *.sops.yaml. The server
-  # key is an out-of-band Secret (platform/vpn-ingress/kustomization.yaml says what to do if that
-  # ever changes).
+  # SOPS decryption is layered per-environment by the overlay (clusters/dev and clusters/prod each
+  # patch THIS Kustomization by name), identical to headlamp/crafty/velero — this path carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml, the WireGuard server private key. The `sops-age`
+  # Secret is created out of band and referenced by NAME only. Without the overlay patch the Secret
+  # applies cleanly full of ENC[...] and only the tunnel is broken, so the patch is as load-bearing
+  # as the encrypted file itself.

clusters/dev/kustomization.yaml

@@ -122,6 +122,24 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
   # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
   # account and the Anthropic subscription token from `claude setup-token`. Without this
@@ -261,15 +279,15 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.1.55"
-      # Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
-      # (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
-      # Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
-      # Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
-      # end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
-      # change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
+      # Bulk (external NFS) storage host. THIS VALUE IS INERT — dev is decommissioned, its nodes
+      # (192.168.1.40/.41/.42) no longer exist, and the Proxmox host this used to name (192.168.1.9)
+      # went with the pools it held. It is restated per this overlay's "supply every knob"
+      # convention, not because anything consumes it. It carries the real address so that no file
+      # in the tree renders a host that is gone; it has NOT been proven from a dev node, because
+      # there is no dev node to prove it from.
       - op: replace
         path: /data/BULK_NFS_SERVER
-        value: "192.168.1.9"
+        value: "192.168.2.30"
       # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
       # barmanObjectStore and Velero's backupStorageLocation write to.
       #

clusters/prod/kustomization.yaml

@@ -113,6 +113,24 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
   # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
   # account and the Anthropic subscription token from `claude setup-token`. Without this
@@ -522,60 +540,48 @@ patches:
         path: /data/LAN_CIDR
         value: "192.168.2.0/24"
       # ------------------------------------------------------------------------------------------
-      # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
+      # BULK (EXTERNAL NFS) STORAGE HOST — the TrueNAS at 192.168.2.30.
       # ------------------------------------------------------------------------------------------
       # Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
-      # radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
+      # radarr, qbittorrent) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
       #
-      # THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
-      # vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
-      # 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
-      # 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
-      # outcomes give DIFFERENT values here:
-      #     port-forward 192.168.2.102:2049 -> 192.168.1.9:2049   =>  "192.168.2.102"
-      #     a WAN->LAN route to 192.168.1.0/24                    =>  "192.168.1.9"
-      # Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
-      # ever became reachable — mounts something unintended. So it stays a placeholder.
+      # THIS WAS THE REPO'S ONE LOAD-BEARING PLACEHOLDER, AND THE REASON IT COULD BE IS NOW GONE.
+      # The datasets lived on a Proxmox host with a single interface on pfSense's LAN
+      # (192.168.1.9), which prod could reach only through a DNAT at 192.168.2.102 — so the correct
+      # value here depended on an undecided operator choice, and guessing it would have authored a
+      # cluster that mounts nothing. Both pools were physically moved onto a standalone TrueNAS at
+      # 192.168.2.30, prod's OWN /24. No crossing to arrange, no candidates to choose between.
+      # Measured 2026-10-07: 192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and :111 do.
       #
-      # WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
-      # NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
-      # over v3 was in part this decision's cost: v3 would have needed four ports and three of them
-      # on ephemeral numbers.
-      #
-      # WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
-      # Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
-      # the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
-      # address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
-      # 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
-      # addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
-      #
-      # *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
-      # A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
-      # placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
-      # failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
-      # TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
-      # ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
-      # apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
-      # a revision containing the bulk volumes; the dev overlay renders a working value, which is
-      # where these manifests are meant to be proven first.
+      # ONE PORT IS STILL ALL IT NEEDS: tcp/2049. The PVs mount nfsvers=4.1 and NFSv4 carries mount
+      # and lock in-protocol — no rpcbind/111, no mountd, no statd. The NAS advertises
+      # `+3 +4 +4.1 +4.2` (/proc/fs/nfsd/versions, measured), so 4.1 negotiates.
       - op: replace
         path: /data/BULK_NFS_SERVER
-        # 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
-        # host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
-        # reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
-        # keeps the base value 192.168.1.9.
+        # VERIFIED ON THE NAS 2026-10-07, before any cluster touched it (`exportfs -v`):
+        #   /mnt/Storage/plex       192.168.2.10/.11/.12  rw,sec=sys,secure,root_squash,no_all_squash
+        #   /mnt/Storage/nextcloud  192.168.2.10/.11/.12  ro,sec=sys,secure,root_squash,no_all_squash
         #
-        # PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
-        #   - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
-        #   - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
-        #     carries no_root_squash, so a root write would have passed and proved nothing, while
-        #     LSIO images default to PUID 911 and would have gone Ready then failed every import
-        #     with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
-        #   - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
-        #     real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
-        # pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
-        # so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
-        value: "192.168.2.102"
+        # no_all_squash is the flag the entire permission model rests on: client uids pass through
+        # untouched, so the arrs' PUID=1000 arrives as the owner of a 0755 1000:3000 tree. sec=sys
+        # keeps that numeric rather than principal-mapped. The export deliberately does NOT carry
+        # no_root_squash, which the Proxmox one did: nothing here needs it, because every write to
+        # the tree happens after the LSIO drop to uid 1000 and the images only chown their Longhorn
+        # /config. Both subPath directories (plex/downloads, nextcloud/nextcloud) already exist at
+        # 0755 1000:3000, so a squashed root can still traverse them for subPath resolution.
+        #
+        # The ro= posture on the archive is now SERVER-enforced, which the old export also managed.
+        # The media tree's read-only half (plex) stays a CLIENT-side guarantee — all four media pods
+        # can land on any node, so the server sees the same three addresses for reader and writers
+        # alike and cannot tell them apart. Unchanged by the move; see docs/BULK-NFS-STORAGE.md §5.
+        #
+        # *** CHANGING THIS VALUE REQUIRES DELETING AND RECREATING THE PVs. *** A bound PV's
+        # spec.nfs is immutable — measured by server dry-run 2026-10-07:
+        #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+        # so Flux's apply FAILS rather than updating and the pods keep their old, dead mount. The
+        # operator sequence is docs/BULK-NFS-STORAGE.md §10. Retain + the renderer's pre-set
+        # claimRef make it safe: no data is touched and each recreated PV rebinds its own PVC.
+        value: "192.168.2.30"
       # ==========================================================================================
       # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
       # ==========================================================================================

decisions/2026-10-05-pm-agent-registry-pull-prereq.md

@@ -0,0 +1,127 @@
+---
+date: 2026-10-05
+branch: docs/pm-agent-registry-pull-prereq
+title: pm-agent reconciled exactly as designed — and is the first workload ever to pull from the in-cluster registry, which does not work yet
+---
+
+## What changed
+
+Documentation only. No manifest, no policy, no code.
+
+PR #57 merged and `pm-agent` reconciled. I measured what landed, and the measurement turned
+up **two prerequisites for the next operator step that nothing in this repository
+declares** — and that the handoff entry for #57 could not have known, because it listed the
+image pull as unverified and that is exactly where it bit.
+
+So this entry records the post-merge measurement, and
+`platform/pm-agent/README.md`, `docs/plans/pm-agent-issue-to-pr.md` and
+`runbooks/pm-agent.md` now name both prerequisites where they say "build, push, pin the
+digest". Before this, that step read as one action; it is three, and the two new ones are
+not an agent's to perform.
+
+**The finding is not specific to pm-agent.** `conformance-runner` has the identical
+registry path (`gitea.home.keithjackson.dev/home-network/...`), the identical "operator
+pushes, a digest goes in a PR" model, and has never been pulled either — so the same two
+blockers sit in front of the L2/L3 conformance wiring. Measured: **no pod anywhere in the
+cluster references that registry** except the pm-agent dispatcher, which is therefore the
+first workload ever to try it.
+
+## What I verified
+
+Read-only, through the pinned toolbox, against the live prod cluster.
+
+**The merge reconciled, completely and exactly as designed.**
+
+- `GitRepository/flux-system` artifact revision: `main@sha1:95597bd6883637…` — the merge
+  commit. `platform-config`, `pm-agent` and `apps-wave-0` all report that same
+  `lastAppliedRevision` with `Ready=True`. Compared field to field, per CLAUDE.md §2b,
+  rather than assumed from the merge.
+- **The security boundary is live.** `ClusterPolicy/pm-agent-job-shape`:
+  `validationFailureAction=Enforce`, `failurePolicy=Fail`, `Ready=True`, and all six rules
+  present (`worker-must-be-the-committed-shape`,
+  `worker-must-be-one-container-with-no-args`,
+  `worker-volumes-must-be-workspace-or-tmp-emptydirs`,
+  `worker-may-not-declare-init-or-ephemeral-containers`,
+  `worker-env-names-must-be-the-committed-set`, `worker-may-not-use-envfrom`). This is the
+  check that mattered most: a ClusterPolicy added to the wrong kustomization list ships
+  green and never reaches the cluster, which is what happened to the conformance boundary
+  on 2026-09-29.
+- Namespace contents: both ServiceAccounts, `pm-agent-config` (17 keys), the
+  `pm-agent-credentials` Secret, the `default-deny` NetworkPolicy, and both
+  CiliumNetworkPolicies with `VALID=True`.
+- **SOPS decryption works through the new overlay patch.** Checked by KEY NAMES and VALUE
+  LENGTHS only, never values (CLAUDE.md §6): keys `gitea-token`, `claude-code-oauth-token`,
+  lengths 76 and 52 bytes — which are the base64 lengths of the committed placeholder
+  strings, so the Secret holds the decrypted placeholders and not ciphertext. A
+  ciphertext apply succeeds silently and would have left `ENC[AES256_GCM,…]` values an
+  order of magnitude longer.
+- **`pm-agent-worker` really is bound to nothing.** The namespace has exactly one
+  RoleBinding (`pm-agent` → Role `pm-agent`, subject `ServiceAccount/pm-agent`), and no
+  ClusterRoleBinding anywhere names either pm-agent ServiceAccount.
+- `gitea-allow-pm-agent-ingress` is applied in the `gitea` namespace, matching
+  `io.kubernetes.pod.namespace: pm-agent`.
+- The dispatcher is `ImagePullBackOff` on the all-zero placeholder digest — the designed,
+  visible failure.
+
+**The two prerequisites, measured.**
+
+1. **The nodes cannot verify the registry's TLS certificate.** containerd's error on the
+   pull is not about the digest:
+
+       failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:000…":
+       tls: failed to verify certificate: x509: certificate is not valid for any names,
+       but wanted to match gitea.home.keithjackson.dev
+
+   "not valid for **any** names" means the certificate presented to the node carries no
+   matching SAN at all. From the operator's workstation the same name is healthy: it
+   resolves to `192.168.2.20` and serves a Let's Encrypt certificate
+   `CN=*.home.keithjackson.dev` with SANs `*.home.keithjackson.dev, home.keithjackson.dev`
+   — and `Certificate gateway/wildcard-home` is `Ready=True` for exactly those names. So
+   the certificate that *should* be served is fine, and what the NODE reaches is not what
+   the workstation reaches. **Correcting the digest will not clear this.**
+2. **Anonymous pulls from this registry are refused.** Not inferred from a bare 401 — that
+   would have been wrong, because this Gitea answers 401 for *everything* anonymously:
+   `/v2/home-network/agent-runner/manifests/latest`, `/v2/home-network/conformance-runner/…`,
+   a randomly-named nonexistent repository, and `nosuchowner/nosuchimage` all return 401,
+   and `/api/v1/packages/Home-Network` returns `{"message":"token is required"}`. The
+   registry *does* issue a token to an anonymous client, but its `access` claim is `null` —
+   no pull scope. So the conclusion that holds is the simple one: **an unauthenticated
+   client gets nothing from this registry**, so a pull with no credential cannot succeed.
+   And `grep` over the tree finds **no `imagePullSecrets` anywhere** — not in
+   `platform/pm-agent/deployment.yaml`, not in `infra/agent/worker-job-template.yaml`, not
+   in `conformance/incluster/`.
+
+## What I did NOT verify
+
+- **Why the node sees a nameless certificate.** I measured that it does, and that the LAN
+  does not. I did not diagnose it: I did not read the nodes' resolver or
+  `/etc/rancher/k3s/registries.yaml` (node files, not reachable read-only through the API),
+  and I did not check whether the node resolves the name to the same `192.168.2.20` the
+  workstation does or to something else.
+
+  **A correction to a remark I first wrote here:** I cited
+  `cluster-vars`'s `GATEWAY_VIP: 192.168.1.53` as suspicious next to the nodes'
+  `192.168.2.x`. That was noise — the operator confirms those base `cluster-vars` values are
+  **dev-era leftovers for a cluster that is decommissioned**, so the 192.168.1.x addresses in
+  that file say nothing about prod and should not be read as evidence about anything. The
+  measured facts stand on their own and none of them depended on that knob.
+- **Whether the package would be pullable once authenticated.** Nothing has ever been
+  pushed to that path, so "a credential is sufficient" is untested. The only thing
+  established is that no credential is insufficient.
+- **Which fix the operator wants.** Two routes, and they are not equivalent:
+  (a) an `imagePullSecret` — a `dockerconfigjson` holding a Gitea token, SOPS-encrypted
+  like `pm-agent-credentials`, referenced from the Deployment *and* from the committed
+  worker Job template; note the Job template's shape is pinned by admission, and
+  `imagePullSecrets` is a field that policy does not currently constrain, so adding one is
+  a template change that needs a deliberate look at whether the policy should pin it too.
+  (b) allowing anonymous reads of that package in Gitea — no manifest change at all, but it
+  makes the image world-readable to anything that can reach the host.
+  I have implemented neither: both need an operator action I must not take, and (b) is a
+  posture decision. The docs now present both.
+- **Everything the #57 entry listed as unverified about RUNTIME remains unverified.** The
+  image has still never been built or pushed. No dispatcher has authenticated to Gitea, no
+  issue has been triaged, no worker Job has been created — so the admission policy, though
+  now provably applied and Enforce, has still never refused or admitted a real Job. The
+  network policies have not carried traffic.
+- **I did not re-run the merge gate against `main` after the merge.** This branch's own gate
+  run is the evidence for this change; the merged state's is CI run on PR #57.

decisions/2026-10-05-vpn-runbook-ansible-command.md

@@ -0,0 +1,112 @@
+---
+date: 2026-10-05
+branch: fix/vpn-runbook-ansible-command
+title: vpn-remote-access runbook — step 1 named a script that does not exist, and step 2 cited the wrong reason the key is gitignored
+---
+
+## What changed
+
+Two defects in `runbooks/vpn-remote-access.md` (merged in #55), both authored by me, both reported
+by the operator the first time they tried to follow it.
+
+**1. Step 1 named a command that does not exist.** It said:
+
+    cd infra/ansible && ./site.sh --tags system --limit k3s_prod
+
+There is **no `site.sh` anywhere in this repo** (`find . -name site.sh` → nothing). The invocation
+was wrong in four separate ways at once, which is what makes it worth recording rather than just
+patching: wrong kind of thing (this repo drives Ansible with `ansible-playbook <playbook>`, never a
+wrapper script — `playbooks/site.yml` exists and is the full bring-up chain, not this); wrong
+playbook (the module work lives in `playbooks/node-baseline.yml`); wrong targeting flag (`--limit`
+instead of this repo's `-e k3s_hosts=...` convention, which `infra/ansible/README.md` "Targeting
+prod" documents for every playbook); and no toolbox wrapper, though Ansible exists only inside the
+pinned container and `ansible.cfg` is only read from `infra/ansible`.
+
+Replaced with the real command, plus the three things about it that are load-bearing:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
+
+- `-e k3s_hosts=k3s_prod` — the playbook defaults to `k3s_servers`, which is the **dev** group;
+  dev is decommissioned, so an unqualified run does nothing useful.
+- `--tags system`, never untagged — the two kernel-module tasks carry that tag, and an untagged run
+  would also hit the `disks` tasks, which assert on block-device variables prod deliberately leaves
+  as placeholders (the playbook's own header warns about this). `--tags system` is broader than the
+  modules alone (sysctls, iscsid, qemu-guest-agent, swap masking) but each is idempotent and
+  already in its desired state on a live node.
+
+**2. Step 2 gave the wrong reason the server key is safe from git — and the right reason is
+fragile.** It told the operator to stage the key at `/work/.cache/server.key` "inside the repo's
+gitignored cache". `.cache/` is **not** gitignored: `git check-ignore -v .cache/serverkey.txt`
+matches nothing. The file was ignored only incidentally, by `.gitignore` line 11 `*.key` — so the
+path happened to be safe for a reason I had not identified, and **any operator who renamed it
+(`server.pem`, `wg-server`, `privatekey`) would have had a committable WireGuard private key sitting
+in the checkout.** The gate would not have caught it either; `no-plaintext-secrets` does not
+recognise a bare WireGuard key.
+
+The step now states the real rule, marks the `.key` suffix as the thing doing the work, and gives
+the operator a verification they can run before trusting it (`git check-ignore -v` must print a
+rule, and `git status --short` must not list the file).
+
+Nothing outside `runbooks/vpn-remote-access.md` changed; no manifest, no behaviour.
+
+## What I verified
+
+- **`site.sh` does not exist**: `find . -name 'site.sh' -not -path './.git/*'` → no output.
+- **The real surface**: `infra/ansible/playbooks/` contains `node-baseline.yml` and `site.yml` (and
+  11 others); `node-baseline.yml`'s own header documents both
+  `-e k3s_hosts=k3s_prod` and the tag-scoping hazard. The `-e <playbook>_hosts=` convention is
+  confirmed by `infra/ansible/README.md` §"Targeting prod" and by the existing
+  `runbooks/node-memory-swap.md`, which invokes `ansible-playbook playbooks/node-health.yml -e
+  node_health_hosts=...`.
+- **`k3s_prod` is a real inventory group**: `[k3s_prod:children]` in `inventory/hosts.ini`, with
+  `[k3s_prod_primary]`/`[k3s_prod_secondary]`.
+- **`--tags system` covers the kernel-module tasks**: both "Persist kernel modules…" and "Load
+  kernel modules now" carry `tags: [system]` in `roles/node_baseline/tasks/main.yml`.
+- **The gitignore claim, measured both directions**:
+  `git check-ignore -v .cache/server.key` → `.gitignore:11:*.key`;
+  `git check-ignore -v .cache/serverkey.txt` → no match. So the suffix is the only thing ignoring it.
+- **Every path the component's docs reference resolves**: all six relative markdown links in
+  `platform/vpn-ingress/README.md` (checked from that file's own directory), and both Ansible
+  playbook paths.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; the rc 3 is the two
+  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no `CLUSTER_AVAILABLE`).
+  **Not green. Reported as rc 3, not as pass.**
+- **This branch touches NO sensitive path** (`runbooks/**` and `decisions/**` are not in
+  `infra/gitea/approval-policy.json`), so unlike #55 it needs no `/approve-override`.
+
+## What I did NOT verify
+
+- **The corrected command has not been RUN.** Ansible is an operator action, and this is the
+  decisive gap: I have verified the playbook, the group, the tag and the flag all exist and are
+  the documented convention, but not that the run succeeds end to end or that
+  `lsmod | grep '^wireguard'` then reports the module. It is a better-evidenced command, not a
+  proven one.
+- **Whether `--tags system` is entirely side-effect-free on a live prod node.** Each task it runs is
+  idempotent by construction, but I did not dry-run it (`--check`) to confirm it reports zero
+  changes beyond the module load.
+- **The rest of the runbook's commands past step 2** are unchanged and still unexercised — in
+  particular `kubectl exec ds/wireguard -- wg show`, which cannot be reached while the pods sit in
+  `Init` waiting on the server key.
+- **Whether any OTHER document I wrote in #55 contains an invented command.** I re-checked every
+  file path and markdown link in the component (all resolve) and every script the docs tell an
+  operator to run, but "the referenced file exists" is weaker than "the command is correct" — which
+  is exactly the gap that produced `site.sh`.
+
+## Post-merge state of #55, recorded here since its own entry predates the merge
+
+Measured on live prod after #55 merged as `ad3f56e`:
+
+- Flux applied it by the real path: Kustomization `vpn-ingress` `lastAppliedRevision:
+  main@sha1:ad3f56e`, equal to the GitRepository's artifact revision, `Ready=True`.
+- All **18** Flux Kustomizations `Ready=True` — the new tier disturbed nothing, and `wait: false`
+  behaved as intended (`vpn-ingress` is Ready while its own pods are not, so nothing downstream
+  stalled on it).
+- **`hostNetwork` is confirmed in effect, not merely configured**: the three pods report pod IPs
+  `192.168.2.10/.11/.12` — the node addresses, one per node.
+- **It fails closed exactly as designed**, with a legible cause: all three pods `Init:0/1`,
+  `MountVolume.SetUp failed for volume "wg-server-key" : secret "wg-server-key" not found`, and no
+  Secret in the namespace. The #55 entry's "the DaemonSet has never been reconciled" is superseded
+  by this paragraph; its remaining unverified claims (the tunnel carrying a packet, the
+  iptables/nftables backend question) still stand.

decisions/2026-10-05-vpn-server-key-in-sops.md

@@ -0,0 +1,187 @@
+---
+date: 2026-10-05
+branch: fix/vpn-server-key-in-sops
+title: VPN server key moves into SOPS-encrypted git — the out-of-band Secret broke R-SEC-01 and the DR story
+---
+
+## What changed
+
+**Operator correction (2026-10-05): "secrets are kept in source through sops".** They are right, and
+#55 shipped the WireGuard server private key the wrong way: as a Secret the operator was told to
+`kubectl create` out of band, referenced by name only. This branch moves it into the tree as
+`platform/vpn-ingress/wg-server-key.sops.yaml`, where every other secret in this repo lives.
+
+**Why the original choice was wrong, not merely unconventional.** Two concrete costs, both of which
+#55's own README half-admitted and then shipped anyway:
+
+1. **R-SEC-01 / spec docs/07 says secrets live in source, SOPS-encrypted, reaching the cluster only
+   through Flux.** An out-of-band Secret is outside the GitOps tree by construction.
+2. **It broke DR for the one credential that cannot be re-derived.** A cluster rebuild restores
+   everything in git; it does not restore a hand-made Secret. Losing this key means **re-enrolling
+   every device**, because each client profile pins its public half. #55 papered over this with
+   "keep a copy in your password manager", which is a workaround for a self-inflicted gap.
+
+Note `platform/kyverno/admission/secret-provenance.yaml` did **not** forbid the original shape —
+it scopes to `app-namespace=true` namespaces and `vpn-ingress` is deliberately a platform namespace,
+so admission would have accepted the hand-applied Secret. The rule it violated was the written
+requirement, not an enforced control. Recording that because "the gate let it through" was never
+evidence it was correct.
+
+**How the unfillable-key problem is solved.** An agent still must not mint a long-lived private key
+(CLAUDE.md §6), so the file is committed **already encrypted, with `privatekey: REPLACE-ME`**. That
+makes the wiring complete and the gate green from the first commit — the resource line and both
+overlay decryption patches are present — and leaves the operator exactly one act: replace the value.
+The alternative, a commented-out resource line to uncomment later, is precisely the "file in git with
+no generator entry: reviewed, merged, apparently done, still broken" shape this repo keeps getting
+bitten by (`alert-runbooks-resolvable` exists because of it).
+
+**The init-container guard is now a validation, not an emptiness check.** It was `[ -z "$key" ]`.
+Because the committed placeholder is non-empty, the likely failure is now a value that is present
+and *wrong*, so the guard pipes the value through `wg pubkey` — which accepts only a real 32-byte
+curve25519 private key — and names which of four cases it hit:
+
+| case | why it matters |
+|---|---|
+| `REPLACE-ME` placeholder | the expected fresh-install state |
+| empty | blanked value |
+| **starts with `ENC[`** | **the Secret reached the cluster UNDECRYPTED.** A missing overlay decryption patch applies cleanly and leaves every Flux condition green — this is the silent failure the guard exists for |
+| rejected, none of the above | mangled value, or a *public* key pasted by mistake |
+
+Files:
+
+- `platform/vpn-ingress/wg-server-key.sops.yaml` (new) — SOPS-encrypted to the repo's age recipient,
+  `stringData.privatekey` only, placeholder value.
+- `platform/vpn-ingress/wg-server-key.sops.yaml.example` (new) — plaintext template + operator
+  hand-off, following the `crafty-oauth2-proxy-credentials.sops.yaml.example` convention.
+- `platform/vpn-ingress/kustomization.yaml` — the resource added; the "DELIBERATELY ABSENT: a
+  Secret" block replaced with the overlay-patch warning.
+- `platform/vpn-ingress/daemonset.yaml` — the `wg pubkey` guard; volume comment rewritten.
+- `clusters/dev/kustomization.yaml`, `clusters/prod/kustomization.yaml` — **the `vpn-ingress`
+  `decryption:` patch, in both.** This is the load-bearing half: #55's own kustomization comment
+  predicted that forgetting it applies ciphertext cleanly.
+- `clusters/base/vpn-ingress.yaml` — its "NO decryption stanza is needed" comment was now false.
+- `platform/vpn-ingress/README.md`, `runbooks/vpn-remote-access.md` — step 2 rewritten.
+- `infra/vpn-ingress/new-peer.sh` — stale "step 1" pointer.
+
+**A second finding: `wg` and `sops` were not in the same place — and the first fix for that was
+also wrong.** Measured: `wg` existed only on the host (`/usr/bin/wg`), `sops` only in the dev
+container, `qrencode` on neither. So `sops -d … | wg pubkey` could not run in one shell, and #55's
+step 2 — which assumed it could — would have failed.
+
+My first fix was to **split every step by where it runs**, and to tell the operator to
+`dnf install qrencode` on their workstation. **Operator correction: "sop, wg, and any other tool
+requirement is supposed to be baked into the devcontainer."** That is CLAUDE.md §5 and the
+devcontainer README's "Don't install the toolchain on the host", and the split fix broke the
+invariant rather than honouring it — it also produced a procedure that could never be reproduced in
+CI, and which no one should have to follow.
+
+**The real fix:** `wireguard-tools` and `qrencode` are now distro packages in
+`infra/devcontainer/Containerfile`, alongside ansible/openssh/diffutils. Both are in Fedora 44's
+repos (`wireguard-tools` 1.0.20260223 — the same upstream version as the container image this
+component runs — and `qrencode` 4.1.1), unpinned like every other dnf package there, because the
+pinned thing is the Fedora base and neither is a gate tool whose version changes what green means.
+
+Three things got *simpler* rather than merely relocated, which is the tell that the operator's rule
+was the right one:
+
+1. **Step 2 is now one pipeline in one shell:** `sops set … "\"$(wg genkey)\""` generates the key
+   straight into the already-encrypted document. The plaintext never lands on disk, never reaches a
+   terminal, and there is no staged copy to shred. The previous version needed a copy in
+   `.cache/server.key`, a `git check-ignore` warning about the `*.key` suffix being the only thing
+   protecting it, and a sha256 comparison to prove the right bytes arrived. All of that is gone.
+2. **`new-peer.sh` now derives the server's public key from `wg-server-key.sops.yaml` itself**
+   (`sops -d --extract … | wg pubkey`), so there is no local key copy to keep in step with the
+   encrypted one. It previously required the operator to keep the private key at a magic host path.
+3. **`.cache/` is now gitignored at the repo root** (mirroring `infra/ansible/.gitignore`), so the
+   container has a persistent, non-committable scratch directory. `new-peer.sh` writes client
+   profiles to `.cache/wg-peers/` by default — necessary because the repo bind-mount is the
+   container's ONLY persistent read-write mount, so a profile written to `/tmp` in there is gone
+   before it can be moved to a desktop. Its old blanket "refuse to write inside the checkout" guard
+   is replaced by `git check-ignore`, which tests the actual property instead of a proxy.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two overlays) AND adds a `**/*.sops.yaml`, so
+`ci/approval-policy.sh` will BLOCK it.** It needs the operator's `/approve-override <reason>` as a
+PR **review** of type "Comment". I have deliberately not posted it (CLAUDE.md §1).
+
+## What I verified
+
+- **The encrypted file is real and correctly targeted.** `sops -e -i` in the pinned toolbox
+  produced `stringData.privatekey: ENC[AES256_GCM,...]` with cleartext keys, `encrypted_regex
+  ^(data|stringData)$`, and exactly one age recipient:
+  `age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg` — the recipient in the root
+  `.sops.yaml`, i.e. the key Flux holds as `sops-age`.
+- **Round-trip, without exposing anything**: `sops -d` then a Python read of key names only →
+  `stringData keys: ['privatekey']`, and `value is still the placeholder: True`.
+- **Both overlay patches are present and name the right Kustomization** (`name: vpn-ingress`,
+  `namespace: flux-system`, `decryption.secretRef.name: sops-age`), inserted after each overlay's
+  `velero` stanza, mirroring headlamp/crafty/velero.
+- **`kubectl kustomize platform/vpn-ingress` builds** and now emits five objects — Namespace,
+  ConfigMap, **Secret**, DaemonSet, NetworkPolicy — with `${WG_CLIENT_CIDR}` still an unexpanded
+  token for Flux.
+- **The tool gap, measured on both host and container before the fix**: `wg` host-only, `sops`
+  container-only, `qrencode` on neither.
+- **Both packages exist in the image's repos**, queried from inside the running container:
+  `wireguard-tools` 1.0.20260223 (repo `updates`) and `qrencode` 4.1.1 (repo `fedora`).
+- **The image REBUILDS with them and both tools resolve inside it** —
+  `./infra/devcontainer/dev.sh build` succeeded, and `wg --version` / `qrencode --version` /
+  `sops --version` all answer in the rebuilt container. This is the claim the change rests on, so
+  it was executed rather than reasoned.
+- **`new-peer.sh`'s new SOPS path fails correctly against the placeholder.** Run in the rebuilt
+  container it refused with "could not derive the server public key … it still holds the REPLACE-ME
+  placeholder", which is both the expected pre-key behaviour and a live test of the
+  `sops -d | wg pubkey` pipeline the whole design now depends on.
+- **`.cache/` is ignored regardless of filename** now: `git check-ignore -v .cache/anything.conf`
+  and `.cache/serverkey.txt` both match `.gitignore:46:.cache/`. Before this change only `*.key`
+  matched, which is the fragility the previous runbook had to warn about.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
+  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no cluster).
+  **Not green. Reported as rc 3, not as pass.**
+- **The L0 gate caught THREE real defects in this branch before it could merge — including one
+  in the fix for another — and I had asserted one of them was fine without running it.** Both are recorded because the gate's value here is the
+  whole point of the exercise:
+  1. **`no-plaintext-secrets` FAILED**: "`wg-server-key.sops.yaml.example`: an unencrypted .example
+     template is referenced by a kustomization — it would be applied as a plaintext Secret". My
+     kustomization named the template in a *comment*, and the check matches comments too (correctly:
+     naming one is how a plaintext Secret gets applied by accident). `platform/crafty/kustomization.yaml`
+     never names its own example either — the convention was there to copy and I missed it. Fixed by
+     not naming the file; the hand-off lives inside the example, where it belongs.
+     **An earlier draft of this entry claimed this check passed. It did not. The claim was written
+     from expectation rather than from a run.**
+  2. **`flux-postbuild-substitution` FAILED** on `${key#ENC[}` in my new guard
+     (`daemonset.yaml:155`), flagged `[SILENT-EMPTY]`: envsubst consumes every `${...}`, so my
+     shell prefix-strip would have rendered to the empty string — the ciphertext branch of the guard
+     would never have fired, which is precisely the silent failure the guard was added to catch.
+     Rewritten with `case`, which uses no braces and so has nothing to escape.
+  3. **And then it failed a THIRD time, on the fix's own comment** — `[FATAL-PARSE]
+     daemonset.yaml:156 ${...}`, the prose explaining the rule, written with the literal form. That
+     class is worse than it looks: a `FATAL-PARSE` hazard fails the **entire** Kustomization, every
+     resource in the path, not just the line. Rewritten as "dollar-brace expansion". This is the
+     second time in this component that writing the hazard literally inside a comment about the
+     hazard has been the defect (the first was in #55), so the habit to build is: **in any
+     substituted path, describe the construct in words, never type it.**
+- **After both fixes, `no-plaintext-secrets` and `flux-postbuild-substitution` pass** (446 and 198
+  assertions respectively), and `l0` is green.
+
+## What I did NOT verify
+
+- **That Flux decrypts this file on prod.** The overlay patches have never reconciled. This is the
+  whole risk of the change and it has exactly one observable outcome: after merge the pods should
+  CrashLoop reporting the **`REPLACE-ME` placeholder** (proving decryption worked), *not* reporting
+  **`ENC[`** (which would prove a patch is wrong). I built the guard to tell those apart because I
+  cannot test them.
+- **That `wg pubkey` accepts the operator's real key through this path.** The guard is reasoned from
+  `wg pubkey`'s behaviour and has never run — `wg` is absent from both this workstation's container
+  and the gate, so the first execution will be in the pod.
+- **The `sops set` command in the hand-off.** I verified `sops -e -i` and `sops -d` in the toolbox;
+  I did **not** run `sops set` with the quoting shown, and nested quoting inside
+  `dev.sh run bash -c` is exactly where a documented command tends to break. If it misbehaves, the
+  editor form (`sops <file>`) is the fallback and does not depend on quoting.
+- **The sha256 comparison in step 2 step 4** is untested end to end (it needs the real key).
+- **Whether anything else in #55 assumed a tool is available where it is not.** I checked `wg`,
+  `sops` and `qrencode` because step 2 depended on them. I did not audit every command in the
+  runbook the same way — and that class of defect has now produced two corrections in a row
+  (`site.sh`, then `sops | wg`), so assume more remain until each step has actually been run.
+- **The live VPN still carries no traffic**, and the earlier gaps stand unchanged: the
+  iptables-vs-nftables question, `allowPrivilegeEscalation: false` versus the image's s6 init, and
+  any off-LAN reachability.

decisions/2026-10-06-new-peer-ignores-template-comments.md

@@ -0,0 +1,73 @@
+---
+date: 2026-10-06
+branch: fix/new-peer-ignores-template-comments
+title: new-peer.sh refused the first VPN address because its duplicate check matched its own commented-out example
+---
+
+## What changed
+
+The operator's first real enrolment failed:
+
+```
+$ ./infra/vpn-ingress/new-peer.sh keith-phone 2
+new-peer.sh: 10.90.0.2/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet.
+```
+
+**Nothing was allocated.** `peers.yaml` had zero peers — the pod's own init container reported
+`wg0.conf assembled: 0 peer(s)`. The duplicate-address check grepped the WHOLE file, and
+`peers.yaml` line 66 is a **commented-out example stanza** I wrote, containing literally
+`#   AllowedIPs = 10.90.0.2/32`. So `.2` — the first address anybody would ask for, and the one my
+own runbook uses in its example — was permanently un-allocatable, with a message asserting the
+opposite of the truth.
+
+Both halves fixed, because either alone leaves the trap half-set:
+
+- **`infra/vpn-ingress/new-peer.sh`**: the check strips comment lines before matching
+  (`grep -vE '^[[:space:]]*#' | grep -q …`). That is the correct predicate rather than a patch: a
+  real stanza's `AllowedIPs` line is never commented, while both the template and every peer's
+  `# <name> — <owner>` label always are.
+- **`platform/vpn-ingress/peers.yaml`**: the example now uses placeholders that cannot match
+  anything real — `# <device-name> — <owner>, enrolled <YYYY-MM-DD>` and
+  `AllowedIPs = 10.90.0.<octet>/32`. A future grep cannot be fooled by it either.
+
+The bug class is worth naming: **a documentation example that is byte-identical to real data will
+eventually be parsed as real data.** The same shape caused two earlier defects in this component —
+`flux-postbuild-substitution` flagged a `${...}` written inside a comment *explaining* that hazard,
+and `no-plaintext-secrets` flagged the `.sops.yaml.example` because the kustomization named it in a
+comment. Three instances now; the rule is to make examples structurally unmatchable, not merely
+commented out.
+
+No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `decisions/**`), so this
+merges on a green gate with no override.
+
+## What I verified
+
+Run in the pinned toolbox against the live tree, 2026-10-06.
+
+- **`.2` now works**, and it went through the REAL key path: `new-peer.sh keith-phone 2` rendered a
+  profile with `Address 10.90.0.2/32`, `AllowedIPs = 10.90.0.2/32` in the printed stanza, full
+  tunnel, at `.cache/wg-peers/keith-phone.conf` (0600). The throwaway keypair was deleted after.
+- **This incidentally closes a gap left open in #60**: the profile could only have been rendered by
+  deriving the server's public key via `sops -d --extract … | wg pubkey` from
+  `wg-server-key.sops.yaml` — so that pipeline is now proven **against the operator's real key**,
+  not just against a placeholder it was expected to reject.
+- **A real duplicate is still refused.** Appending an uncommented
+  `AllowedIPs = 10.90.0.5/32` stanza and asking for octet 5 produced the correct refusal; the tree
+  was restored afterwards and `git diff --stat` showed only the intended template edit.
+- **Zero active allocations remain**: `grep -vE '^[[:space:]]*#' peers.yaml | grep -c AllowedIPs`
+  → `0`.
+- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
+  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips (`gitea-settings`,
+  conformance L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **No peer is enrolled by this branch.** The keypair I generated was a throwaway and was deleted;
+  the operator must run the script themselves, since the device's private key must exist only on
+  the device. So the end of the enrolment path — a stanza merged into `peers.yaml`, Flux rolling
+  the DaemonSet, and `wg show` listing a peer — is still unexercised.
+- **The duplicate-NAME warning** (`grep -qi "^ *# *${NAME} "`) deliberately still inspects
+  comments, because per-peer labels are comments. It is no longer tripped by the template, but I
+  did not test it against a real enrolled peer.
+- **Nothing about client traffic.** The WAN port-forward is still not configured and there is still
+  no off-LAN vantage point, so the tunnel has carried no packet.

decisions/2026-10-06-peers-configmap-content-hash.md

@@ -0,0 +1,103 @@
+---
+date: 2026-10-06
+branch: fix/peers-configmap-content-hash
+title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
+---
+
+## What changed
+
+The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
+2026-10-06, with the peer already on `main`:
+
+| observation | value |
+|---|---|
+| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
+| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
+| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
+| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
+| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
+| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
+| init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
+
+The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
+fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
+no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
+into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
+would have had **no effect at all** until something unrelated happened to restart the pods. For a
+*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
+see a green PR, and the device would keep working.
+
+My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
+never true.
+
+**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
+*"The generated name carries kustomize's content hash, so editing any runbook rolls the server
+automatically and a stale projection is impossible."*
+
+So the register becomes a **content-hashed generator**:
+
+- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
+  plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
+  carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
+  documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
+  stripped.
+- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
+  ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
+  `nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
+  the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
+  `wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
+  by building the output and reading the volumes block.
+- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
+  register is now **fatal** instead of skipped. The old guard was
+  `[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
+  duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
+  as a defect in its own right.
+- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
+  `networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
+  operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
+- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
+  `generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
+  merge.
+
+Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
+
+No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
+`decisions/**`), so this merges on a green gate with no override.
+
+## What I verified
+
+Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
+
+- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
+  namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
+  `namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
+  bearing, measured both ways.
+- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
+  append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
+  and reversible, so an edit necessarily changes the pod template.
+- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
+  `peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
+  `main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
+- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
+  NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
+  transformer did not rewrite it).
+- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
+  "append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
+  real `.2` now occupies it — the first time the duplicate check has been exercised against real
+  data rather than against its own example.
+- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
+  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
+  as rc 3.**
+
+## What I did NOT verify
+
+- **That the roll actually happens on the cluster.** The whole point of the change is unproven
+  until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
+  `wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
+- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
+  port-forward is reportedly configured but has never been exercised by a packet I can see, and I
+  have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
+- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
+  should now roll the pods and drop the peer; untested.
+- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
+  `prune: true` on the Kustomization, not observed.

decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md

@@ -0,0 +1,126 @@
+---
+date: 2026-10-06
+branch: docs/pm-agent-registry-pull-prereq
+title: The nodes resolve gitea to the WAN, so the failing certificate is the router's — the second node process to need the issuer pin
+---
+
+## What changed
+
+The image pull for `pm-agent` failed TLS verification, and the previous entry on this branch
+honestly recorded it as **not diagnosed**. The operator asked for the diagnosis before any
+registry design. This is it, and the fix turns out to be a pattern this repository already
+has.
+
+`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins
+`gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the
+Keycloak issuer pin that is there for the identical reason.
+
+**It reaches the nodes the way every other node setting in this project does — through
+Ansible, from the inventory.** Node configuration is Ansible's (and, for the dev VMs,
+OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the
+*only* surface: `infra/PROD-PROVISIONING.md` records that prod is 3× bare metal and that the
+dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no
+cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command:
+
+    cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
+      -e k3s_hosts=k3s_prod --tags hosts
+
+`--tags hosts` runs ONLY the `/etc/hosts` task, so applying this pin cannot drift any other
+baseline setting. Running the play is an operator action (it has become/root on prod), but
+the configuration itself is declared, reviewed and version-controlled — not typed on a node.
+
+The docs that described this as an undiagnosed blocker now describe the cause and the fix:
+`platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and
+the play), and `docs/plans/pm-agent-issue-to-pr.md`.
+
+Also recorded, in the two places an operator reads when creating the credential: **the agent
+authenticates to Gitea with a local account and a scoped PAT, by decision** (operator,
+2026-10-06), not with a federated identity. Written down because the alternative looks
+obviously better until you check whether it is possible — see below.
+
+## What I verified
+
+Every step measured, read-only, in this order.
+
+**The failing certificate is the router's, not the Gateway's.**
+
+- containerd's error names TLS, not the digest: `tls: failed to verify certificate: x509:
+  certificate is not valid for any names, but wanted to match gitea.home.keithjackson.dev`.
+  "not valid for **any** names" means the presented certificate carries no SANs at all.
+- `192.168.2.20` is the shared Gateway's own LoadBalancer VIP
+  (`gateway/cilium-gateway-shared`, from the Cilium pool `192.168.2.20-30`), and its `https`
+  listener serves `*.home.keithjackson.dev` with `wildcard-home-tls`. From the operator's
+  workstation the name resolves there and serves the real Let's Encrypt
+  `CN=*.home.keithjackson.dev` with correct SANs. **So the certificate and the Gateway are
+  healthy.**
+- **The node asks a different resolver and gets a different answer.** Prod's
+  `node_baseline_dns_servers` is the LAN gateway, not Pi-hole — deliberately, as that file
+  explains. Public DNS for this zone is a CNAME to `wan.home.keithjackson.dev`: measured via
+  both `1.1.1.1` and `8.8.8.8`, `gitea.home.keithjackson.dev` →
+  `wan.home.keithjackson.dev.` → `136.51.115.252`, and the apex answers the same.
+- **That address serves a certificate with no names.** Measured from the workstation against
+  the WAN address with the right SNI:
+
+      subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
+      issuer=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
+      No extensions in certificate
+
+  No extensions means no `subjectAltName`, which is exactly "not valid for any names". The
+  `CN=%D` is the router's placeholder.
+
+**This repository had already measured the same thing, for Keycloak, and I did not need to
+re-derive it.** `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` says, above the pin:
+"Since platform/gateway/gateway.yaml set an external-dns `target`, every public name is a
+CNAME to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from
+the LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
+k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18)." Same cause, same symptom,
+one name later. The pin exists precisely for node processes that must reach the Gateway
+directly; containerd is one, and nobody had added it.
+
+**The parity the pin has to satisfy:** `192.168.2.20` equals the live
+`GATEWAY_VIP`, and `k3s_base_domain` equals the live `BASE_DOMAIN` — both read from the
+cluster, not from the comment that claims prod serves `prod.home.keithjackson.dev` (it does
+not; `flux-system/cluster-vars` reports `BASE_DOMAIN=home.keithjackson.dev` and
+`GATEWAY_VIP=192.168.2.20`, and Ansible's `k3s_base_domain` is `home.keithjackson.dev`).
+That comment is stale history.
+
+**The credential decision, and why the obvious alternative is not available.** Gitea's API
+accepts basic auth, PATs, SSH signatures and its own OAuth2 bearer tokens; as an OAuth2
+provider Gitea implements **only the Authorization Code Grant** (plus PKCE/OIDC), so there
+is no client-credentials grant, and nothing in Gitea validates a token issued by a
+third-party OIDC provider. A Keycloak service-account token therefore cannot authenticate to
+Gitea's API or to git-over-HTTP. Read from Gitea's own documentation
+(`docs.gitea.com/development/api-usage`, `/development/oauth2-provider`), not assumed.
+
+**Gate:** `./ci/gate.sh` in the pinned toolbox from a §0 worktree — see the branch's run; rc 3
+with `gitea-settings` the only stage that did not run.
+
+## What I did NOT verify
+
+- **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the
+  nodes is written by the `node_baseline` play's `hosts`-tagged task, which is an operator
+  action. I did confirm the mechanism is not inert — `roles/node_baseline/tasks/main.yml`
+  renders `node_baseline_hosts_pins` into a marked `blockinfile` with `backup: true`, and
+  `state: absent` when the list empties, so a removed entry really leaves the node. Until that runs
+  and an image exists, "the node now reaches the Gateway" is a design claim. The evidence
+  that it *will* is the Keycloak pin working for the same cause — which I also did not
+  re-measure.
+- **Why public DNS answers at all for an `internal` service.** `catalog/gitea.yaml` sets no
+  `exposure`, so it inherits `internal`, and the renderer contract says an internal service's
+  HTTPRoute carries `external-dns.alpha.kubernetes.io/controller: none` and publishes no
+  public record. Yet `gitea.home.keithjackson.dev` resolves publicly to the WAN CNAME. I did
+  not chase that: it may be a zone-level wildcard rather than a published record, and it did
+  not change the diagnosis. It is worth someone's attention, because "internal services
+  publish no public DNS" is an L0-checked claim and this looks like a case where the *name*
+  resolves regardless of what the renderer withheld.
+- **Registry authentication.** Unchanged and still open: an unauthenticated client gets
+  nothing from this registry, and no `imagePullSecret` exists anywhere in the tree. With the
+  credential model now decided (local Gitea accounts with PATs) the shape is a
+  `dockerconfigjson` for an account whose token carries `read:package`, referenced from the
+  Deployment and the committed worker Job template — and the worker Job template's shape is
+  pinned by admission, so whether the policy should also pin `imagePullSecrets` deserves a
+  deliberate look rather than a quiet addition. Not built: minting that token is an operator
+  action, and I did not want to guess at a second credential's scope.
+- **Anything about runtime.** The image still does not exist. No dispatcher has
+  authenticated, no issue has been triaged, no worker Job has been created, and the admission
+  policy — live, `Enforce`, six rules — has still never admitted or refused a real Job.

decisions/2026-10-06-vpn-ingress-ip-forward.md

@@ -0,0 +1,113 @@
+---
+date: 2026-10-06
+branch: fix/vpn-ingress-ip-forward
+title: The VPN tunnel never came up — a hostNetwork pod cannot write net.ipv4.ip_forward, and wg-quick treats a failed PostUp as fatal
+---
+
+## What changed
+
+The remote-access VPN reconciled for the first time with a real server key (#60 then #62) and
+**the tunnel did not come up**. One line in my own manifest was the whole cause.
+
+Measured in the `wireguard` container on prod, 2026-10-06:
+
+```
+[#] ip link add dev wg0 type wireguard
+[#] wg addconf wg0 /dev/fd/63
+[#] ip -4 address add 10.90.0.1/24 dev wg0
+[#] ip link set mtu 1420 up dev wg0
+[#] sysctl -q -w net.ipv4.ip_forward=1
+sysctl: permission denied on key "net.ipv4.ip_forward"
+[#] ip link delete dev wg0
+**** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! ****
+```
+
+Everything up to that point worked. **The cause is not a missing capability.** With
+`hostNetwork: true` the runtime mounts `/proc/sys` **read-only** in the container — confirmed in the
+pod's own `/proc/mounts`: `proc /proc/sys proc ro,nosuid,nodev,noexec,relatime` — precisely because
+those sysctls are the *host's* globals and a container must not rewrite them. `CAP_NET_ADMIN` cannot
+lift a read-only mount, so this can never succeed from here. And `wg-quick` treats any failed
+`PostUp` as fatal, so it deleted the interface it had just finished building.
+
+**The line was never needed.** `node_baseline` writes `net.ipv4.ip_forward=1` to
+`/etc/sysctl.d/99-k3s-cilium.conf`, and reading the value from inside the pod returns `1`. I had
+written the header comment *correctly* — "ip_forward is ALREADY 1 on these nodes … so no pod sysctl
+is needed" — and then emitted the sysctl anyway, justified in its own comment as "re-asserted so
+this component does not silently depend on another role's sysctl file surviving". That defensive
+re-assertion is the entire outage. **The lesson is narrow and reusable: in a hostNetwork pod,
+verify host state, never write it.**
+
+Fix:
+
+- **`platform/vpn-ingress/daemonset.yaml`**: the `PostUp = sysctl …` line is gone, replaced by a
+  **read** in the init container — `grep -qx 1 /proc/sys/net/ipv4/ip_forward` — which fails closed
+  with a message naming the `node_baseline` role. Reading `/proc/sys` is permitted; only writing is
+  not. Fail-closed still matters: a tunnel that comes up but cannot forward would satisfy every
+  probe in this component and silently drop every client's traffic.
+- The header bullet that said "no pod sysctl is needed" now also says **do not attempt one**, with
+  the mechanism, so the next author does not re-add it for the same reason I did.
+- The `securityContext` comment claimed NET_ADMIN was for "create the wg0 link, set the host sysctl,
+  and write the netfilter rules". The middle clause was false; corrected.
+- The image's `chown: … Operation not permitted` startup run — plus its own "Permissions could not
+  be set … we will not provide support for it" — is now documented as **expected and harmless**, in
+  both the manifest and the runbook, so nobody chases it. Capabilities are dropped to `NET_ADMIN`
+  only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs as root,
+  and `wg-quick` proceeds past it (as it did here — the failure was 30 lines later).
+- **`runbooks/vpn-remote-access.md`**: a "what a healthy start looks like" block with the real log
+  lines; `ip_forward is 0` added to the init-failure table; and the note that `0/1 Running` with
+  `Startup probe failed: Unable to access interface: No such device` is the signature of `wg-quick`
+  tearing the tunnel down — read the `[#]` line it died on.
+- **`platform/vpn-ingress/README.md`**: a table of what the first reconcile settled (below).
+
+No sensitive paths: `platform/vpn-ingress/**`, `runbooks/**` and `decisions/**` only. This merges on
+a green gate with no override.
+
+## What I verified
+
+Live prod, read-only, 2026-10-06. **Four questions this component's design rested on are now
+measured rather than assumed — three went the right way, one was the defect above.**
+
+- **The GitOps path is proven end to end.** `Kustomization vpn-ingress` `lastAppliedRevision:
+  main@sha1:cf7d5e4` equals the GitRepository artifact revision. The Secret carries
+  `kustomize.toolkit.fluxcd.io/name: vpn-ingress`, so Flux delivered it.
+- **SOPS decryption works, proven by digest rather than by eye.** The value in
+  `platform/vpn-ingress/wg-server-key.sops.yaml` on `main`, decrypted, is 44 bytes, is **not** the
+  placeholder, and **`wg pubkey` accepts it**. Its SHA-256 equals the SHA-256 of the live Secret's
+  decoded `data.privatekey` (`d7c8f8de6f252649…` both sides). No key material was printed at any
+  point — lengths, booleans and digests only.
+- **The `wg pubkey` guard fired correctly, and its `ENC[` branch was never needed.** During the
+  kubelet window before the updated Secret propagated, all three pods reported *"It is still the
+  committed REPLACE-ME PLACEHOLDER"* — the guard distinguishing a stale/unfilled value from
+  ciphertext, exactly as designed. It cleared on its own once the mount refreshed; **a transient
+  `REPLACE-ME` CrashLoop for a minute or two after the key PR merges is expected, not a fault.**
+- **The kernel module reaches the pod**: `ip link add dev wg0 type wireguard` succeeded and
+  `/proc/modules` lists `wireguard`. So the `node_baseline` addition works and `CAP_SYS_MODULE` is
+  correctly not required.
+- **`allowPrivilegeEscalation: false` does NOT break the image's s6 init** — the container reaches
+  `wg-quick` normally. The flag stays; the fear recorded in #55 is closed.
+- **The iptables backends MATCH, which was my single biggest documented doubt.** The image ships
+  `iptables v1.8.13 (nf_tables)` via `/usr/sbin/xtables-nft-multi`, and Debian 13 is nf_tables. The
+  nightmare case — rules accepted into a table the kernel never consults, so handshakes succeed and
+  every forwarded connection hangs — cannot occur on this pairing.
+- **`net.ipv4.ip_forward` is `1`** in the host netns the pod shares, and `/proc/sys` is `ro` there.
+  Both read directly from the pod.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
+  environmental skips (`gitea-settings`, conformance L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **That the tunnel now comes up.** This fix has not reconciled. The next reconcile is the test, and
+  its success criterion is specific: pods `1/1 Running`, and `wg show` reporting `interface: wg0`
+  with `listening port: 51820`.
+- **That the MASQUERADE and FORWARD rules actually apply.** They are the next statements after the
+  one that failed, so **they have never executed**. The backends matching (above) removes the worst
+  failure mode but proves nothing about the rules themselves. First check after the pods go Ready:
+  `iptables -t nat -S POSTROUTING | grep 10.90.0.0/24` on a node.
+- **Anything about client traffic.** Still zero peers enrolled, still no WAN port-forward, still no
+  off-LAN vantage point. The VPN has carried no packet.
+- **Whether the transient `REPLACE-ME` window has an upper bound.** It cleared within ~2 minutes
+  here; I did not determine the kubelet refresh behaviour, so a longer stall after a key change is
+  uncharacterised.
+- **The `sops set` hand-off command**, still. The operator filled the key in by their own route
+  (#62), so the exact quoting I documented inside `dev.sh run bash -c` remains unexercised.

decisions/2026-10-07-bulk-nfs-truenas.md

@@ -0,0 +1,85 @@
+---
+date: 2026-10-07
+branch: fix/bulk-nfs-truenas
+title: Repoint bulk NFS at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
+---
+
+## What changed
+
+Both ZFS pools were physically moved off the Proxmox host onto a standalone TrueNAS Community
+Edition at **192.168.2.30**, which sits on prod's own `192.168.2.0/24`. `BULK_NFS_SERVER` moves from
+`192.168.2.102` (the pfSense DNAT prod used to mount through) to `192.168.2.30`.
+
+- `clusters/prod/kustomization.yaml` — `BULK_NFS_SERVER` → `192.168.2.30`. The comment block that
+  justified the DNAT, and the "NOT YET DECIDED / THIS PLACEHOLDER IS LOAD-BEARING" banner, are
+  replaced with the measured export configuration and the recreate requirement below.
+- `clusters/base/cluster-vars.yaml` — base default `192.168.1.9` → `192.168.2.30`. That host no
+  longer exists.
+- `clusters/dev/kustomization.yaml` — same; marked explicitly **inert**, since dev is decommissioned
+  and its nodes are gone. It carries the real address only so nothing in the tree renders a dead one.
+- `docs/BULK-NFS-STORAGE.md` — §1 and §5 rewritten for TrueNAS (exports, service settings, the ZFS
+  `sharenfs` properties that had to be cleared, and the ACL hazard); §7 rewritten now that the
+  placeholder is gone; §9 retargeted from dev to prod; **new §10** documents the PV
+  delete-and-recreate. §2/§3/§4/§6 are unchanged by the move.
+- `catalog/qbittorrent.yaml`, `platform/monitoring/node-pressure-dashboard.yaml` — two comment/prose
+  references to `192.168.2.102` as the live export address.
+
+**The headline operational fact: repointing this knob is NOT a plain reconcile.** A bound PV's NFS
+source is immutable, so Flux's apply fails rather than updating and the pods keep their old, dead
+mount. The five PVs must be deleted and recreated. That sequence is §10 and is an operator action;
+it is **not** performed by this PR.
+
+## What I verified
+
+**On the NAS (192.168.2.30), measured 2026-10-07 by the operator at my request:**
+
+- `zfs list` — `Storage` → `/mnt/Storage`, `Storage/plex` → `/mnt/Storage/plex`,
+  `Storage/nextcloud` → `/mnt/Storage/nextcloud`. The PV paths resolve unchanged.
+- Ownership intact: `/mnt/Storage/plex`, `tv/`, `downloads/`, `movies/`,
+  `downloads/complete/`, `nextcloud/nextcloud/` all `drwxr-xr-x 1000:3000`. `movies/` 1171 entries
+  and `tv/` 107, against 1165 / 107 measured on Proxmox.
+- `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`), `aclinherit=passthrough`, and no
+  non-trivial ACLs — mode bits still govern.
+- Three stale `sharenfs` dataset properties (`plex`, `nextcloud`, `docker-volumes`) carried over from
+  Proxmox; cleared with `zfs inherit`, re-checked empty.
+- `exportfs -v`: both shares to `192.168.2.10/.11/.12` with `sec=sys,secure,root_squash,no_all_squash`,
+  `rw` on plex and `ro` on nextcloud.
+- `/proc/fs/nfsd/versions` → `+3 +4 +4.1 +4.2`, so the PVs' `nfsvers=4.1` negotiates.
+- **A defect was found and fixed during setup:** the plex share listed `192.168.2.13` — an address no
+  node holds — and so omitted `192.168.2.12` (k3s-prod-3). Corrected and re-verified.
+
+**From the cluster:**
+
+- Node InternalIPs are `192.168.2.10/.11/.12` (`kubectl get nodes`), confirming `.13` was wrong.
+- All five NFS PVs currently name `192.168.2.102`.
+- `192.168.2.30:2049` and `:111` accept connections; `192.168.2.102:2049` does not.
+- **`spec.nfs` immutability, by server-side dry-run** — this is the finding that shapes the change:
+  `spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation`.
+- The pods report `Ready` while their mounts are dead: a `stat /media` answers from attribute cache,
+  but a `readdir` blocks indefinitely (the PVs mount `hard`). Hence "scale to zero first" in §10.
+
+**Gate:** see the PR; run in the pinned toolbox from this worktree.
+
+## What I did NOT verify
+
+- **Nothing has mounted 192.168.2.30.** No pod has touched the new server — the PVs still point at
+  the old address and cannot be updated in place. Every §9 probe (numeric-id passthrough, the uid-1000
+  write, the hardlink/EXDEV check, Plex's read-only posture) is **outstanding**, not passing. The
+  equivalents passed against the old host on 2026-09-12; that is not evidence about this one.
+- **The §10 sequence is reasoned, not executed.** Retain + the renderer's pre-set `claimRef` are read
+  from the manifests and from `catalog/render.py`; I have not watched a PVC rebind after a PV delete
+  in this cluster.
+- **`root_squash` is a behaviour change from the Proxmox export**, which carried `no_root_squash`. The
+  argument that nothing needs root on the tree (LSIO drops to 1000; the images chown only their
+  Longhorn `/config`; both subPath directories already exist at 0755) is reasoned from the images and
+  the measured permissions. It has not been exercised. If a mount or subPath resolution fails with
+  EACCES, Maproot `root:root` reproduces the old export and is the first thing to try.
+- **Whether NFSv3 can be disabled.** Enabled Protocols is `NFSv3, NFSv4`; the PVs only need v4.1, so
+  narrowing would close the rpcbind/mountd/statd surface. Not done — it is a change to make after the
+  cutover is proven, not during it.
+- **Whether pfSense / the legacy stack is actually retired.** I measured only that
+  `192.168.2.102:2049` stopped answering. The other legacy consumers of these datasets
+  (`192.168.1.6`, `10.123.1.10`, `192.168.1.18`) were dropped from the new exports; if any is still
+  live, it has lost access. Not checked.
+- **L2/L3 conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline subset
+  only. Nothing in the gate exercises an NFS mount.

decisions/2026-10-17-keycloak-passkeys.md

@@ -0,0 +1,311 @@
+---
+date: 2026-10-17
+branch: feat/keycloak-passkeys
+title: Passkeys declared for the homelab realm — and the realm import will not apply them
+---
+
+## What changed
+
+`platform/keycloak/keycloak-realm-import.yaml` now declares passkey (WebAuthn Passwordless)
+login for the `homelab` realm, in two parts:
+
+1. **The WebAuthn Passwordless policy.** Ten `webAuthnPolicyPasswordless*` fields on the realm:
+   RP ID `${BASE_DOMAIN}`, RP entity name `Homelab SSO`, signature algorithms `[ES256, RS256]`,
+   `requireResidentKey: "Yes"`, `userVerificationRequirement: required`,
+   `authenticatorAttachment: "not specified"`, `attestationConveyancePreference: none`,
+   `acceptableAaguids: []`, `createTimeout: 300`, `avoidSameAuthenticatorRegister: true`.
+   Deliberately the **passwordless** family only; the two-factor `webAuthnPolicy*` family is left
+   at Keycloak's defaults because nothing in this realm uses `webauthn-authenticator`.
+
+2. **The browser flow.** `browser-mfa` keeps its alias and stays bound as `browserFlow`; its
+   interior is restructured so a passkey is an **ALTERNATIVE** beside the password form:
+
+   ```
+   browser-mfa                            (top level, unchanged)
+     browser-mfa-forms                    ALTERNATIVE
+       auth-username-form                   REQUIRED      <- was auth-username-password-form
+       browser-mfa-passkey-or-password      REQUIRED      <- NEW sub-flow
+         webauthn-authenticator-passwordless  ALTERNATIVE <- NEW: the passkey
+         browser-mfa-password-and-otp         ALTERNATIVE <- NEW sub-flow
+           auth-password-form                   REQUIRED  <- was half of the combined form
+           browser-mfa-conditional-otp          CONDITIONAL (unchanged, relocated intact)
+   ```
+
+The design decisions, with their reasoning, are written into the manifest comments in this repo's
+usual density. The ones that matter:
+
+* **Additive, not replacing. Enrolment optional, not forced.** This IdP gates Grafana, Gitea,
+  Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the \*arr stack *and* `kubectl` via the
+  `kubernetes` OIDC client. A mandatory passkey flow that misfires is a total lock-out from the
+  platform and the cluster API at once, leaving only the Keycloak bootstrap admin and the
+  break-glass cert kubeconfig. The gain from forcing it (one human stops typing a password) does
+  not price that risk.
+* **No `requiredActions` block was added, on purpose.** Declaring one would have been the obvious
+  way to add `webauthn-register-passwordless`, and it would have been actively dangerous — see
+  "What I verified".
+* **RP ID is `${BASE_DOMAIN}`, not `keycloak.${BASE_DOMAIN}`.** A passkey's RP ID is fixed at
+  enrolment and can never be re-keyed, so it must not be bound to a name that can move. This realm
+  already declares an `auth.${BASE_DOMAIN}` redirect URI, and this repo already owns a reconciler
+  built because a hostname move silently failed to reach the realm. Residual risk, unavoidable:
+  changing `BASE_DOMAIN` itself invalidates every enrolled passkey.
+* **A passkey login is not also asked for TOTP.** `browser-mfa-conditional-otp` hangs off the
+  password branch only. The justification is `userVerificationRequirement: required` — the
+  ceremony already proves possession plus a verified human, phishing-resistantly and origin-bound.
+  Stacking a replayable shared secret on top of that buys friction. The manifest names the exact
+  one-line edit to reverse this decision.
+* **Login becomes two screens** (username, then password) for every user, passkey or not, because
+  the server must know who is logging in before it can decide whether a passkey exists. Accepted
+  cost; it is the enabling change.
+* **Usernameless login deliberately not used**, although Keycloak 26.0.6 supports it — see "What I
+  verified" for the measured reason it would have made every login open on a passkey prompt.
+
+`platform/keycloak/README.md` gains a `## Passkeys` section: the decision table, the fact that
+this does not reach the live realm, the two operator paths that would apply it, and the four-step
+browser verification nobody has run.
+
+No secret, key, credential or AAGUID was minted or committed. No file under `clusters/**`,
+`conformance/**`, `ci/**`, `platform/kyverno/**`, `**/rbac.yaml` or `**/*.sops.yaml` was touched,
+so this should not trip `ci/approval-policy.sh`'s sensitive-path review requirement.
+
+## What I verified
+
+**The live realm import has already completed and will never re-run.** Measured on prod,
+2026-10-17:
+
+```
+$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
+{"conditions":[{"message":"","status":"True","type":"Done"},
+               {"message":"","status":"False","type":"Started"},
+               {"message":"","status":"False","type":"HasErrors"}]}
+```
+
+`Done=True` / `Started=False`, exactly the one-shot state `platform/keycloak/README.md` already
+documents for client secrets and client URLs. **So the change in this PR has zero effect on how
+anyone logs in today**, and the client-secret/client-URL reconciler does not and cannot close the
+gap: its hard endpoint allow-list makes `/authentication` unreachable by design, and realm-level
+policy fields are not client fields.
+
+**The CRD accepts every field I used.** Measured against the live CRD (not the upstream docs):
+`kubectl --context k3s-prod get crd keycloakrealmimports.k8s.keycloak.org` lists all twelve
+`webAuthnPolicyPasswordless*` properties with the types I wrote (`RequireResidentKey`,
+`RpEntityName`, `RpId`, `UserVerificationRequirement`, `AttestationConveyancePreference`,
+`AuthenticatorAttachment` as `string`; `SignatureAlgorithms`, `AcceptableAaguids`, `ExtraOrigins`
+as `array/string`; `CreateTimeout` as `integer`; `AvoidSameAuthenticatorRegister` as `boolean`),
+plus `authenticationFlows`, `authenticatorConfig` and `requiredActions`. The `spec.realm` schema
+has **no** `x-kubernetes-preserve-unknown-fields`, so a misspelled field would have been pruned
+silently rather than rejected — which is why this was checked rather than assumed.
+
+**Running Keycloak is 26.0.6.** `quay.io/keycloak/keycloak:26.0.6` on `keycloak-0`/`keycloak-1`,
+operator `quay.io/keycloak/keycloak-operator:26.0.6`. Every source claim below is read from the
+`26.0.6` tag, not from memory or from docs.
+
+**Every provider ID is quoted from upstream source, not guessed.**
+
+| String I used | Where it is defined at tag 26.0.6 |
+|---|---|
+| `webauthn-authenticator-passwordless` | `WebAuthnPasswordlessAuthenticatorFactory.PROVIDER_ID`, display "WebAuthn Passwordless Authenticator" |
+| `auth-username-form` | `UsernameFormFactory.PROVIDER_ID`, display "Username Form" |
+| `auth-password-form` | `PasswordFormFactory.PROVIDER_ID`, display "Password Form" |
+| `webauthn-register-passwordless` | `WebAuthnPasswordlessRegisterFactory.PROVIDER_ID` (referenced in comments only) |
+
+**`requiredActions` is REPLACE, not merge — measured, and it is why I did not declare one.**
+`model/storage-private/.../DefaultExportImportManager.java` lines 327-338:
+
+```java
+if (rep.getRequiredActions() != null) {
+    for (RequiredActionProviderRepresentation action : rep.getRequiredActions()) { ... }
+    DefaultRequiredActions.addDeleteAccountAction(newRealm);
+} else {
+    DefaultRequiredActions.addActions(newRealm);
+}
+```
+
+Declaring one required action therefore skips the entire default set. `RealmManager` is no
+backstop: `setupRequiredActions` is `if (realm.getRequiredActionProvidersStream().count() == 0)`,
+and the count is already non-zero by then (`RepresentationToModel.importRealm` runs at
+`RealmManager.java:585`, the backstop at `:612`). A realm built that way would have had the passkey
+action plus `delete_account` and nothing else — **no `CONFIGURE_TOTP`**, which would leave
+`auth-otp-form` demanding a TOTP that no platform-admin could ever enrol: a hard lock-out
+delivered by a change whose purpose was to *add* a login method.
+
+**And declaring it would have bought nothing.** `DefaultRequiredActions.addWebAuthnPasswordlessRegisterAction`
+registers `webauthn-register-passwordless` with `setEnabled(true)`, `setDefaultAction(false)`,
+priority 80, gated on `Profile.Feature.WEB_AUTHN` — which is `Type.DEFAULT` in
+`common/.../Profile.java:74`, i.e. on unless explicitly disabled. `enabled=true` /
+`defaultAction=false` *is* the additive-and-optional posture this change wants.
+
+**`authenticationFlows` has the opposite semantics, which is why declaring flows is safe.**
+`importAuthenticationFlows` adds the declared flows, then backfills each unbound binding
+(browser, registration, direct grant, reset credentials, client auth, first broker login, docker
+auth) from the built-in defaults individually (`DefaultExportImportManager.java` ~1320-1410).
+
+**The graceful degradation to password is measured from the flow engine, not assumed.** For a user
+with no passkey: `WebAuthnPasswordlessAuthenticator.requiresUser()` returns `false`, so
+`DefaultAuthenticationFlow.processSingleFlowExecutionModel` takes the `else` branch (`daf:456-462`),
+whose four conditions all hold — a user is set (from `auth-username-form`), `configuredFor` is
+false, `WebAuthnPasswordlessAuthenticatorFactory.isUserSetupAllowed()` is `false`, and
+`WebAuthnAuthenticator implements Authenticator, CredentialValidator<...>` — so it raises
+`CREDENTIAL_SETUP_REQUIRED`. The ALTERNATIVE loop (`daf:289-302`) catches exactly that, marks the
+execution `ATTEMPTED` and proceeds to the next alternative, with the upstream comment *"the user,
+since he has alternatives, should be able to go to another alternative and continue"*. Hence: no
+passkey, no passkey screen, straight to the password form.
+
+**Why usernameless was rejected, measured.** The `configuredFor` guard sits inside the
+`requiresUser()` branch. With `requiresUser()` false and no user yet in context, that guard is
+skipped and the engine just calls `authenticate(context)` — so a passkey execution placed before
+the username form is offered to *everyone*, including users who will never own one.
+
+**The policy sentinels are read from the shipped code, and one of my initial assumptions was
+wrong.** `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED = "not specified"` is the
+realm-representation sentinel — but `WebAuthnConstants.OPTION_NOT_SPECIFIED = ""` (empty string) is
+the WebAuthn-API-level one. I checked both because they disagree. The login theme's
+`webauthnRegister.js` at 26.0.6 settles the behaviour:
+
+```js
+if (input.requireResidentKey !== 'not specified') {
+    if (input.requireResidentKey === 'Yes') { authenticatorSelection.requireResidentKey = true; }
+    else { authenticatorSelection.requireResidentKey = false; }
+}
+...
+if (input.createTimeout !== 0) { publicKey.timeout = input.createTimeout * 1000; }
+rp: {id: input.rpId, name: input.rpEntityName}
+```
+
+So `"Yes"` is a literal string compare (any other non-sentinel value means *false*), `"not
+specified"` is compared literally for four fields, and `createTimeout` is **seconds** with `0`
+meaning "send no timeout". It also shows 26.0.6 emits only the legacy `requireResidentKey` member,
+never `residentKey: "required"` — the client maps that per WebAuthn L2, but the policy cannot
+express `residentKey: "preferred"` at all.
+
+`[ES256, RS256]` is exactly `WebAuthnPolicy.DEFAULT_POLICY` /
+`Constants.DEFAULT_WEBAUTHN_POLICY_SIGNATURE_ALGORITHMS` — declared explicitly, not changed.
+`getWebAuthnPolicyPasswordless` was read field by field to confirm `[]` for `acceptableAaguids`
+means "no restriction" and that `RpEntityName` must be non-empty or the **update** path
+(`DefaultExportImportManager.java:842`) drops the whole policy silently.
+
+**The manifest parses and renders.** `yaml.safe_load` round-trips the file and the flow graph reads
+back exactly as designed (five flows, the executions and priorities above, `browserFlow:
+browser-mfa`, `requiredActions` absent). `python3 ci/render-keycloak-client-urls.py --check` →
+`client-urls.yaml is in sync`, rc 0 (no client URL changed, so no regeneration was needed).
+`yamllint -c ci/yamllint.yaml platform/keycloak/keycloak-realm-import.yaml` → rc 0; no line exceeds
+the 160-column ceiling.
+
+**`browser-mfa` keeps its alias deliberately.** `conformance/tests/l2-behavioral/keycloak-config-as-code`
+asserts `kubectl get keycloakrealmimport homelab -o jsonpath={.spec.realm.browserFlow} | grep -q
+'browser-mfa'`. Renaming the top-level flow would have failed that test.
+
+**Envsubst safety.** `platform-config`'s Flux Kustomization declares
+`postBuild.substituteFrom: cluster-vars` (`clusters/base/platform-config.yaml:26-29`), so this file
+is substituted. The only brace token I added is `${BASE_DOMAIN}`, which is declared at
+`clusters/base/cluster-vars.yaml:42` (`home.keithjackson.dev`). Every comment I wrote avoids braces
+entirely — the RP-ID discussion says `keycloak.<BASE_DOMAIN>` with angle brackets precisely so an
+unparseable token cannot fail the whole Kustomization.
+
+**Gate result: rc 3, zero failures, `gitea-settings` the only stage the gate counts as not-run** —
+the documented honest best local outcome. Run in the pinned toolbox from this worktree with all
+three §3 workarounds (`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main
+checkout's `.git` and the sibling spec repo with `SPEC_REPO=/home-network-k3s`):
+
+```
+[ ok ] stage spec-immutability: pass      <- NOT skipped; both shim mounts worked, R-TEST-06 covered
+[ ok ] stage decisions-log: pass          <- archive frozen, 48 entries well-formed
+[ ok ] stage lint: pass
+[ ok ] stage l0: pass                     <- incl. flux-postbuild-substitution + keycloak-client-secret-reconciler
+[ ok ] stage render-golden: pass
+[ ok ] stage l1: pass
+[ ok ] stage approval-policy: pass        <- no sensitive path touched, so no non-author review needed
+[skip] stage gitea-settings: SKIPPED      <- needs SETTINGS_API_TOKEN, CI-only; expected
+[skip] stage conformance: SKIPPED         <- cluster-gated L2/L3; expected
+MERGE GATE: UNDETERMINED — Stages that did not run: gitea-settings
+```
+
+No lint finding and no check failure names either file I changed. Touched requirements resolved to
+`R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06`; coverage 51/51, 0 gaps.
+
+**A note on how that number was obtained, because the first attempt was worthless.** My first gate
+run returned the same rc 3 and the same seven passes — against a tree that did not contain this
+change at all. The gate's `tracked_files()` reads **git**, and my edits were still uncommitted, so
+every stage gated `origin/main`. The tell was in the log and I nearly missed it: the
+touched-requirement subset reported UNDETERMINED with *"check out the feature branch you mean to
+gate (not the base branch)"*, and `decisions-log` counted **47** entries rather than 48. **A green
+gate run on an uncommitted worktree is a gate run on the base branch.** The numbers above are from
+the second run, after committing and rebasing.
+
+**The base moved under me mid-run.** `origin/main` advanced from `9a41a8e` to `1a0b208` (PR #70,
+oauth2-proxy OIDC startup race) while I was working. This branch is rebased onto `1a0b208`, so the
+gate above ran on the base CI will actually merge into. The rebase was clean — PR #70 touched
+`platform/oauth2-proxy/helmrelease.yaml` and its own decisions entry, disjoint from these files.
+
+## What I did NOT verify
+
+**This change is unapplied. It changes nothing about the live realm, and I did not attempt to make
+it.** `Done=True` / `Started=False` was measured, not inferred, and I ran no mutating `kubectl`,
+no `flux reconcile`, no admin-API call and no `kcadm`. Whether the operator's eventual
+admin-console reconstruction of this flow matches what is declared here is, by construction,
+unverifiable from git.
+
+**I never enrolled a passkey and never pointed a browser at any of this.** Nothing here has been
+exercised by a real WebAuthn ceremony on any device. Specifically unverified:
+
+* **That the RP ID works.** `home.keithjackson.dev` is a registrable-domain suffix of
+  `keycloak.home.keithjackson.dev`, so it *should* satisfy the browser's check — but I verified
+  that by reading the WebAuthn rule, not by watching a browser accept it. A wrong RP ID fails
+  **silently, in the browser, before Keycloak is involved**, with no server-side log line. If
+  enrolment produces no OS prompt and no error, this is the first suspect.
+* **That `"not specified"` is the right literal for `authenticatorAttachment` in a realm import.**
+  I traced it to `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED` and to the theme's string
+  compare, but I have not seen a realm import accept it. Omitting the field would default to the
+  same value; if an import ever rejects the literal, drop the line.
+* **That `createTimeout: 300` behaves.** Seconds-vs-milliseconds is read from the theme JS. The
+  claim that 300 is long enough for the cross-device QR/hybrid flow, and that Chrome clamps rather
+  than rejects it, is reasoning about browser behaviour, not a measurement.
+* **Whether a passkey user can still reach the password form.** I argued it from
+  `createAuthenticationSelectionList` producing a "Try another way" option, and I noted that for a
+  user holding both credentials the *first* screen is chosen by the user's own credential ordering
+  rather than by my priorities. Neither was observed.
+* **The two-screen login regression for ordinary users.** `auth-username-form` +
+  `auth-password-form` replacing `auth-username-password-form` is a visible change for every user
+  including non-passkey family accounts, and nobody has logged in through it.
+* **That conditional OTP for platform-admins still fires on the password branch.** The sub-flow and
+  its `conditional-otp-platform-admin` config are byte-identical and merely relocated one level
+  deeper, but "identical YAML" is not "still enforced at runtime". This is the highest-value
+  regression check and it is untested. The previous breakage of this exact step
+  (`condUserRole` vs a non-existent property key) 403'd every platform-admin login.
+* **That a platform-admin authenticating by passkey is not challenged for TOTP.** That is the
+  intended behaviour of the shape I wrote; it is unobserved, and it is also a deliberate
+  relaxation of R-IDP-04 as previously implemented, which the operator may want to reverse.
+
+**Whether the realm import reconciles into an existing realm: measured as NO for the completed
+one-shot, but I did not test a re-import.** I did not delete and recreate the
+`KeycloakRealmImport`, so I cannot say what a re-run would do to the existing `homelab` realm —
+whether it would merge the new policy and flows, duplicate flow aliases, or fail. The 2026-09-06
+total SSO outage documented in `platform/keycloak/README.md` is the reason I did not experiment.
+**Treat the re-import path in the README as untested advice, not a runbook.**
+
+**Nothing was proved about the live realm's current state.** I did not read the realm's existing
+required actions, its existing WebAuthn policies, or its live flow executions — doing so needs the
+admin credential and I did not decrypt or use it. So I cannot confirm that the live realm's
+required-action set actually contains `webauthn-register-passwordless` today; I can only show that
+Keycloak 26.0.6's defaults register it and that this repo's import never declared any required
+actions to suppress them. **That inference has not been checked against the realm.**
+
+**L2/L3 conformance did not run** (no cluster from the gate container; `vars.CLUSTER_AVAILABLE` is
+unset in CI anyway), so the existing `keycloak-config-as-code` behavioural test did not re-execute
+against this change. I reasoned about its assertion by reading it.
+
+**No new conformance test was added.** Nothing in `conformance/` asserts the passkey policy or the
+new flow shape, so a future edit can silently remove either and no gate stage will notice. That is
+a real gap this PR leaves open; closing it would touch `conformance/**`, which is a sensitive path
+needing non-author review.
+
+**The quoted gate run predates this file's own gate-result paragraph.** It gated commit `207f5db`;
+the gate numbers and the rebase note above were then written into this entry, so the pushed tree
+differs from the gated one by this handoff's prose only — no manifest, no check, no generated file.
+`decisions-log` is the only stage that reads this file, and it validates structure (frontmatter,
+the three headings, non-stub bodies), which is unchanged. I did not re-run the full gate for a
+prose edit; CI will gate the pushed commit.
+
+**`gitea-settings` did not run locally** — it needs `SETTINGS_API_TOKEN`, which exists only in CI.
+I did not re-measure branch protection, the runner, or `required_approvals` against the Gitea API
+this session; the claims in `CLAUDE.md` §2 are taken on trust here and are explicitly flagged there
+as needing re-measurement.

decisions/2026-10-17-oauth2-proxy-oidc-startup-race.md

@@ -0,0 +1,99 @@
+---
+date: 2026-10-17
+branch: fix/oauth2-proxy-oidc-startup-race
+title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
+---
+
+## What changed
+
+Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
+discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
+serving yet, so the shared Gateway answers 503 and the proxy dies:
+
+```
+[provider.go:55] Performing OIDC Discovery...
+[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
+             failed to discover OIDC configuration: unexpected status "503": no healthy upstream
+```
+
+It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
+replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
+retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
+is already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+
+`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
+polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
+accrue CrashLoopBackOff backoff and retries promptly, which is the point.
+
+**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
+case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
+for a pod wedged in `Init` forever, which is not an improvement.
+
+## Why this shape and not the alternatives
+
+* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
+  example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
+  pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
+  at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
+  "reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
+* **No new base image.** The chart already runs an `alpine` init container for its built-in
+  `initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
+  `3.22` (verified to exist upstream) rather than the chart's own `latest`.
+* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
+  this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
+  the chart already has a hook for.
+* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
+  more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
+  oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
+  the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
+  the operator out of all of them, so the reversible, additive option wins.
+
+The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
+with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
+and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
+silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
+the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.
+
+## What I verified
+
+* **The failure is measured, not theorised.** The log above is from the live pod
+  (`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
+  with `--previous` after the cold boot. The surviving replica's log shows the same startup
+  succeeding once Keycloak was up, which is what identifies this as a race rather than a config
+  fault.
+* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
+  clean and both replicas have been healthy since.
+* **The chart key is real**, at the pinned version range: `helm search repo` resolves
+  `>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
+  `extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
+* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
+  assumed from memory.
+* The edited HelmRelease parses as YAML.
+
+## What I did NOT verify
+
+* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
+  seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
+  replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
+  seven apps, so **do it when you can watch it**, not unattended.
+* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
+  oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
+  from the measured failure, not demonstrated against it.
+* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
+  `ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
+  failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
+  degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
+  anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
+  init container's log on first apply**: it should print "Keycloak discovery reachable" within
+  seconds on a warm cluster, not the give-up message.
+* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
+  re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
+  good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
+  (`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
+  Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
+  this should hold — unverified.
+* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
+  not a measured bound.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
+  subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

docs/BULK-NFS-STORAGE.md

@@ -1,61 +1,43 @@
 # Bulk NFS storage — the media library and the legacy Nextcloud archive
 
-The operator-facing companion to the `class: bulk` storage entries in `catalog/{plex,sonarr,radarr,nextcloud}.yaml`.
-Everything here was **measured** on the storage host on 2026-09-07, over read-only SSH; nothing is
-assumed. It resolves the long-standing **OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
+The operator-facing companion to the `class: bulk` storage entries in
+`catalog/{plex,sonarr,radarr,qbittorrent,nextcloud}.yaml`.
+
+**Rewritten 2026-10-07 for the move off Proxmox onto a standalone TrueNAS at 192.168.2.30.** The
+layout (§2), the mount design (§3, §4) and the uid model (§6) are unchanged by the move and were
+measured on the old host on 2026-09-07. Everything about the storage host itself — §1, §5, §10 —
+was re-measured on the NAS on 2026-10-07. Nothing here is assumed. It resolves the long-standing
+**OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
 case that actually exists here — attaching datasets that already hold data — and leaves dynamic bulk
 provisioning unimplemented, because nothing needs it.
 
 ---
 
-## 1. The storage host, and why prod cannot reach it
+## 1. The storage host
 
-The bulk data lives on the **Proxmox host's ZFS pool**. That host has exactly one interface:
+The bulk data lives on a **standalone TrueNAS** (Community Edition) at **192.168.2.30** — the same
+`192.168.2.0/24` as prod's nodes (`192.168.2.10/.11/.12`), so it is reached directly, with no NAT
+and no routing decision.
 
-```
-vmbr0  192.168.1.9/24        default via 192.168.1.1   (pfSense LAN)
-```
-
-| | nodes | reaches 192.168.1.9? |
-|---|---|---|
-| **dev** | 192.168.1.40 / .41 / .42 | **Yes** — same L2 segment, and it already holds an NFS export from this host (`/mnt/Storage/k8s-volumes`, in `/etc/exports`) |
-| **prod** | 192.168.2.10 / .11 / .12 | **No** — 192.168.2.0/24 is pfSense's **WAN**. Ports 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. 2049 and 111 do not. |
-
-Opening the storage path for prod is a **pending operator decision**. Its two candidate outcomes give
-different values for the `BULK_NFS_SERVER` cluster-var:
-
-| outcome | `BULK_NFS_SERVER` for prod |
-|---|---|
-| port-forward `192.168.2.102:2049` → `192.168.1.9:2049` | `192.168.2.102` |
-| a WAN→LAN route to 192.168.1.0/24 | `192.168.1.9` |
-
-Until it lands, `clusters/prod` holds `TBD-PROD-BULK-NFS-SERVER`. **See §7 for why that placeholder
-behaves differently from the repo's other ones.**
+**This section used to be the hard part of this page.** The datasets were on the Proxmox host, whose
+only interface was `vmbr0 192.168.1.9/24` on pfSense's **LAN**, while prod sits on pfSense's **WAN**.
+tcp/2049 did not traverse that, so prod mounted through a DNAT at `192.168.2.102`, and which address
+belonged in `BULK_NFS_SERVER` depended on an undecided operator choice. Both pools were physically
+moved on 2026-10-07 (`docs/plans/proxmox-to-n5-air-migration.md`, Phase C/D) and all of that is
+simply gone. Measured 2026-10-07: **192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and
+:111 do.**
 
 ### It only needs ONE port
 
 The PVs mount `nfsvers=4.1`. **NFSv4 carries mount and lock in-protocol**, so the only thing that
 must be reachable is **tcp/2049** — no rpcbind/111, no mountd, no statd. NFSv3 would have needed
-four ports, three of them on ephemeral numbers, which is a materially worse firewall story. The
-server advertises `+3 +4 +4.1 +4.2` (`/proc/fs/nfsd/versions`), so v4.1 is available and is the
-conservative floor.
+four ports, three of them on ephemeral numbers. The NAS advertises `+3 +4 +4.1 +4.2`
+(`/proc/fs/nfsd/versions`, measured 2026-10-07), so 4.1 is available and is the conservative floor.
 
-### The export's client list is the SAME under both outcomes — measured, not assumed
-
-**pfSense DNATs the existing port-forward WITHOUT source NAT.** Connecting from this workstation
-(192.168.2.132) to `192.168.2.102:22`, the storage host logged the peer as `192.168.2.132`:
-
-```
-SSH_CONNECTION=192.168.2.132 39768 192.168.1.9 22
-ss -tn  ->  192.168.1.9:22   192.168.2.132:39768
-```
-
-The original WAN-side source address survives the translation (return traffic finds its way back via
-the host's default route, 192.168.1.1). So under **either** outcome the NFS server will see prod's
-**real** node addresses — 192.168.2.10/.11/.12 — and the `sharenfs` change in §5 is the same either
-way. **It does not have to wait for the routing decision.**
-
----
+**NFSv4 is not on by default and its absence does not look like a storage problem.** TrueNAS ships
+NFSv3-only until NFSv4 is added to *Services → NFS → Enabled Protocols*. Without it the mount never
+completes and the pods sit in `ContainerCreating` — no permission error, nothing in the app logs,
+just no volume.
 
 ## 2. The layout, as found
 
@@ -107,16 +89,18 @@ why it is mounted read-only; see §6.
 | plex | `/mnt/Storage/plex` | `/media` | — | **ro** | ReadOnlyMany |
 | sonarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
 | radarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
+| qbittorrent | `/mnt/Storage/plex` | `/downloads` | `downloads` | rw | ReadWriteMany |
 | nextcloud | `/mnt/Storage/nextcloud` | `/mnt/legacy-nextcloud` | `nextcloud` | **ro** | ReadOnlyMany |
 
 In-app paths that follow: Plex libraries at `/media/movies`, `/media/tv`, `/media/photos`; Sonarr
 Root Folder `/media/tv`; Radarr Root Folder `/media/movies`; both importing from
 `/media/downloads/complete/{tv-sonarr,radarr}`.
 
-**All three media consumers use the identical mountPath on purpose.** When qbittorrent is eventually
-given the same export (it is not in this change), the arrs and the download client will already agree
-on absolute paths, which is the second precondition for hardlinked imports — otherwise every import
-needs a Remote Path Mapping and quietly falls back to copying when one is missing.
+**The three library consumers use the identical mountPath on purpose**, and qbittorrent has since
+been given the same export (narrowed by `subPath: downloads`, presented at `/downloads` to match its
+existing `Session\DefaultSavePath`). The arrs and the download client therefore agree on absolute
+paths, which is the second precondition for hardlinked imports — otherwise every import needs a
+Remote Path Mapping and quietly falls back to copying when one is missing.
 
 **`/media` and not `/data`, and that is not cosmetic.** The arrs' Longhorn config PVC renders at the
 renderer default `/data/config`. A media mount at `/data` would be mounted **over** it and mask the
@@ -147,62 +131,123 @@ Narrowing that would cost hardlinks, which is the worse trade.
 
 ---
 
-## 5. The export change the operator must apply — STRICTLY ADDITIVE
+## 5. The exports, as configured on TrueNAS — and the settings that decide permissions
 
-**Do not apply this from the repo.** It is an operator step on the storage host, recorded here so the
-exact value is reviewable. The existing consumers **must keep working**: 192.168.1.6, 10.123.1.10 and
-192.168.1.18 on the media tree, and 192.168.1.6 and 192.168.1.5 on the Nextcloud dataset, are the
-live non-k3s prod stack.
+**Do not apply this from the repo.** It is operator configuration on the NAS, recorded here so the
+exact values are reviewable. Everything below was **measured on 192.168.2.30 on 2026-10-07**, after
+the move.
 
-Current values (`zfs get sharenfs Storage`):
+### The exports
 
 ```
-Storage/plex       rw=@192.168.1.6:@10.123.1.10:@192.168.1.18,no_subtree_check,no_root_squash
-Storage/nextcloud  [email protected]:@192.168.1.5,no_subtree_check,no_root_squash
+/mnt/Storage/plex       192.168.2.10 192.168.2.11 192.168.2.12
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)
+/mnt/Storage/nextcloud  192.168.2.10 192.168.2.11 192.168.2.12
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,root_squash,no_all_squash)
 ```
 
-### `Storage/plex` — append the six k3s node addresses to the existing `rw=` list
+Per-node addresses rather than `@192.168.2.0/24`, following the precedent of the old host. A `/24`
+would work and would survive adding a node; it also grants the whole subnet.
+
+**All three nodes must appear on BOTH shares.** Any media pod can be scheduled on any node, so a
+missing node is a latent failure, not an immediate one: it surfaces the first time a pod reschedules
+there, as `access denied by server while mounting`, with the cause weeks stale. One was caught
+exactly this way during setup — the plex share listed `192.168.2.13`, an address no node holds, and
+so was missing `192.168.2.12` (k3s-prod-3). Nothing was failing at the time.
+
+### The option flags ARE the permission contract
+
+| flag | why it matters here |
+|---|---|
+| **`no_all_squash`** | The one the whole model rests on. Client uids pass through untouched, so the arrs' `PUID=1000` arrives as the owner of a `0755 1000:3000` tree. In the UI this is **Mapall User/Group left EMPTY** — setting either squashes every client to one identity and erases the reader/writer distinction entirely. |
+| **`sec=sys`** | AUTH_SYS. Numeric uid/gid, not Kerberos principals. §6 is only true because of this. |
+| **`root_squash`** | Maproot left empty. The Proxmox export carried `no_root_squash`; **nothing here needs it** — every write to the tree happens after the LSIO drop to uid 1000, and the images only chown their Longhorn `/config`. This is a deliberate tightening over the old export, not parity. |
+| **`rw` / `ro`** | The archive's read-only posture is **server-enforced**, as it was before. The media tree cannot get the same treatment — see below. |
+| **`secure`** | Reserved source ports required; "Allow non-root mount" off. The kubelet mounts as root from a reserved port and does not need the exception. |
+
+**Plex's read-only posture is still a CLIENT-side guarantee.** Plex is ro and the arrs are rw, but
+all four can land on any node, so the server sees the same three source addresses for all of them
+and cannot distinguish the reader from the writers. `spec.nfs.readOnly: true` on `plex-media` plus
+the volumeMount flag is the whole enforcement. Unchanged by the move; an honest limitation, not an
+oversight. If it ever needs to be server-enforced, the lever is a dedicated node pool for Plex.
+
+### Service settings that are NOT obvious from the export list
+
+- **Enabled Protocols must include NFSv4** (above).
+- **Require Kerberos for NFSv4: off.** The numeric AUTH_SYS pass-through is the model.
+- **Manage Groups Server-side: off.** This is `rpc.mountd --manage-gids`: the server *discards the
+  client's supplied group list* and looks that uid's groups up in TrueNAS's own user database
+  instead. With no local uid 1000 defined, the arrs' `PGID=3000` would simply vanish. Writes would
+  survive anyway — uid 1000 **owns** the tree, so the owner bits grant access without the group —
+  but it is a free way to lose group semantics.
+- **NFSv4 DNS Domain: blank.** It only feeds `idmapd.conf`, which numeric mapping bypasses.
+- **Bind IP 192.168.2.30** means loopback does not answer: check with `showmount -e 192.168.2.30`,
+  not `localhost`.
+
+There is **no** "NFSv3 ownership model for NFSv4" control on current TrueNAS — it was an older
+TrueNAS/CORE option, and no replacement is needed: on a Linux client against a Linux server over
+AUTH_SYS, both the `nfs` and `nfsd` modules default to `nfs4_disable_idmapping=Y`, so NFSv4 carries
+raw numeric ids rather than `user@domain` principals. §9's `ls -n` probe is how you confirm it.
+
+### The ZFS `sharenfs` properties must be CLEARED, not edited
+
+The pools arrived with the old host's exports still set as dataset properties:
+
+```
+Storage/plex              [email protected]:@10.123.1.10:@192.168.1.18:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
+Storage/nextcloud         [email protected]:@192.168.1.5,[email protected]:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
+Storage/docker-volumes    [email protected]:@192.168.1.5:@192.168.1.10:@192.168.1.18,no_subtree_check,no_root_squash
+```
+
+TrueNAS generates `/etc/exports` from its own middleware database and will fight a property it did
+not set — the failure mode is two differently-configured exports of one path, with no indication
+which is in force. Cleared 2026-10-07:
 
 ```sh
-zfs set sharenfs='[email protected]:@10.123.1.10:@192.168.1.18:@192.168.1.40:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/plex
+zfs inherit sharenfs Storage/plex
+zfs inherit sharenfs Storage/nextcloud
+zfs inherit sharenfs Storage/docker-volumes
 ```
 
-Every pre-existing host is preserved verbatim and no option changes; only clients are added.
+`Storage` is `off`, so each resolves to `off`.
 
-### `Storage/nextcloud` — add the k3s nodes to a **`ro=`** list, not to `rw=`
+### Do NOT touch the dataset's permissions or ACLs
 
-```sh
-zfs set sharenfs='[email protected]:@192.168.1.5,[email protected]:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/nextcloud
-```
+The tree's ownership **is** the access story (§6), and TrueNAS offers two ways to destroy it
+irreversibly across 8.33 T.
 
-This is the one place read-only can be enforced **server-side**, and it is worth taking: only
-Nextcloud consumes this dataset from k3s, so the export itself can refuse writes. Even a
-misconfigured client cannot then damage the archive.
+`Storage/plex` carries `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`) and
+`aclinherit=passthrough` (inherited). Measured 2026-10-07: the directories carry **trivial ACLs
+only** — `nfs4xdr_getfacl` shows just `owner@`/`group@`/`everyone@` matching the mode — so `0755
+1000:3000` governs exactly as it did under Proxmox.
 
-**The media tree cannot get the same treatment.** Plex is read-only and the arrs are read-write, but
-all three may be scheduled on any node, so the server sees the same source addresses for all of
-them and cannot distinguish the reader from the writers. **Plex's read-only posture is therefore a
-client-side guarantee** (`spec.nfs.readOnly: true` on `plex-media`, plus the volumeMount flag) — an
-honest limitation, not an oversight. If it ever needs to be server-enforced, the lever is a
-dedicated node pool for Plex, which is a much larger change.
+- **Stay out of the dataset's Edit Permissions / Edit ACL screen.** On an `acltype=nfsv4` dataset
+  that is a full ACL editor, and its presets write *real* NFSv4 ACLs, recursively if permitted.
+  There is no undo.
+- `aclinherit=passthrough` means a single ACL written at `/mnt/Storage/plex` would be **inherited by
+  everything created under it afterwards** — the blast radius is the whole future library, not one
+  directory.
+- Note GNU `ls`'s `+` marker is **not** a reliable ACL indicator here: it looks for a POSIX ACL
+  xattr, and this dataset stores NFSv4 ACLs elsewhere. Use `nfs4xdr_getfacl`.
 
-Per-node addresses rather than `@192.168.2.0/24` follows the precedent already on this host
-(`/etc/exports` lists the dev nodes individually, as `/32`). A `/24` would work and would survive
-adding a node; it also grants the whole subnet. Per-node is the tighter default.
+`xattr=on` rather than `sa` is a performance detail, not a permissions one, and changing it would
+not affect existing files.
 
 ### Verifying, without changing anything
 
 ```sh
-exportfs -v                  # the new client lines appear, existing ones unchanged
-zfs get sharenfs Storage/plex Storage/nextcloud
+cat /proc/fs/nfsd/versions          # +4.1 must be present
+exportfs -v                         # the two shares, three nodes each, flags as tabulated above
+showmount -e 192.168.2.30           # NOT localhost -- the service is bound to one address
+zfs get -r -o name,property,value,source sharenfs Storage    # every row off or -
 ```
 
----
-
 ## 6. uid / gid — the failure this design exists to avoid
 
-The export uses `sec=sys` and `no_root_squash`, so **the client's numeric uid is passed through
-verbatim** and the server's on-disk ownership is the entire access story. The media tree is
+The export uses `sec=sys` and `no_all_squash`, so **the client's numeric uid is passed through
+verbatim** and the server's on-disk ownership is the entire access story. (It no longer carries
+`no_root_squash`, which the Proxmox export did — see §5. Nothing here needs it, and the uid that
+matters is 1000, not 0.) The media tree is
 `0755 1000:3000` throughout (some directories `1000:1000`), so **only uid 1000 can write it**.
 
 **`fsGroup` cannot fix this, and reaching for it is how this gets misdiagnosed.** The in-tree NFS
@@ -233,24 +278,27 @@ Nextcloud needs no uid work at all, which is one more reason its mount is read-o
 
 ---
 
-## 7. The prod placeholder does NOT fail at apply time — read this before merging
+## 7. Why this knob is unvalidated at create AND immutable at update
+
+**The `TBD-PROD-BULK-NFS-SERVER` placeholder is gone** (2026-10-07) — prod now carries a real
+address, because the NAS is on prod's own subnet and there is no longer a decision to defer. The
+hazard that made it load-bearing is worth keeping on the page, because it is the half that explains
+why §10 exists.
 
 Every other `TBD-PROD-*` in `clusters/prod/kustomization.yaml` is rejected loudly by CRD validation
-(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one is not.**
+(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one was not.**
 
-A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepts
-`TBD-PROD-BULK-NFS-SERVER`, the PV is created, and the PVC **binds** to it — binding is pure
-control-plane and never touches the server. The failure appears only when a pod mounts:
+A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepted
+`TBD-PROD-BULK-NFS-SERVER`, the PV was created, and the PVC **bound** to it — binding is pure
+control-plane and never touches the server. The failure appeared only when a pod mounted:
 
 ```
 MountVolume.SetUp failed ... Failed to resolve server TBD-PROD-BULK-NFS-SERVER
 ```
 
-so **plex, sonarr, radarr and nextcloud would sit in ContainerCreating on prod.**
-
-**Consequently: set `BULK_NFS_SERVER` in `clusters/prod/kustomization.yaml` BEFORE prod reconciles a
-revision that carries these bulk volumes.** Dev renders a working value and is where these manifests
-are meant to be proven first.
+**The same field is immutable on UPDATE** (§10), which is the other half of the same awkwardness: a
+wrong value is accepted silently at create, and then cannot be corrected in place. Together they are
+why repointing this knob is a delete-and-recreate rather than a reconcile.
 
 ### The static guard this convention leans on is missing from `main` — but the gate now refuses the value anyway
 
@@ -287,10 +335,9 @@ Without the arm, HALF A is what holds the line, by design.
    EPHEMERAL layer"). Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it
    means those apps start from an empty config volume, so it belongs in its own change with the
    operator present. **This change does not make it worse** — that is why the media mount is `/media`.
-2. **qbittorrent has no bulk mount.** Today the arrs' import source is written by the **legacy
-   non-k3s** download client, which is coherent while the legacy stack stays live. Giving the k3s
-   qbittorrent the same export at the same `/media` path is the natural follow-up, and the paths were
-   chosen so it needs no Remote Path Mapping.
+2. **RESOLVED — qbittorrent now has the bulk mount** (`subPath: downloads` at `/downloads`), so the
+   import source is written by the k3s download client rather than the legacy non-k3s one. See
+   `catalog/qbittorrent.yaml`.
 3. **`/mnt/Storage/plex/downloads/downloads` is a symlink to `/shared/downloads`**, which is outside
    the export. Over NFS it is a dangling symlink on every client. Harmless, but expect Sonarr to log
    it while scanning `downloads/`.
@@ -300,10 +347,10 @@ Without the arm, HALF A is what holds the line, by design.
 5. **`nconnect` was not set.** It would likely help throughput on the media tree, but it is a tuning
    change that should be measured, not guessed, and it changes client behaviour under server
    failure.
-6. **Nothing was verified against a live cluster.** No `kubectl` context is usable from this
-   workstation (browser-OIDC only), and this change is manifest-authoring under an explicit
-   do-not-apply constraint. Every claim about the *host* is measured; every claim about *binding and
-   mounting* is reasoned from the manifests.
+6. **Every claim about the NAS in §5 is measured (2026-10-07); the post-move MOUNT is not.** The PVs
+   still have to be recreated (§10) before any pod mounts 192.168.2.30, so the §9 probes are the
+   outstanding verification, not a formality. The pre-move equivalents did pass against the old host
+   on 2026-09-12.
 7. **The bulk-tier requirement ID is deliberately not typed on any non-`.md` line of this change.**
    `ci/touched-requirements.sh` attributes requirements from R-IDs appearing in added lines, and the
    in-cluster conformance test for the bulk tier is still a gap, so typing it would select a
@@ -312,36 +359,95 @@ Without the arm, HALF A is what holds the line, by design.
 
 ---
 
-## 9. Validating on dev, once the export is extended
+## 9. Validating, after the PVs are recreated
 
-Dev can prove all of this before prod's network path exists. Read-only checks first:
+These run against **prod**; dev is decommissioned. They are the outstanding verification for the
+move — nothing below has been run against 192.168.2.30. Read-only checks first:
 
 ```sh
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pv
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pvc -A | grep -E 'media|legacy-archive'
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr get pod -o wide
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pv -o custom-columns=\
+NAME:.metadata.name,SERVER:.spec.nfs.server,PATH:.spec.nfs.path,STATUS:.status.phase'
 ```
 
-Expect **four** PVs — `plex-media`, `sonarr-media`, `radarr-media`, `nextcloud-legacy-archive` (three
-of them naming the *same* export, one per namespace, which is the cross-namespace model) — each
-`Bound` to the same-named PVC, with `RECLAIM POLICY Retain` and an empty `STORAGECLASS`.
+Expect **five** NFS PVs — `plex-media`, `sonarr-media`, `radarr-media`, `qbittorrent-downloads`,
+`nextcloud-legacy-archive` (four of them naming the *same* export, one per namespace, which is the
+cross-namespace model) — each `Bound`, `RECLAIM POLICY Retain`, empty `STORAGECLASS`, and
+**`SERVER 192.168.2.30`**. A PV still showing `192.168.2.102` means the recreate in §10 did not
+happen; Flux will NOT have updated it in place.
 
-Then the two things that actually matter, from inside a pod:
+Then the four things that actually matter, from inside a pod:
 
 ```sh
-# (a) the writer can write — this is the uid check, and it is the one that fails in practice
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
-  sh -c 'id; touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+# (a) numeric ids survived the wire -- the idmapping check, new since the TrueNAS move
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- sh -c 'ls -n /media | head -3'
+#     expect uid 1000 / gid 3000 on the entries. 65534 / nobody means ids are being mapped, not
+#     passed through, and every import will fail EACCES while the pod still reports Ready.
 
-# (b) hardlinks work BETWEEN the library and the download path — the root-mount check
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
+# (b) the writer can write -- the uid check, and the one that fails in practice
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
+  sh -c 'touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+
+# (c) hardlinks work BETWEEN the library and the download path -- the root-mount check
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
   sh -c 'set -e; d=/media/downloads/complete/tv-sonarr/.probe; t=/media/tv/.probe-link;
          echo x > "$d"; ln "$d" "$t"; ls -i "$d" "$t"; rm -f "$d" "$t"; echo HARDLINK-OK'
 
-# (c) the reader really is read-only
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n plex exec deploy/plex -- \
-  sh -c 'touch /media/.probe 2>&1 | head -1'      # expect "Read-only file system"
+# (d) the reader really is read-only
+kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c 'touch /media/.probe 2>&1 | head -1'
+#     expect "Read-only file system"
 ```
 
-`(b)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
+`(c)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
 prints `EXDEV`, something has split the single mount back into two.
+
+Reading the failures together is what localises the fault: **(b) EACCES with (a) showing `nobody`**
+is idmapping (§5 service settings). **(b) EACCES with (a) showing `1000`** is on-disk ownership —
+stop and check `ls -ldn` on the NAS before changing anything else.
+
+---
+
+## 10. Repointing BULK_NFS_SERVER means DELETING AND RECREATING THE PVs
+
+**A bound PersistentVolume's NFS source is immutable.** Measured by server-side dry-run on
+2026-10-07:
+
+```
+$ kubectl patch pv sonarr-media --type=json --dry-run=server \
+    -p '[{"op":"replace","path":"/spec/nfs/server","value":"192.168.2.30"}]'
+The PersistentVolume "sonarr-media" is invalid: spec.persistentvolumesource: Forbidden:
+  spec.persistentvolumesource is immutable after creation
+```
+
+Note the pairing with §7: `spec.nfs.server` is **unvalidated at create** (any string is accepted and
+the PVC binds) and **immutable at update**. So editing `BULK_NFS_SERVER` alone does not fail loudly
+and does not work either — Flux's apply errors on the five PVs and the pods keep their previous,
+now-dead mount.
+
+This is safe to do because `persistentVolumeReclaimPolicy: Retain` means deleting a PV never touches
+the data, and `catalog/render.py` emits a **pre-set `claimRef`** on each one, so a recreated PV
+rebinds its own PVC with no manual step.
+
+**Operator sequence** (mutating `kubectl` is an operator action, not an agent action):
+
+```sh
+# 1. Stop the consumers. This also releases the hung hard-mounts against the old server:
+#    the pods report Ready, but any real I/O on /media blocks indefinitely once 2049 stops
+#    answering, because the PVs mount `hard`.
+for ns in plex sonarr radarr qbittorrent nextcloud; do
+  kubectl --context k3s-prod -n $ns scale deploy --all --replicas=0
+done
+
+# 2. Delete the five PVs. Retain + no live claim means nothing is reclaimed or erased.
+kubectl --context k3s-prod delete pv plex-media sonarr-media radarr-media \
+  qbittorrent-downloads nextcloud-legacy-archive
+
+# 3. Let Flux reconcile the revision carrying BULK_NFS_SERVER=192.168.2.30, which recreates them.
+#    Confirm .status.lastAppliedRevision matches the GitRepository artifact before continuing.
+
+# 4. Bring the consumers back, then run every probe in §9.
+```
+
+The PVCs pass through `Lost` between steps 2 and 3; that is expected and resolves when the PV with
+the matching `claimRef` reappears. If a PVC is still `Lost` after Flux reconciles, the PV name or
+its `claimRef` namespace does not match — do **not** delete the PVC to "fix" it.
+

docs/plans/pm-agent-issue-to-pr.md

@@ -233,9 +233,21 @@ about runtime behaviour in this document is a design claim, not a measurement.
 
 ## Follow-ups, in the order they are worth doing
 
-1. **Build, push, pin.** Operator runs `infra/agent/build-image.sh`, pushes, and one PR
-   replaces both placeholder digests. Expect the build to need iteration: the nodejs module
-   stream and the npm install have never run against this base.
+1. **Make the registry pullable, THEN build, push, pin.** Measured after the merge, and
+   neither half is declared anywhere in this repository
+   (`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`):
+   (i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the
+   router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning
+   `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same
+   pin the Keycloak issuer already carries, applied by
+   `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`;
+   (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
+   the credential decision now being local Gitea accounts with PATs, that means a
+   `dockerconfigjson` for an account whose token carries `read:package`. Only then does the
+   digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
+   placeholder digests. Expect the build itself to need iteration too: the nodejs module
+   stream and the npm install have never run against this base. The same two blockers sit in
+   front of `conformance-runner`, which shares the registry path and has never been pulled.
 2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the
    bot's permission is `write`, not `admin`** — that measurement is the control that keeps
    the agent out of its own approval gate.

infra/agent/pm-worker

@@ -90,9 +90,19 @@ esac
 # pointing at an external host would be REACHABLE: it could clone a repository that is not
 # this one and push this repository's work somewhere else. The allowlist is baked in here,
 # where changing it requires rebuilding the image whose digest admission pins.
+#
+# ONE ENTRY, AND IT IS THE CLUSTER-LOCAL SERVICE NAME. This agent runs INSIDE the cluster, so
+# it reaches Gitea the way every other in-cluster consumer does — `*.svc.cluster.local` —
+# which is by design a different path from how a client outside the cluster reaches the same
+# service. The external hostname `gitea.home.keithjackson.dev` used to be a second accepted
+# value here; it is removed deliberately. It was never needed (nothing sets it), it routes
+# in-cluster traffic out through the Gateway and back for no reason, and as an ACCEPTED value
+# it widened this allowlist to a host reachable over the `world:443` egress rule — which is
+# the exact shape of injection the allowlist exists to refuse. The cluster-local name also
+# depends on none of public DNS, Cloudflare, the Gateway or the certificate.
 case "${PM_GITEA_URL}" in
-  http://gitea.gitea.svc.cluster.local:3000|https://gitea.home.keithjackson.dev) ;;
-  *) die "PM_GITEA_URL '${PM_GITEA_URL}' is not an allowed git host (see this script's header)" ;;
+  http://gitea.gitea.svc.cluster.local:3000) ;;
+  *) die "PM_GITEA_URL '${PM_GITEA_URL}' is not the allowed in-cluster git host (see this script's header)" ;;
 esac
 [[ "${PM_GITEA_OWNER}" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,38}$ ]] || die "PM_GITEA_OWNER is malformed"
 [[ "${PM_GITEA_REPO}"  =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$ ]] || die "PM_GITEA_REPO is malformed"

infra/agent/probe-validation.sh

@@ -86,12 +86,22 @@ refuses "branch equal to the base branch"       "must not be the base branch" \
 # --- where code comes FROM and goes TO -------------------------------------
 # The namespace's egress policy permits tcp/443 to the world (the model needs it), so an
 # injected host would be REACHABLE. These are the cases that matter most.
-refuses "a redirected git host"                 "is not an allowed git host" \
+refuses "a redirected git host"                 "is not the allowed in-cluster git host" \
         PM_GITEA_URL=https://evil.invalid
-refuses "a real, reachable, WRONG git host"     "is not an allowed git host" \
+refuses "a real, reachable, WRONG git host"     "is not the allowed in-cluster git host" \
         PM_GITEA_URL=https://github.com
-refuses "the right host on a smuggled path"     "is not an allowed git host" \
+refuses "the right host on a smuggled path"     "is not the allowed in-cluster git host" \
         PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:3000/../..
+# This agent is IN the cluster, so it reaches Gitea by its cluster-local service name. The
+# EXTERNAL hostname is refused too — it used to be a second accepted value, and as an
+# accepted value it widened this allowlist to a host reachable over the `world:443` egress
+# rule, which is the injection shape the allowlist exists to refuse.
+refuses "this Gitea's EXTERNAL hostname"        "is not the allowed in-cluster git host" \
+        PM_GITEA_URL=https://gitea.home.keithjackson.dev
+refuses "the cluster-local name over https"     "is not the allowed in-cluster git host" \
+        PM_GITEA_URL=https://gitea.gitea.svc.cluster.local:3000
+refuses "the cluster-local name, wrong port"    "is not the allowed in-cluster git host" \
+        PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:8080
 refuses "a malformed owner"                     "PM_GITEA_OWNER is malformed" \
         'PM_GITEA_OWNER=../other-org'
 refuses "a malformed repository"                "PM_GITEA_REPO is malformed" \

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -327,6 +327,29 @@ node_baseline_hosts_pins:
   - ip: "192.168.2.20"
     names:
       - "keycloak.{{ k3s_base_domain }}"
+      # gitea: the SECOND node process to need this pin, for the identical reason, found
+      # 2026-10-06. containerd pulls images as a NODE process, so it resolves through the
+      # public upstreams above and gets the CNAME to wan.home.keithjackson.dev — the WAN IP,
+      # which this router does not hairpin. The pull then fails on the router's own
+      # placeholder certificate rather than on anything to do with the image:
+      #
+      #   failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:…":
+      #   tls: failed to verify certificate: x509: certificate is not valid for any names,
+      #   but wanted to match gitea.home.keithjackson.dev
+      #
+      # Measured from the workstation the same day, against the WAN address with that SNI:
+      #   subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
+      #   No extensions in certificate          <- no SANs at all, hence "not valid for ANY names"
+      # while the same name via PiHole answers 192.168.2.20 and serves the real
+      # Let's Encrypt *.home.keithjackson.dev. So the certificate and the Gateway are fine;
+      # only the node's ANSWER is wrong. Same root cause, same fix, as the issuer pin above.
+      #
+      # WHO NEEDS IT: any workload whose image lives in the in-cluster Gitea registry —
+      # today platform/pm-agent (agent-runner) and conformance/incluster
+      # (conformance-runner), neither of which has ever been pulled. A pull is still
+      # UNAUTHENTICATED after this pin, which is a separate thing to settle; this only makes
+      # the node reach the right endpoint.
+      - "gitea.{{ k3s_base_domain }}"
 
 # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
 # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/

infra/devcontainer/Containerfile

@@ -32,9 +32,23 @@ ARG TARGETARCH=amd64
 # Distro packages: libvirt client libs (harmless disk-image utilities), ansible, ssh,
 # disk-image + cloud-init seed tooling, python3-pip (for the pip-installed test tools),
 # diffutils (provides `diff`, used by L1 golden render checks), and common utilities.
+#
+# wireguard-tools + qrencode serve the REMOTE-ACCESS VPN (platform/vpn-ingress):
+#   * wireguard-tools gives `wg genkey` / `wg pubkey`, needed to mint the server key and to derive
+#     every client profile's `[Peer] PublicKey` (infra/vpn-ingress/new-peer.sh).
+#   * qrencode renders a client profile as a QR code, which is how a PHONE gets enrolled.
+# They are here rather than on the operator's host because this image is THE toolchain
+# (CLAUDE.md §5; README "Don't install the toolchain on the host"). That rule was briefly broken
+# by a runbook that told the operator to run `wg` on the host and `sops` in here, and then split
+# every step across the boundary to work around it — a procedure nobody should have to follow, and
+# one that cannot be reproduced in CI. Both tools in one image means `sops -d … | wg pubkey` is
+# just a pipeline again.
+# Unpinned on purpose, like every other dnf package here: the Fedora base is what is pinned
+# (FEDORA_VERSION), and these are not gate tools whose version changes what green means.
 RUN dnf -y install --setopt=install_weak_deps=False \
       libvirt-client qemu-img cloud-utils genisoimage openssh-clients \
       ansible-core python3-libvirt python3-pip diffutils \
+      wireguard-tools qrencode \
       git make jq unzip tar gzip xz which ca-certificates curl \
  && dnf clean all
 

infra/devcontainer/README.md

@@ -14,6 +14,7 @@ baseline, and the *same* binaries CI uses.
 | Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
 | GitOps / cluster | `flux`, `kubectl`, `helm` |
 | Secrets | `sops`, `age` + `age-keygen` |
+| VPN (`platform/vpn-ingress`) | `wg` (wireguard-tools) — mint the server key, derive client public keys; `qrencode` — render a client profile as a phone-scannable QR |
 | **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
 | Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |
 

infra/vpn-ingress/new-peer.sh

@@ -2,10 +2,24 @@
 # =================================================================================================
 # new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
 #
-# Generates a WireGuard keypair ON THIS MACHINE, renders the device's client profile, optionally
-# prints a QR code for a phone, and prints the one `[Peer]` stanza to paste into
-# platform/vpn-ingress/peers.yaml. The private half NEVER leaves this machine: it is not sent to the
-# cluster, not committed, and not written to any log.
+# Generates a WireGuard keypair, renders the device's client profile, prints a QR code for a phone,
+# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.conf. The device's
+# private half is written to ONE 0600 file and nowhere else: not sent to the cluster, not committed,
+# never logged.
+#
+# RUN IT IN THE DEV CONTAINER — that is where the toolchain lives (CLAUDE.md §5):
+#
+#     ./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2'
+#
+# It needs `wg` (wireguard-tools), `qrencode` and `sops`, all three baked into the image. It will
+# run on a host that happens to have them too, but the container is the supported way and the only
+# one that is reproducible.
+#
+# THE SERVER'S PUBLIC KEY IS DERIVED FROM THE ENCRYPTED FILE, not from a copy you keep lying around:
+# `sops -d` the private half out of platform/vpn-ingress/wg-server-key.sops.yaml and pipe it
+# straight into `wg pubkey`. The plaintext never touches disk and never reaches a terminal. (This
+# is why both tools being in ONE image matters — see the Containerfile comment.) Override with
+# WG_SERVER_PUBKEY or WG_SERVER_KEY_FILE if you are working off-tree.
 #
 # WHY A SCRIPT AND NOT A RUNBOOK STEP. Three of these fields are easy to get wrong in ways that are
 # either silently insecure or silently broken, and all three are mechanical:
@@ -25,24 +39,30 @@
 #   <last-octet>    2-254, the device's address inside WG_CLIENT_CIDR. Must be unused.
 #   --split         split tunnel: route ONLY the LAN and the tunnel subnet. Default is a FULL
 #                   tunnel (0.0.0.0/0) so the device also egresses the house WAN.
-#   --out DIR       where to write the profile (default: a 0700 dir under $TMPDIR).
+#   --out DIR       where to write the profile. Default: `.cache/wg-peers/` in the repo, which is
+#                   GITIGNORED (`.gitignore` `.cache/`) and, unlike /tmp inside the container,
+#                   survives the container exiting — the repo is its only persistent read-write
+#                   mount, so a profile written anywhere else is gone before you can move it to a
+#                   desktop. The script refuses any destination git does not ignore.
 #
 # ENVIRONMENT
-#   WG_SERVER_KEY_FILE  server PRIVATE key (default ~/.config/wireguard-home/server.key); its public
-#                       half is derived from it. Alternatively set WG_SERVER_PUBKEY directly.
+#   WG_SERVER_PUBKEY    the server's public key, if you want to skip the SOPS decrypt entirely.
+#   WG_SERVER_KEY_FILE  a local copy of the server PRIVATE key to derive the public half from,
+#                       instead of reading the SOPS file.
+#   WG_SERVER_SOPS_FILE default platform/vpn-ingress/wg-server-key.sops.yaml
 #   WG_ENDPOINT_HOST    default wan.home.keithjackson.dev
 #   WG_CLIENT_CIDR      default 10.90.0.0/24   (must match clusters/base/cluster-vars.yaml)
 #   WG_LAN_CIDR         default 192.168.2.0/24 (prod LAN_CIDR)
 #   WG_DNS              default 192.168.2.23   (prod PIHOLE_DNS_VIP)
 #
-# REQUIRES: wireguard-tools (`wg`). `qrencode` is optional and only used for the QR code.
+# REQUIRES: `wg`, `qrencode`, `sops` — all three in the dev container.
 # =================================================================================================
 set -euo pipefail
 
 die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; }
 
 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
+PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
 
 # --- defaults, each mirroring a value that is DECLARED elsewhere in this repo -------------------
 # These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to
@@ -51,7 +71,7 @@ PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
 : "${WG_CLIENT_CIDR:=10.90.0.0/24}"                 # clusters/base/cluster-vars.yaml
 : "${WG_LAN_CIDR:=192.168.2.0/24}"                  # clusters/prod LAN_CIDR
 : "${WG_DNS:=192.168.2.23}"                         # clusters/prod PIHOLE_DNS_VIP
-: "${WG_SERVER_KEY_FILE:=${HOME}/.config/wireguard-home/server.key}"
+: "${WG_SERVER_SOPS_FILE:=${REPO_ROOT}/platform/vpn-ingress/wg-server-key.sops.yaml}"
 # wan.<base-domain> is an UNPROXIED A record that platform/externaldns/ddns.yaml keeps pointed at
 # the current WAN IP (checked every 5 min). Use it, never a literal address: this house has a
 # DYNAMIC WAN IP, and a profile pinned to today's address stops working without warning and with no
@@ -83,14 +103,28 @@ printf '%s' "$NAME" | grep -Eq '^[a-z0-9][a-z0-9-]{0,30}$' \
 printf '%s' "$OCTET" | grep -Eq '^[0-9]+$' || die "octet must be a number: got '$OCTET'"
 [ "$OCTET" -ge 2 ] && [ "$OCTET" -le 254 ] || die "octet must be 2-254 (.1 is the server): got '$OCTET'"
 
-command -v wg >/dev/null || die "wireguard-tools not installed ('wg' not on PATH). Debian/Ubuntu: apt install wireguard-tools"
+command -v wg >/dev/null || die "'wg' not on PATH. Run this inside the dev container:
+  ./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh $NAME $OCTET'"
 
 # --- the server's public key --------------------------------------------------------------------
+# Order: an explicit pubkey, then a local private-key file, then the SOPS file in the tree. The
+# last is the normal path and needs no local copy of anything.
 if [ -z "${WG_SERVER_PUBKEY:-}" ]; then
-  [ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read server key '$WG_SERVER_KEY_FILE'.
-  Either point WG_SERVER_KEY_FILE at it, or set WG_SERVER_PUBKEY to the public half.
-  If the server key does not exist yet, see runbooks/vpn-remote-access.md step 1."
-  WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
+  if [ -n "${WG_SERVER_KEY_FILE:-}" ]; then
+    [ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read WG_SERVER_KEY_FILE '$WG_SERVER_KEY_FILE'"
+    WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
+  else
+    command -v sops >/dev/null || die "'sops' not on PATH. Run this inside the dev container (see --help)."
+    [ -r "$WG_SERVER_SOPS_FILE" ] || die "cannot read '$WG_SERVER_SOPS_FILE'"
+    # The decrypted private key goes straight into `wg pubkey` — never to a file, never to a tty.
+    # `wg pubkey` fails on the committed REPLACE-ME placeholder, which is the useful error here.
+    WG_SERVER_PUBKEY="$(sops -d --extract '["stringData"]["privatekey"]' "$WG_SERVER_SOPS_FILE" \
+                          | tr -d ' \r\n' | wg pubkey 2>/dev/null)" || WG_SERVER_PUBKEY=""
+    [ -n "$WG_SERVER_PUBKEY" ] || die "could not derive the server public key from
+  $WG_SERVER_SOPS_FILE
+  Either it still holds the REPLACE-ME placeholder (see runbooks/vpn-remote-access.md step 2),
+  or this shell cannot decrypt it (the age keyring is mounted at ~/.config/sops in the container)."
+  fi
 fi
 
 CLIENT_BASE="${WG_CLIENT_CIDR%/*}"; CLIENT_BASE="${CLIENT_BASE%.*}"
@@ -99,22 +133,41 @@ CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
 # --- refuse a collision, because the failure is otherwise invisible -----------------------------
 # Two devices sharing an address does not error anywhere: the server routes that address to
 # whichever peer handshook most recently, so BOTH devices appear to work intermittently.
-if [ -r "$PEERS_FILE" ] && grep -q "AllowedIPs *= *${CLIENT_ADDR}/32" "$PEERS_FILE"; then
-  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
+# COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the
+# COMMENTED-OUT EXAMPLE STANZA in the register, which made `.2` — the first address anyone would
+# ask for — permanently un-allocatable with a message claiming it was already taken. Strip
+# comments first: a real stanza's `AllowedIPs` line is never commented, while both the template
+# and every peer's `# <name> — <owner>` label are.
+# FAIL if the register is missing rather than skipping the check. The path was silently wrong once
+# (it still said peers.yaml after the file became peers.conf), and because the old guard was
+# `[ -r "$PEERS_FILE" ] && ...` the only symptom would have been duplicate detection quietly
+# never firing again — the exact silent-skip shape this repo treats as a defect.
+[ -r "$PEERS_FILE" ] || die "cannot read the peer register '$PEERS_FILE'.
+  Duplicate-address detection cannot run, so this script refuses to issue a profile."
+if grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
+  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.conf. Pick another octet."
 fi
-if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
-  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2
+if grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
+  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.conf.\n' "$NAME" >&2
 fi
 
 # --- output location: 0700, and never inside the repo -------------------------------------------
-if [ -z "$OUT_DIR" ]; then
-  OUT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/wg-peer-XXXXXX")"
-else
-  mkdir -p "$OUT_DIR"
-fi
+[ -n "$OUT_DIR" ] || OUT_DIR="${REPO_ROOT}/.cache/wg-peers"
+mkdir -p "$OUT_DIR"
 chmod 700 "$OUT_DIR"
-case "$(cd "$OUT_DIR" && pwd)" in
-  "$REPO_ROOT"|"$REPO_ROOT"/*) die "refusing to write a private key inside the git checkout ($OUT_DIR)" ;;
+OUT_ABS="$(cd "$OUT_DIR" && pwd)"
+# A profile holds a private key, so the destination must not be committable. Inside the repo that
+# means git must IGNORE it — test the real property rather than a proxy, because the repo is the
+# container's only persistent read-write mount and so the only sane place to write. Outside the
+# repo, anywhere goes.
+case "$OUT_ABS" in
+  "$REPO_ROOT"|"$REPO_ROOT"/*)
+    if ! git -C "$REPO_ROOT" check-ignore -q "$OUT_ABS/probe.conf" 2>/dev/null; then
+      die "refusing to write a private key to '$OUT_ABS': it is inside the git checkout and NOT
+  gitignored, so the profile would be committable. Use the default (.cache/wg-peers) or add an
+  ignore rule first. Verify with: git check-ignore -v <path>"
+    fi
+    ;;
 esac
 PROFILE="${OUT_DIR}/${NAME}.conf"
 
@@ -162,7 +215,7 @@ cat <<EOF
   Profile     : ${PROFILE}   (0600; contains the private key)
 
   ------------------------------------------------------------------------------------------
-  STEP 1 — add this to platform/vpn-ingress/peers.yaml, under data.peers.conf, then PR it:
+  STEP 1 — append this to platform/vpn-ingress/peers.conf, then PR it:
 
     # ${NAME} — <owner>, enrolled $(date -u +%Y-%m-%d)
     [Peer]
@@ -181,7 +234,7 @@ cat <<EOF
   STEP 3 — delete the profile from this machine once the device has it:
     shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE}
 
-  The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet
+  The tunnel will not work until the peers.conf PR is MERGED and Flux has rolled the DaemonSet
   (~1 min). Until then the device handshakes and gets silence, exactly like an unknown key.
   ------------------------------------------------------------------------------------------
 EOF
@@ -190,6 +243,8 @@ if command -v qrencode >/dev/null; then
   printf '  QR code (this IS the private key — your terminal scrollback now holds a credential):\n\n'
   qrencode -t ansiutf8 < "$PROFILE"
 else
-  printf '  qrencode not installed, so no QR code was printed (Debian/Ubuntu: apt install qrencode).\n'
-  printf '  Transfer %s to the device by another route instead.\n' "$PROFILE"
+  # Should not happen in the dev container: qrencode is baked into the image. Kept as a real
+  # fallback for a host run, rather than failing after the profile has already been written.
+  printf '  qrencode not on PATH, so no QR code was printed. It is in the dev container —\n'
+  printf '  rerun there, or transfer %s to the device by another route.\n' "$PROFILE"
 fi

platform/keycloak/README.md

@@ -154,6 +154,74 @@ NEW client, a new protocol mapper or a changed scope still needs the realm impor
 (it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
 a realm edit landed.
 
+## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17)
+
+`keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
+policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
+**ALTERNATIVE** beside the password form. Design decisions and their reasoning are in the manifest
+comments, not repeated here. The headline choices:
+
+| Decision | Value | One-line reason |
+|---|---|---|
+| Additive or replacing? | **Additive** — passkey *or* password | A misfiring mandatory flow locks the single operator out of every SSO app *and* kubectl |
+| Enrolment | **Optional**, never a default action | Same reason; `webauthn-register-passwordless` is already enabled-not-forced by default |
+| Relying-party ID | `${BASE_DOMAIN}` (not `keycloak.${BASE_DOMAIN}`) | An unre-keyable credential must not be bound to a hostname that can move |
+| User verification | `required` | Without it, mere possession of an unlocked device is a complete login |
+| Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
+| OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |
+
+### This change does NOT reach the live realm. At all.
+
+Same one-shot failure this file already documents twice above, third field over. Measured on prod
+2026-10-17:
+
+```
+$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
+{"conditions":[{"status":"True","type":"Done"},{"status":"False","type":"Started"},
+               {"status":"False","type":"HasErrors"}]}
+```
+
+`Done=True` / `Started=False` — the import has completed and **will not re-run**, so a policy block
+and three new sub-flows added to this file change nothing about how anyone logs in. The
+client-secret/client-URL reconciler does **not** close this gap and was never meant to: its hard
+endpoint allow-list makes `/authentication` unreachable **by design** (see the table above), and
+realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
+mapper or a changed scope still needs the realm import to actually re-run" case named above.
+
+**So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until
+an operator applies it by hand.** Two paths, both operator actions:
+
+1. **Admin console (expected path).** *Authentication → Policies → WebAuthn Passwordless Policy*
+   for the policy block, then *Authentication → Flows → browser-mfa* to rebuild the executions to
+   match the manifest. The flow edit is the fiddly half: split the existing Username Password Form
+   into Username Form + Password Form and add the two sub-flows exactly as declared.
+2. **A deliberate re-import.** Deleting and recreating the `KeycloakRealmImport` re-runs it against
+   the **existing** realm. Do not reach for this casually — it is the operation whose failure mode
+   is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.
+
+Either way the realm is then configured by hand and git is only the record of intent, which is a
+live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it.
+
+### Verifying it actually works (nobody has, as of 2026-10-17)
+
+No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
+it, in order:
+
+```sh
+# 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
+#    Authentication -> Policies -> WebAuthn Passwordless Policy, in the admin console.
+# 2. enrol: account console (https://keycloak.${BASE_DOMAIN}/realms/homelab/account) -> Signing in
+#    -> Passwordless -> set up. A browser/OS passkey prompt must appear; if it does not, the RP ID
+#    is wrong for the origin and NOTHING will say so.
+# 3. log in: a fresh private window to any SSO-gated app. Expect username screen, then a passkey
+#    prompt. A platform-admin must NOT be asked for TOTP on this path.
+# 4. regression (the important one): a user with NO passkey must still get the password screen,
+#    and a platform-admin must still be asked for TOTP there.
+```
+
+A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak
+is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.
+
 ## Exceptions that keep native auth (do not force SSO) — ADR-0014
 
 Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).

platform/keycloak/keycloak-realm-import.yaml

@@ -74,6 +74,142 @@ spec:
     bruteForceProtected: true
     permanentLockout: false
     sslRequired: external
+    # ===========================================================================================
+    # PASSKEYS — the WebAuthn PASSWORDLESS policy (R-IDP-04). 2026-10-17.
+    # ===========================================================================================
+    # Read this together with the browser-mfa flow at the bottom of the file: THIS block decides
+    # what a passkey must BE, the FLOW decides when one is OFFERED. Keycloak keeps TWO independent
+    # WebAuthn policies per realm and they are not interchangeable:
+    #
+    #   webAuthnPolicy*             -> the TWO-FACTOR policy, consumed by the `webauthn-authenticator`
+    #                                  authenticator (a security key presented AFTER a password).
+    #   webAuthnPolicyPasswordless* -> the PASSWORDLESS policy, consumed by
+    #                                  `webauthn-authenticator-passwordless` (a passkey presented
+    #                                  INSTEAD of a password). This is the one passkeys use.
+    #
+    # Setting the wrong family is a silent no-op: the passwordless authenticator reads
+    # getWebAuthnPolicyPasswordless() only (WebAuthnPasswordlessAuthenticator.getWebAuthnPolicy),
+    # so a policy written into the two-factor fields is simply never consulted by a passkey login.
+    # Only the passwordless family is declared here; the two-factor family is deliberately left at
+    # Keycloak's defaults because nothing in this realm uses `webauthn-authenticator` — TOTP
+    # (auth-otp-form) is the second factor for platform-admins, see the flow section.
+    #
+    # ADDITIVE AND OPTIONAL — THE DECISION THIS BLOCK EXISTS TO MAKE
+    # -------------------------------------------------------------
+    # Passkeys are offered ALONGSIDE the password, never instead of it, and enrolment is NOT forced.
+    # This is a single-operator homelab whose IdP gates Grafana, Gitea, Nextcloud, Headlamp,
+    # Semaphore, PiHole, Crafty, the *arr stack AND kubectl itself (the `kubernetes` client below).
+    # A mandatory passkey flow that misfires is therefore not an inconvenience, it is a total
+    # lock-out from the whole platform AND from the cluster API in one step — and the only repair
+    # paths left would be the Keycloak bootstrap admin and the break-glass cert kubeconfig. The
+    # expected gain from forcing enrolment (one human stops typing a password) is small; the
+    # expected loss is the platform. So: additive, optional, and password login keeps working for
+    # every user who never enrols.
+    #
+    # Concretely that means: no `requiredActions` block is declared anywhere in this file (see the
+    # long note in the flow section for why declaring one would be actively DANGEROUS), and the
+    # passkey step sits as an ALTERNATIVE beside the password form rather than ahead of it.
+    #
+    # RELYING-PARTY ID — THE FIELD THAT SILENTLY BREAKS EVERY PASSKEY IF IT IS WRONG
+    # ------------------------------------------------------------------------------
+    # The RP ID is baked into every credential at enrolment and can never be changed afterwards:
+    # a passkey created under one RP ID is unusable under another, with no error the user can act
+    # on — the browser simply reports that no credential is available. It is also browser-enforced,
+    # so it cannot be "fixed" server-side: per WebAuthn, the RP ID must be equal to, or a
+    # registrable-domain suffix of, the origin's effective domain. Keycloak is served at
+    # keycloak.<BASE_DOMAIN> (httproute.yaml, and the Keycloak CR's hostname), so the legal choices
+    # are keycloak.<BASE_DOMAIN>, <BASE_DOMAIN>, or the registrable domain one level up. It is NOT
+    # the full URL and NOT a scheme — "https://..." here would fail every ceremony.
+    #
+    # We choose BASE_DOMAIN, the shared platform suffix, NOT the keycloak host, for one concrete
+    # reason drawn from this repo's own history: a credential that cannot be re-keyed must not be
+    # bound to a name that can move. The oauth2-proxy client below ALREADY declares a redirect URI
+    # on auth.<BASE_DOMAIN>, i.e. this platform already contemplates serving auth on a second
+    # hostname; and the client-URL reconciler exists precisely because moving a hostname in git
+    # silently fails to reach the realm (see README, "Client URLs: the same one-shot failure").
+    # Binding passkeys to the keycloak subdomain would mean that relocating Keycloak invalidates
+    # every enrolled passkey; binding them to BASE_DOMAIN survives it. The cost is scope: any
+    # future WebAuthn relying party under this base domain could assert the same RP ID. In a
+    # homelab where every such party is already behind THIS IdP that is not a trust boundary worth
+    # buying. Note the residual, unavoidable risk: changing BASE_DOMAIN itself invalidates every
+    # passkey, which is a re-enrolment, not a repair.
+    #
+    # Written as the cluster-var so it is substituted by the same Flux postBuild pass, with the same
+    # value, as every hostname in this file — there is no second place the domain is decided.
+    webAuthnPolicyPasswordlessRpId: ${BASE_DOMAIN}
+    # The human-readable name the browser/OS shows in the passkey prompt and stores beside the
+    # credential ("Save a passkey for Homelab SSO?"). Matches the realm displayName above so the
+    # OS keychain entry is recognisable. It is also load-bearing on the UPDATE path: Keycloak's
+    # updateRealm applies the passwordless policy ONLY when this field is non-empty
+    # (DefaultExportImportManager line ~842), so an admin-API realm update that omits it drops the
+    # whole policy silently.
+    webAuthnPolicyPasswordlessRpEntityName: "Homelab SSO"
+    # ES256 then RS256 — exactly Keycloak's own DEFAULT_POLICY, declared explicitly rather than
+    # inherited so the realm's crypto is auditable from git. ES256 is what every platform
+    # authenticator (Touch ID/Face ID, Windows Hello, Android) and every FIDO2 key produces;
+    # RS256 is the fallback for older Windows Hello TPM-backed credentials. EdDSA/Ed25519 is
+    # deliberately NOT offered: authenticator and browser support is uneven, and an authenticator
+    # that picks an algorithm the verifier handles badly fails at enrolment, which is the worst
+    # moment to discover it. Order is a preference list, not an allow-list ordering requirement.
+    webAuthnPolicyPasswordlessSignatureAlgorithms:
+      - ES256
+      - RS256
+    # DISCOVERABLE CREDENTIAL REQUIRED. "Yes" is a Keycloak sentinel, not a boolean, and the login
+    # theme compares it as the literal string: webauthnRegister.js maps 'Yes' to
+    # authenticatorSelection.requireResidentKey=true and ANY other non-sentinel value to false.
+    # This is not optional for passkeys — a non-discoverable credential cannot be found by the
+    # browser without the server first naming it, so it is a second factor, not a passkey.
+    # (Keycloak 26.0.6 emits only the LEGACY requireResidentKey member, not the newer
+    # residentKey:"required"; per WebAuthn L2 the client maps requireResidentKey=true onto
+    # residentKey="required", so discoverable credentials are genuinely requested — but it also
+    # means this policy cannot express residentKey:"preferred" at all.)
+    webAuthnPolicyPasswordlessRequireResidentKey: "Yes"
+    # USER VERIFICATION REQUIRED. The authenticator must verify the human (biometric or PIN) and
+    # say so in the assertion. This is the field that makes a passkey login at least as strong as
+    # the password it replaces: with verification merely "preferred", possession of an unlocked
+    # phone or an unattended security key would be a COMPLETE login to every SSO-gated app — a
+    # single-factor possession check, strictly weaker than the password+TOTP path beside it.
+    # Required makes the passkey branch inherently two-factor (something you have + something you
+    # are/know), which is the premise the flow section relies on when it does NOT ask a
+    # passkey-authenticated platform-admin for TOTP as well.
+    webAuthnPolicyPasswordlessUserVerificationRequirement: required
+    # "not specified" is Keycloak's sentinel for "either kind of authenticator"
+    # (Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED; the theme JS omits the WebAuthn member
+    # entirely when it sees this exact string). Deliberately NOT narrowed: "platform" would reject
+    # every YubiKey and every phone used as a cross-device passkey, and "cross-platform" would
+    # reject Touch ID and Windows Hello — i.e. either choice silently halves the usable hardware.
+    # The operator should be able to enrol a laptop AND a phone AND a hardware key.
+    webAuthnPolicyPasswordlessAuthenticatorAttachment: "not specified"
+    # ATTESTATION: none. Requesting "direct" would hand Keycloak an attestation statement it has
+    # no trust anchors to validate here (no metadata service is configured) and nothing to compare
+    # it against, since the AAGUID allow-list below is deliberately empty — so the statement would
+    # be collected, unverifiable, and would add an extra "share model information?" consent prompt
+    # on Safari/iOS for no security gain. Attestation is how you ENFORCE an approved-hardware list;
+    # we do not have one, so asking for it is cost without benefit.
+    webAuthnPolicyPasswordlessAttestationConveyancePreference: none
+    # Empty = accept any authenticator model. An AAGUID allow-list is a hardware whitelist: it
+    # would have to be maintained by hand, and every new phone, laptop or key the operator buys
+    # would be rejected at enrolment with a message that does not explain itself. It is also moot
+    # while attestation is "none" (above) — with no attestation statement there is no trustworthy
+    # AAGUID to match against in the first place.
+    webAuthnPolicyPasswordlessAcceptableAaguids: []
+    # Ceremony timeout in SECONDS (the login theme multiplies by 1000; 0 means "send no timeout and
+    # let the browser decide"). 300s is chosen over a tighter value because the slowest LEGITIMATE
+    # path is the cross-device hybrid flow — Keycloak renders a QR code, the operator unlocks a
+    # phone, scans it, and completes a biometric check — which routinely takes over a minute. A
+    # 60-120s timeout looks tidy and converts that flow into an unexplained failure. Browsers clamp
+    # this value (Chrome to at most 10 minutes), so 300 is honoured rather than reinterpreted.
+    webAuthnPolicyPasswordlessCreateTimeout: 300
+    # Send the user's existing passkeys as excludeCredentials at enrolment, so re-enrolling an
+    # authenticator the user already registered is refused by the authenticator itself instead of
+    # producing a second, indistinguishable credential in the account console. Cheap, and it keeps
+    # "which passkey is which" answerable — which matters when the recovery story is "delete the
+    # lost one".
+    webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister: true
+    # webAuthnPolicyPasswordlessExtraOrigins is intentionally NOT set. It exists to admit non-web
+    # origins (native app facet IDs, e.g. android:apk-key-hash:...). Every ceremony here happens in
+    # a browser on Keycloak's own origin, which is already covered by the RP ID above; adding
+    # origins is only ever a widening of who may ask for these credentials.
     # --- Authorization model (R-IDP-06): groups drive access; platform-admins carry a realm role. ---
     roles:
       realm:
@@ -528,14 +664,105 @@ spec:
               "access.token.claim": "true"
               "userinfo.token.claim": "true"
               "claim.name": groups
-    # --- MFA for platform-admins (R-IDP-04): a browser flow whose OTP step is CONDITIONAL on the
-    #     platform-admin role. Non-admins log in without OTP; platform-admins must enroll/present TOTP.
-    #     Set as the realm browserFlow below. VERIFY at bring-up (see runbook): a malformed flow fails
-    #     the whole import — if so, the fallback is to comment this block + browserFlow and configure
-    #     the conditional-OTP-by-role flow via the admin console. ---
+    # ===========================================================================================
+    # MFA for platform-admins (R-IDP-04) + PASSKEY login (2026-10-17). The realm browserFlow is
+    # bound to browser-mfa at the bottom of this file.
+    # ===========================================================================================
+    # Non-admins log in without OTP; platform-admins must enroll/present TOTP. VERIFY at bring-up
+    # (see runbook): a malformed flow fails the whole import — if so, the fallback is to comment
+    # this block + browserFlow and configure the flow via the admin console.
+    #
+    # THE SHAPE, AND WHY IT IS THIS SHAPE
+    # -----------------------------------
+    #   browser-mfa                           (top level, bound as browserFlow)
+    #     auth-cookie                     ALTERNATIVE   already-authenticated SSO cookie
+    #     identity-provider-redirector    ALTERNATIVE   (no IdPs configured; kept for parity)
+    #     browser-mfa-forms               ALTERNATIVE   the interactive path:
+    #       auth-username-form              REQUIRED      WHO are you (no password yet)
+    #       browser-mfa-passkey-or-password REQUIRED      ...and PROVE it, one of two ways:
+    #         webauthn-authenticator-passwordless  ALTERNATIVE   a passkey, or
+    #         browser-mfa-password-and-otp         ALTERNATIVE   the password path:
+    #           auth-password-form                   REQUIRED      password, then
+    #           browser-mfa-conditional-otp          CONDITIONAL   TOTP iff platform-admin
+    #
+    # WHY THE USERNAME/PASSWORD SCREEN HAD TO BE SPLIT IN TWO. The previous flow used the combined
+    # `auth-username-password-form`, which collects both on one page. A passkey branch cannot live
+    # behind that: the server must know WHICH user is logging in before it can decide whether a
+    # passkey is even available, and the combined form has already taken the password by the time
+    # it knows. So it is replaced by the `auth-username-form` / `auth-password-form` pair — the
+    # same decomposition Keycloak's own passwordless recipe uses. THIS IS A VISIBLE UX CHANGE FOR
+    # EVERY USER, passkey or not: login becomes two screens (username, then password) instead of
+    # one. That cost is accepted deliberately — it is the enabling change, it is what Google and
+    # Microsoft logins already do, and the alternative (below) is worse.
+    #
+    # WHY NOT USERNAMELESS ("just tap your passkey", no username screen at all). Keycloak 26.0.6
+    # does support it: WebAuthnPasswordlessAuthenticator.requiresUser() returns FALSE, so the
+    # authenticator can legally run as a first step with no user in context. We deliberately do not
+    # do that, because of how the flow engine then behaves. In DefaultAuthenticationFlow the
+    # "does this user actually have this credential?" guard sits INSIDE the requiresUser() branch;
+    # with requiresUser() false and no user yet set, the engine skips that guard and simply invokes
+    # the authenticator. A passkey execution placed before the username form is therefore offered
+    # to EVERYONE, so every login — including every password-only user and every family member who
+    # will never own a passkey — would open on a passkey prompt they must dismiss. That is the
+    # opposite of additive. Keeping the passkey step AFTER auth-username-form means a user is in
+    # context, which is what makes the graceful degradation below actually work.
+    #
+    # WHY THE PASSKEY BRANCH DEGRADES CLEANLY TO A PASSWORD (the property the whole "additive"
+    # claim rests on). With a user in context, the engine takes the non-requiresUser branch and
+    # raises CREDENTIAL_SETUP_REQUIRED when the user has no credential of this type and the
+    # authenticator's factory does not allow self-setup — and WebAuthnPasswordlessAuthenticatorFactory
+    # .isUserSetupAllowed() is false, and the authenticator is a CredentialValidator, so both
+    # conditions hold. That exception is raised INSIDE the ALTERNATIVE loop, which catches it,
+    # marks the execution ATTEMPTED and proceeds to the next alternative. Net effect: a user with
+    # no passkey never sees a passkey screen and lands on the password form; a user WITH one gets
+    # the passkey screen and keeps the password as a "Try another way" option. Note that for a
+    # user holding BOTH, which screen appears FIRST is decided by Keycloak's authentication
+    # selection list (the user's own credential ordering), not by the priorities below.
+    #
+    # WHY A PASSKEY LOGIN IS NOT ALSO ASKED FOR TOTP. browser-mfa-conditional-otp hangs off the
+    # PASSWORD branch only, so a platform-admin who authenticates with a passkey is not challenged
+    # for TOTP. That is intentional, and it rests on the policy above: userVerification is
+    # `required`, so the authenticator has already proved possession of hardware AND verified the
+    # human (biometric or PIN) in one phishing-resistant, origin-bound ceremony. Stacking TOTP on
+    # top would add a replayable, phishable shared secret to a credential that is neither, for
+    # friction on every login. If a future operator disagrees, the change is small and local: move
+    # the browser-mfa-conditional-otp reference out of browser-mfa-password-and-otp and into
+    # browser-mfa-passkey-or-password's parent as a REQUIRED step, so it applies to both branches.
+    #
+    # WHY THERE IS NO `requiredActions` BLOCK IN THIS FILE — DO NOT ADD ONE
+    # --------------------------------------------------------------------
+    # Enrolment uses Keycloak's `webauthn-register-passwordless` required action, and the obvious
+    # move is to declare it here. DO NOT. In Keycloak 26.0.6 the realm import treats
+    # requiredActions as REPLACE, NOT MERGE: DefaultExportImportManager.importRealm branches on
+    # `if (rep.getRequiredActions() != null)` and, when the representation declares ANY, imports
+    # exactly those plus delete_account — the `else` branch that installs the full default set
+    # (DefaultRequiredActions.addActions) never runs. RealmManager.setupRequiredActions is no
+    # backstop either: it fires only when the realm has ZERO required-action providers.
+    #
+    # So declaring one action would create this realm with that action and delete_account and
+    # NOTHING ELSE — no CONFIGURE_TOTP, no UPDATE_PASSWORD, no VERIFY_EMAIL, no delete_credential,
+    # no UPDATE_PROFILE. The first casualty would be the MFA requirement this very flow enforces:
+    # with CONFIGURE_TOTP unregistered, a platform-admin who does not yet have TOTP could never
+    # enrol one, while auth-otp-form below still demands it — a hard lock-out of every
+    # platform-admin from every SSO-gated app, including Headlamp and kubectl, delivered by a
+    # change whose stated purpose was to ADD a login method.
+    #
+    # And the declaration buys nothing, because the default set ALREADY CONTAINS IT.
+    # DefaultRequiredActions registers `webauthn-register-passwordless` with enabled=true,
+    # defaultAction=false, priority 80, gated on Profile.Feature.WEB_AUTHN — which is Type.DEFAULT
+    # in 26.0.6, i.e. on unless explicitly disabled. enabled=true / defaultAction=false IS the
+    # additive-and-optional posture this change wants: the action exists and can be triggered for
+    # a user on demand, but it is not forced on anyone at next login. Declaring it would restate
+    # the default while deleting ten siblings.
+    #
+    # (The symmetric question for authenticationFlows has the opposite answer, which is why THIS
+    # block is safe: importAuthenticationFlows adds the declared flows and then backfills each
+    # unbound binding — browser, registration, direct grant, reset credentials, client auth, first
+    # broker login, docker auth — from the built-in defaults individually. Declaring flows does not
+    # suppress the built-ins; declaring required actions does suppress them.)
     authenticationFlows:
       - alias: browser-mfa
-        description: "Browser flow with conditional OTP required for platform-admins."
+        description: "Browser flow: passkey or password, with conditional OTP for platform-admins."
         providerId: basic-flow
         topLevel: true
         builtIn: false
@@ -555,13 +782,63 @@ spec:
             priority: 30
             flowAlias: browser-mfa-forms
             userSetupAllowed: false
+      # Identify the user FIRST (username only), then prove it. Both steps are REQUIRED: the
+      # username is never optional, and neither is proving possession of the account.
       - alias: browser-mfa-forms
-        description: "Username/password then conditional OTP."
+        description: "Username, then passkey or password (+ conditional OTP)."
         providerId: basic-flow
         topLevel: false
         builtIn: false
         authenticationExecutions:
-          - authenticator: auth-username-password-form
+          # auth-username-form = Keycloak's "Username Form" (UsernameFormFactory) — resolves the
+          # user WITHOUT taking a password, which is the prerequisite for offering a passkey.
+          # Replaces the previous auth-username-password-form; see the header for the UX cost.
+          - authenticator: auth-username-form
+            authenticatorFlow: false
+            requirement: REQUIRED
+            priority: 10
+            userSetupAllowed: false
+          - authenticatorFlow: true
+            requirement: REQUIRED
+            priority: 20
+            flowAlias: browser-mfa-passkey-or-password
+            userSetupAllowed: false
+      # THE FORK. Two ALTERNATIVEs, so satisfying either one completes the flow. Priority puts the
+      # passkey first for users who have one; users who do not fall through to the password branch
+      # automatically (CREDENTIAL_SETUP_REQUIRED is caught by the ALTERNATIVE loop — see header).
+      - alias: browser-mfa-passkey-or-password
+        description: "A passkey (WebAuthn passwordless), or password with conditional OTP."
+        providerId: basic-flow
+        topLevel: false
+        builtIn: false
+        authenticationExecutions:
+          # webauthn-authenticator-passwordless = "WebAuthn Passwordless Authenticator"
+          # (WebAuthnPasswordlessAuthenticatorFactory). It reads the realm's PASSWORDLESS policy
+          # declared above — NOT the two-factor webAuthnPolicy* family. Keep it ALTERNATIVE: as
+          # REQUIRED it would force a passkey on every user, which is the lock-out this change is
+          # explicitly designed not to create.
+          - authenticator: webauthn-authenticator-passwordless
+            authenticatorFlow: false
+            requirement: ALTERNATIVE
+            priority: 10
+            userSetupAllowed: false
+          - authenticatorFlow: true
+            requirement: ALTERNATIVE
+            priority: 20
+            flowAlias: browser-mfa-password-and-otp
+            userSetupAllowed: false
+      # The unchanged pre-passkey behaviour, relocated intact into the password branch: password,
+      # then TOTP iff the user holds the platform-admin role. Nothing about the OTP condition
+      # changed; it simply no longer sits on the path a passkey login takes.
+      - alias: browser-mfa-password-and-otp
+        description: "Password, then conditional OTP for platform-admins."
+        providerId: basic-flow
+        topLevel: false
+        builtIn: false
+        authenticationExecutions:
+          # auth-password-form = Keycloak's "Password Form" (PasswordFormFactory) — the password
+          # half of the old combined form, now that the username has already been collected.
+          - authenticator: auth-password-form
             authenticatorFlow: false
             requirement: REQUIRED
             priority: 10

platform/monitoring/node-pressure-dashboard.yaml

@@ -199,7 +199,7 @@ data:
           "id": 15,
           "type": "timeseries",
           "title": "CAUSE 4 — NFS RPC retransmission ratio, per node",
-          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.102, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
+          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.30, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
           "gridPos": {"h": 8, "w": 12, "x": 12, "y": 39},
           "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
             {"color": "green", "value": null}, {"color": "red", "value": 0.01}

platform/oauth2-proxy/helmrelease.yaml

@@ -63,6 +63,75 @@ spec:
       # gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
       skip-provider-button: "true"
       scope: "openid email profile"
+    # ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ──────────
+    # MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at
+    # startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so
+    # the Gateway answers 503 and the proxy dies:
+    #
+    #   [provider.go:55] Performing OIDC Discovery...
+    #   [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC
+    #                configuration: unexpected status "503": no healthy upstream
+    #
+    # It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was
+    # still down minutes later and only a manual pod delete cleared it. There is no retry flag; the
+    # process simply does not survive a cold start. A readinessProbe cannot help (the container is
+    # already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+    #
+    # `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in
+    # values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init
+    # container for its built-in waitForRedis gate, so this adds no new base image to the platform.
+    # A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole
+    # point.
+    #
+    # BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst
+    # case is then exactly today's behaviour, never something new. An unbounded wait would trade a
+    # CrashLoop for a pod wedged in Init forever, which is not an improvement.
+    #
+    # SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is
+    # rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which
+    # CONSUMES braced expansions -- so a braced shell variable renders to the empty string and
+    # silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars
+    # knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud
+    # OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.)
+    extraInitContainers:
+      - name: wait-for-idp
+        image: docker.io/library/alpine:3.22
+        imagePullPolicy: IfNotPresent
+        command: ["/bin/sh", "-c"]
+        args:
+          - |
+            set -u
+            url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
+            i=0
+            while [ "$i" -lt 60 ]; do
+              # busybox wget validates against alpine's ca-certificates-bundle, so a real TLS
+              # failure is a non-zero exit here rather than a false pass.
+              if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then
+                echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy"
+                exit 0
+              fi
+              i=$(( i + 1 ))
+              sleep 5
+            done
+            echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2
+            echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2
+            exit 0
+        securityContext:
+          allowPrivilegeEscalation: false
+          runAsNonRoot: true
+          runAsUser: 65534
+          runAsGroup: 65534
+          readOnlyRootFilesystem: true
+          capabilities:
+            drop:
+              - ALL
+        resources:
+          requests:
+            cpu: 10m
+            memory: 16Mi
+          limits:
+            memory: 32Mi
+
     # Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
     metrics:
       enabled: true

platform/pm-agent/README.md

@@ -22,6 +22,25 @@ requires.
 
 ---
 
+## Why the agent authenticates to Gitea with a local account and a PAT
+
+Not Keycloak, and that is a decision rather than an omission (operator, 2026-10-06): the
+agents only ever talk to Gitea, so a local Gitea account with a scoped personal access token
+is the whole requirement, and it is drastically simpler than federating a machine identity.
+
+It is also what Gitea can actually do. Keycloak service accounts were considered and are not
+available for this: Gitea's API accepts basic auth, PATs, SSH signatures and **its own**
+OAuth2 bearer tokens, and as an OAuth2 provider Gitea implements **only the Authorization
+Code Grant** — there is no client-credentials grant, and nothing in Gitea validates a token
+issued by a third-party OIDC provider. So a Keycloak service-account token cannot
+authenticate to Gitea's API or to git-over-HTTP at all. Keycloak remains what it is for human
+access: the browser login source registered on Gitea (`catalog/gitea.yaml`).
+
+What still matters, and is enforced by the hand-off rather than by Keycloak: **the account
+must not be an administrator** — `infra/gitea/approval-policy.json` authorises
+`/approve-override` on live repository permission alone, so an admin-permission bot could
+waive the sensitive-path human review on its own pull request.
+
 ## It does nothing at all until an operator does two things
 
 Both are deliberate, and both fail **visibly** rather than silently:
@@ -31,6 +50,36 @@ Both are deliberate, and both fail **visibly** rather than silently:
    `infra/agent/build-image.sh` (which verifies it and refuses to push), push it as the
    operator, then update **both** places with the digest `podman push --digestfile` returned,
    in one pull request. Until then: `ImagePullBackOff`.
+
+   **Two things have to be true before any digest will pull, and neither is declared in this
+   repository.** Both were measured after PR #57 merged — this service is the first workload
+   in the cluster ever to reference the in-cluster registry, so nothing had exercised the
+   path. Details and the exact commands:
+   [`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`](../../decisions/2026-10-05-pm-agent-registry-pull-prereq.md).
+
+   - **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is
+     the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a
+     *node* process, so it resolves through the node's public upstreams
+     (`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public
+     answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN
+     IP — which this router does not hairpin; it answers :443 with its own placeholder
+     certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely
+     `x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
+     and the digest were never implicated. The fix is the pin this repository already uses
+     for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
+     in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the
+     inventory, never typed on a node. Apply it scoped to that one task:
+     `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
+     --tags hosts`.
+   - **The pull is still unauthenticated** once the node reaches the right endpoint. An
+     unauthenticated client gets nothing from this Gitea's registry, and there is no
+     `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
+     (local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea
+     account whose token carries `read:package`, referenced from the Deployment and the
+     committed worker Job template — not built yet, because minting that token is yours.
+
+   The same two apply to `conformance-runner`: same registry path, same push-then-pin model,
+   never pulled either.
 2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the
    repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off —
    which Gitea account to create, which scopes, and why it must *not* be an administrator —
@@ -96,7 +145,7 @@ Job from a committed template. **The model's output is data, never a command.**
 | **cannot** | merge anything — the gate is the approver, and the service makes no merge call |
 | **cannot** | reach the Kubernetes API: no ServiceAccount token is mounted, the `pm-agent-worker` identity is bound to nothing, and the egress policy does not include the API server. `kubectl` is on its `PATH` (the gate's toolchain needs it) and is inert |
 | **cannot** | reach anything except this Gitea and tcp/443 — no package registry, no GitHub |
-| **cannot** | clone from or push to any host but this Gitea: `pm-worker` validates `PM_GITEA_URL` against an allowlist baked into the image |
+| **cannot** | clone from or push to anywhere but `gitea.gitea.svc.cluster.local:3000`: `pm-worker` validates `PM_GITEA_URL` against a one-entry allowlist baked into the image. In-cluster consumers reach a service by its cluster-local name — a different path by design from how a client outside the cluster reaches it — so the external hostname is not an accepted value |
 | **cannot** | authenticate a `git push` **during the model session**: `GIT_ASKPASS` is unset for its duration, so the restriction is structural rather than a request in a prompt |
 | **cannot** | be created in any shape but the committed one — see the admission policy |
 

platform/pm-agent/pm-agent-credentials.sops.yaml.example

@@ -51,6 +51,15 @@
 # (2) MINT THE GITEA TOKEN as that user (log in as `pm-agent`, or use the admin API to
 #     create a token for it): Settings -> Applications -> Generate New Token.
 #
+#     A LOCAL GITEA ACCOUNT WITH A PAT IS THE DECIDED MODEL (operator, 2026-10-06), not
+#     Keycloak. The agents only ever talk to Gitea, so this is the whole requirement and it
+#     is far simpler than federating a machine identity. It is also the only thing that
+#     works: Gitea's API accepts basic auth, PATs, SSH signatures and its OWN OAuth2 bearer
+#     tokens, and as a provider Gitea implements only the Authorization Code Grant — no
+#     client-credentials grant, and nothing that validates a third-party OIDC token. A
+#     Keycloak service-account token therefore cannot authenticate to Gitea's API or to
+#     git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that.
+#
 #       Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT
 #       `write:admin`, NOT `write:organization`, NOT `write:user`.
 #

platform/vpn-ingress/README.md

@@ -13,8 +13,8 @@ egress out the house WAN.
 | Transport | WireGuard, **udp/51820**, kernel implementation |
 | Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
 | Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
-| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
-| The one secret | the server's private key, Secret `wg-server-key`, **created out of band** |
+| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
+| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
 | Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
 | Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
 | Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
@@ -129,12 +129,18 @@ translations win here.
 
 | half | where | why |
 |---|---|---|
-| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
-| the server's **private** key | Secret `wg-server-key`, **out of band** | the only real secret; an agent must not mint a long-lived private key (CLAUDE.md §6) |
+| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
+| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
 
 The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
 so the assembled config holding the private key never touches any node's disk and dies with the pod.
 
+Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with
+everything else. An out-of-band `kubectl create secret` — the first shape this component shipped in,
+corrected after review — would have put the one credential every enrolled device depends on *outside*
+the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this
+key's public half.
+
 Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
 new line granting a new device access to the whole LAN.
 
@@ -160,21 +166,40 @@ NetworkPolicy applies.
 ## Operating it
 
 * **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
-  stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
-* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
+  stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
+* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
+  stops working as soon as the pods roll (the generator's content hash guarantees they do).
 * **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
 * **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
   (`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
 * **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
 
+## Measured on prod, 2026-10-06 — what is now known rather than assumed
+
+The first real reconcile settled four things this component's design rested on. Three went the
+right way; one was a defect in my own manifest.
+
+| question | answer |
+|---|---|
+| Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. |
+| Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. |
+| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
+| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
+
 ## Prerequisites outside this directory
 
 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
    (`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
    `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
    CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
-2. **The `wg-server-key` Secret.** Runbook step 1. Until it exists the DaemonSet cannot start —
-   the correct fail-closed state for an unenrolled VPN.
+2. **The server private key.** `wg-server-key.sops.yaml` is committed, encrypted, with a
+   `REPLACE-ME` placeholder, and both overlays already carry its decryption patch — so the wiring
+   is complete and the only outstanding act is replacing the value (runbook step 2). Until then the
+   DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the
+   value with `wg pubkey`, which rejects the placeholder, an empty value, a public key pasted by
+   mistake, **and** a Secret that arrived as undecrypted `ENC[...]` ciphertext. That last case is
+   the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves
+   every Flux condition green.
 3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
    and the half CI can never check.
 
@@ -187,7 +212,7 @@ NetworkPolicy applies.
   covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
   kube-prometheus-stack. **This is the top follow-up.**
 * **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
-  handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
+  handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
   reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
   `wg-server-key` Secret.
 * **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6

platform/vpn-ingress/daemonset.yaml

@@ -37,6 +37,10 @@
 #     /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
 #     is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
 #     required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change).
+#     COROLLARY LEARNED THE HARD WAY (prod, 2026-10-06): this pod must not merely avoid NEEDING
+#     the sysctl, it must not ATTEMPT it. hostNetwork implies /proc/sys is mounted read-only, so
+#     `sysctl -w` fails with EPERM regardless of capabilities, and wg-quick treats a failed PostUp
+#     as fatal and deletes the interface. Verify the value; never write it.
 #   * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost
 #     this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all.
 #   * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a
@@ -104,7 +108,7 @@ spec:
       dnsPolicy: ClusterFirstWithHostNet
       automountServiceAccountToken: false
       # ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
-      # The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
+      # The two halves are deliberately stored apart (peers.conf explains why) and are only ever
       # joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
       # key is therefore never written to any node's disk, never committed, and never logged.
       initContainers:
@@ -136,9 +140,51 @@ spec:
               # The key arrives as a Secret file; strip any trailing newline the operator's shell
               # added, or wg-quick rejects the config with "Key is not the correct length or format".
               key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey)
-              if [ -z "$key" ]; then
-                echo "FATAL: /run/wg-server-key/privatekey is empty." >&2
-                echo "       See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2
+              # VALIDATE, don't just check for empty. The Secret ships committed with a REPLACE-ME
+              # placeholder so the GitOps wiring is complete before the key exists, so the common
+              # failure is a value that is present and WRONG — a placeholder, a public key pasted by
+              # mistake, or a SOPS document that reached the cluster undecrypted (ENC[...], which
+              # happens if an overlay is missing its `vpn-ingress` decryption patch). `wg pubkey`
+              # accepts a value only if it is a real 32-byte curve25519 private key, so it rejects
+              # all four. Without this the bad value would be written into wg0.conf and the pod would
+              # fail later and less legibly, or come up with a server nobody can handshake with.
+              if ! printf '%s' "$key" | wg pubkey >/dev/null 2>&1; then
+                echo "FATAL: the server private key in Secret 'wg-server-key' is not a valid" >&2
+                echo "       WireGuard private key (wg pubkey rejected it)." >&2
+                if [ -z "$key" ]; then
+                  echo "       It is EMPTY." >&2
+                elif [ "$key" = "REPLACE-ME" ]; then
+                  echo "       It is still the committed REPLACE-ME PLACEHOLDER — the key has not" >&2
+                  echo "       been generated yet. This is the expected state on a fresh install." >&2
+                else
+                  # `case`, NOT a dollar-brace prefix-strip. This manifest is rendered through
+                  # envsubst, which consumes every dollar-brace expansion it sees, so a shell
+                  # parameter expansion here renders to the empty string and silently disables this
+                  # very branch. `case` uses no braces, so there is nothing to escape.
+                  # (ci: flux-postbuild-substitution caught both the original expansion AND the
+                  # first version of this comment, which spelled the hazard out literally.)
+                  case "$key" in
+                    ENC\[*)
+                      echo "       It is still SOPS CIPHERTEXT (starts with ENC[). The Secret" >&2
+                      echo "       reached the cluster undecrypted — check that this environment's" >&2
+                      echo "       overlay carries the 'vpn-ingress' decryption patch." >&2
+                      ;;
+                  esac
+                fi
+                echo "       Fix: runbooks/vpn-remote-access.md step 2." >&2
+                exit 1
+              fi
+
+              # The pod shares the HOST netns, so this reads the node's real setting. Reading
+              # /proc/sys is permitted even though writing is not (see the PostUp note below).
+              # Fail closed: a tunnel that comes up but cannot forward would look healthy to every
+              # probe here and simply drop every client's traffic.
+              if ! grep -qx 1 /proc/sys/net/ipv4/ip_forward; then
+                echo "FATAL: net.ipv4.ip_forward is 0 on this node, so forwarded VPN traffic would" >&2
+                echo "       be dropped. This pod CANNOT set it: with hostNetwork the runtime mounts" >&2
+                echo "       /proc/sys read-only, because it is the host's global setting." >&2
+                echo "       Fix on the NODE: run the node_baseline role (it writes" >&2
+                echo "       /etc/sysctl.d/99-k3s-cilium.conf) — runbooks/vpn-remote-access.md step 1." >&2
                 exit 1
               fi
 
@@ -155,9 +201,20 @@ spec:
                 echo 'MTU = 1420'
                 printf 'PrivateKey = %s\n' "$key"
                 # ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ──
-                # ip_forward is already 1 from node_baseline; re-asserted so this component does not
-                # silently depend on another role's sysctl file surviving.
-                echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1'
+                # NO `sysctl -w net.ipv4.ip_forward=1` HERE. It is IMPOSSIBLE from this pod and it was
+                # the only thing stopping the tunnel coming up — MEASURED on prod 2026-10-06:
+                #     [#] sysctl -q -w net.ipv4.ip_forward=1
+                #     sysctl: permission denied on key "net.ipv4.ip_forward"
+                #     [#] ip link delete dev wg0
+                # and wg-quick then tore the whole tunnel down. The cause is not a missing
+                # capability: with `hostNetwork: true` the runtime mounts /proc/sys READ-ONLY in the
+                # container (confirmed in /proc/mounts: `proc /proc/sys proc ro,...`), precisely
+                # because those sysctls are the HOST's globals and a container must not rewrite
+                # them. CAP_NET_ADMIN does not and cannot lift a read-only mount.
+                # It was never needed anyway: node_baseline writes net.ipv4.ip_forward=1 to
+                # /etc/sysctl.d/99-k3s-cilium.conf, and reading it from this pod returns 1. The
+                # init container now VERIFIES that (a read, which /proc/sys does allow) and fails
+                # closed with a pointed message, which is what the write was pretending to achieve.
                 # SNAT client traffic to the node's address. REQUIRED: without it a LAN host would
                 # reply to 10.90.0.x, find no route, hand the packet to the router, and the router
                 # would drop it — so every connection would hang rather than fail. Matching on
@@ -217,8 +274,10 @@ spec:
             # `kubectl get pod -o wide` shows what this listener owns.
             - { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP }
           securityContext:
-            # NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and
-            # write the two netfilter rules above. Everything else is dropped, and the container is
+            # NET_ADMIN is the whole privilege budget: create the wg0 link and write the netfilter
+            # rules above. (It does NOT let the pod set a host sysctl — /proc/sys is read-only under
+            # hostNetwork — which is why nothing here tries.) Everything else is dropped, and the
+            # container is
             # NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule.
             # SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node
             # (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container
@@ -232,6 +291,16 @@ spec:
                 - NET_ADMIN
             # readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and
             # /var during startup. Same deferral as crafty/headlamp.
+            #
+            # EXPECTED, HARMLESS STARTUP NOISE — do not "fix" it. Because capabilities are dropped
+            # to NET_ADMIN only, the image's s6 init cannot chown /config (no CAP_CHOWN) and logs a
+            # run of `chown: ... Operation not permitted` plus "Permissions could not be set. This
+            # is probably because your volume mounts are remote or read-only" and a scary "we will
+            # not provide support for it". It is wrong about the cause and it does not matter:
+            # /config is a writable tmpfs emptyDir, everything here runs as root (uid 0) anyway, and
+            # wg-quick proceeds normally — verified on prod 2026-10-06, where the tunnel's only
+            # failure was the sysctl above, well past these warnings. Granting CAP_CHOWN to silence
+            # log noise would widen the privilege set of an internet-facing pod for nothing.
           resources:
             requests:
               cpu: 25m
@@ -272,14 +341,22 @@ spec:
             # during startup rather than at a point anyone would connect to the volume size. Still
             # counted against the container's 128Mi limit, and still far more than wg0.conf needs.
             sizeLimit: 8Mi
-        # The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by
-        # name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this
-        # tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key
-        # that passed through a session transcript would have to be treated as compromised.
-        # DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace
-        # and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy
-        # in the operator's password manager as well — restoring it is what makes every already
-        # enrolled client work again without re-enrolling.
+        # The ONLY secret in this component. SOPS-encrypted IN THIS TREE (R-SEC-01, spec docs/07) as
+        # wg-server-key.sops.yaml, delivered by Flux, referenced here by name only (R-IDP-04).
+        #
+        # It ships committed with a REPLACE-ME placeholder: an agent must not mint a long-lived
+        # private key (CLAUDE.md §6), so the operator generates it and `sops set`s it in
+        # (runbooks/vpn-remote-access.md step 2). Shipping the encrypted placeholder rather than no
+        # file at all is what lets the resource line and both overlay decryption patches be present
+        # and GREEN from the first commit — the alternative, a commented-out resource to uncomment
+        # later, is exactly the "file in git with no generator entry: reviewed, merged, apparently
+        # done, still broken" shape this repo keeps getting bitten by.
+        #
+        # DR: in git means a cluster rebuild restores it with everything else — the property an
+        # out-of-band `kubectl create secret` would NOT have had, and losing this key means
+        # re-enrolling every device because each client profile pins its public half. Velero also
+        # captures this namespace (its `daily` schedule excludes only `monitoring` and `unifi`).
+        # Keep a password-manager copy too: that is the one that survives losing cluster AND backups.
         - name: wg-server-key
           secret:
             secretName: wg-server-key

platform/vpn-ingress/kustomization.yaml

@@ -4,27 +4,66 @@
 # same placement as crafty/headlamp/homepage/runbooks, for the same reason: a leaf whose pod can be
 # legitimately unhealthy must never sit in the gating `platform` tier, where one component's
 # unhealth stalls platform-config and every apps wave behind it (DECISIONS-LOG: "Headlamp gating
-# stall"). That rationale is unusually concrete here: until the operator creates the `wg-server-key`
-# Secret (runbooks/vpn-remote-access.md step 1) this DaemonSet CANNOT start, so putting it in the
-# gating tier would have taken the whole platform down on merge.
+# stall"). That rationale is unusually concrete here: until the operator replaces the `REPLACE-ME`
+# placeholder in wg-server-key.sops.yaml (runbooks/vpn-remote-access.md step 2) this DaemonSet
+# CANNOT start, so putting it in the gating tier would have taken the whole platform down on merge.
 #
 # NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
+# REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated
+# ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's
+# nameReference transformer will not rewrite a reference across a namespace boundary — so without
+# this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the
+# pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1).
+# Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own
+# content generator.
+namespace: vpn-ingress
 resources:
   - namespace.yaml
-  # The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
-  - peers.yaml
   # The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
   - daemonset.yaml
   # Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
   # says so at length rather than implying a protection it cannot provide.
-  - networkpolicy.yaml
+  - networkpolicy.yaml  # The server's WireGuard private key — the ONE secret here. SOPS-encrypted (R-SEC-01, spec
+  # docs/07), decrypted per-env by the `vpn-ingress` Kustomization, referenced by name only from
+  # daemonset.yaml (R-IDP-04). Its plaintext `.example` sibling carries the operator hand-off --
+  # deliberately NOT named here: conformance/tests/l0-static/no-plaintext-secrets fails any
+  # kustomization that mentions an unencrypted template, because naming one is how a plaintext
+  # Secret gets applied by accident. (It matches comments too. That is correct, not overzealous.)
+  #
+  # SHIPS WITH A `REPLACE-ME` PLACEHOLDER, already encrypted. An agent must not
+  # mint a long-lived private key (CLAUDE.md §6), and the init container validates the value with
+  # `wg pubkey`, so the unfilled state is a loud CrashLoop naming this file, never a server that
+  # quietly cannot handshake.
+  - wg-server-key.sops.yaml
 #
-# DELIBERATELY ABSENT: a Secret. The server's private key (`wg-server-key`) is created out of band
-# by the operator and referenced BY NAME from daemonset.yaml. There is no `*.sops.yaml` here, so
-# this component needs NO SOPS decryption stanza in clusters/dev or clusters/prod — unlike
-# crafty/headlamp/velero. If that key is ever migrated into SOPS, BOTH halves must land together:
-# the file here, and a `decryption:` patch naming the `vpn-ingress` Kustomization in each overlay.
-# Without the patch, kustomize hands Flux the raw ciphertext and the pod mounts a key full of
-# ENC[...] — which applies cleanly and fails only at handshake time.
+# THIS COMPONENT CARRIES AN ENCRYPTED SECRET, so BOTH overlays must patch the `vpn-ingress`
+# Kustomization with a `decryption:` stanza — clusters/dev and clusters/prod both do, added by the
+# same change as this file. That patch is as load-bearing as the file itself: without it kustomize
+# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
+# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
+# that into a named error rather than a silently non-working tunnel.
+
+# ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ─────────────────────────────────────────
+# peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a
+# GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic:
+# kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's
+# volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE
+# PODS automatically.
+#
+# A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux
+# updates it in place, the pod template is byte-identical, no rollout is triggered — and because
+# the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps
+# the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the
+# ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having
+# never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this.
+#
+# DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism.
+configMapGenerator:
+  - name: wg-peers
+    files:
+      - peers.conf
+    options:
+      labels:
+        app.kubernetes.io/name: vpn-ingress

platform/vpn-ingress/networkpolicy.yaml

@@ -19,7 +19,7 @@
 # protects this port, and it would be enforcing nothing. So there isn't one.
 #
 # WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
-#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
+#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.conf gets
 #      NO REPLY — see README "Why this is safe to expose". This is the real control.
 #   2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
 #   3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement

platform/vpn-ingress/peers.conf

@@ -1,4 +1,16 @@
----
+# THIS FILE IS THE MOUNTED CONTENT. It is turned into the `wg-peers` ConfigMap by
+# `configMapGenerator` in kustomization.yaml, whose generated name carries a CONTENT HASH — so
+# editing this file changes the ConfigMap name, which changes the DaemonSet's pod template, which
+# ROLLS THE PODS. That is the whole reason it is a generator and not an inline ConfigMap manifest:
+# a fixed-name ConfigMap updates in place, the pod template never changes, no rollout happens, and
+# the init container's wg0.conf (built once at pod start, in tmpfs) keeps the OLD peer list
+# forever. Measured 2026-10-06: a merged peer sat in the ConfigMap while `wg show` listed none and
+# the DaemonSet reported generation=1 — it had never rolled. Same mechanism and same fix as
+# runbooks/kustomization.yaml.
+#
+# Every line here is WireGuard config. `#` is a legal comment in it, so the documentation below
+# travels inside the file that is actually mounted, rather than in a wrapper that gets stripped.
+#
 # ============================================================================================
 # THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
 # ============================================================================================
@@ -41,33 +53,40 @@
 # the Noise handshake and would be a genuine secret — which would force this register into SOPS
 # and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
 # belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: wg-peers
-  namespace: vpn-ingress
-  labels:
-    app.kubernetes.io/name: vpn-ingress
-data:
-  # Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
-  #
-  # *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
-  # An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
-  # authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
-  # DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
-  # The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
-  # (`wg show` reports an interface with no peers).
-  #
-  # COPY THIS TEMPLATE, uncomment, one block per device:
-  #
-  #   # pixel-8 — Keith's phone, enrolled 2026-10-05
-  #   [Peer]
-  #   PublicKey = <the public key new-peer.sh printed>
-  #   AllowedIPs = 10.90.0.2/32
-  #
-  # Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
-  # tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
-  # it an incident responder cannot tell which stanza to delete.
-  peers.conf: |
-    # No peers enrolled yet. See the template in this file's comments and
-    # ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
+#
+# name: wg-peers
+# namespace: vpn-ingress
+# labels:
+# app.kubernetes.io/name: vpn-ingress
+# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
+#
+# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
+# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
+# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
+# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
+# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
+# (`wg show` reports an interface with no peers).
+#
+# COPY THIS TEMPLATE, uncomment, one block per device:
+#
+#   # <device-name> — <owner>, enrolled <YYYY-MM-DD>
+#   [Peer]
+#   PublicKey = <the public key new-peer.sh printed>
+#   AllowedIPs = 10.90.0.<octet>/32
+#
+# The placeholders above are deliberately NOT valid values. An earlier version of this template
+# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
+# the whole file — matched it, so `.2` could never be allocated and the script insisted it was
+# already taken. The check now ignores comments; keeping the example unmatchable too means a
+# future grep cannot be fooled either.
+#
+# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
+# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
+# it an incident responder cannot tell which stanza to delete.
+#
+# No peers enrolled yet. See the template in this file's comments and
+# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
+# keith-phone — <owner>, enrolled 2026-10-06
+[Peer]
+PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
+AllowedIPs = 10.90.0.2/32

platform/vpn-ingress/wg-server-key.sops.yaml

@@ -0,0 +1,34 @@
+# WireGuard SERVER private key for platform/vpn-ingress — SOPS-ENCRYPTED (R-SEC-01, spec docs/07).
+# Keys stay cleartext for reviewable diffs; only `stringData` values are encrypted (.sops.yaml
+# `encrypted_regex`). Plaintext template + operator hand-off: wg-server-key.sops.yaml.example.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: wg-server-key
+    namespace: vpn-ingress
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:fUoPkfWvasisdTtwdhjeOVOlmdQkTl6Bp9YxYyji8g6k+ClATds3BnHJWOj2iplaGpT2XaNakq1+D4YyOi7QbFvNJui+e4qRpqcY8hWnazvZIHJYxjxfAfzW,iv:xtKd6oMZQumAQJAsNLHqJB+o0wdQJ+vL1kI1mc9jL7w=,tag:aImqLgbSmhsRI36/t5kpyg==,type:comment]
+    #ENC[AES256_GCM,data:npDsCW6rQleolIpur5gFkX2jSRrIQzBlGaIPs2Schv0YU3q8AanG39z4QqT3ASDmskV5YuH29Bo0ujEAgBf52YIjQCp58PDDETofrVKOJzhHMa7bV+HCxUFE,iv:7E7thyFBleyqZpqJRGApKvIq3ubCFOkshF/mxim3YPc=,tag:joIrhMe2I6kUnx1ouOPdZw==,type:comment]
+    #ENC[AES256_GCM,data:wHUFx09FgP/2+LTyw61EFbKA5UaqfjU7CeGk6Xc5av2y4r4AvQJRefDM5EGlahKecDFEFsf7fSQRFRHtUJqrQTtw+6tcj3ubSCPW4iJvhyJJv0xdb/0puRmcgj+2NfgF,iv:xgDOSDLqMQ1wcI9vrkKqfNMgIDFEusFnnO7Rzq/aOa4=,tag:m1m/fgjkBsG91c+3daqOpA==,type:comment]
+    privatekey: ENC[AES256_GCM,data:ML8CPNm4JwhQHUTUUxZO/I+9X15BCIGqOb1Uw9EmX5RTAsxQ9N97YvkYlbY=,iv:5ZUsFdMMmeGsy4fNbk6qbJkzweMVWXZ5g6E7NciAGUg=,tag:pAHv3ieLjoEyFbqbWXsVgg==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMays2OXY0VkpRc2FQQmhU
+            Q0owM3N0VkhYR0lYTUdNZWRqeTFIMjNHOVJ3CnJSMnc5NFEyU0IzWTFtUkw2YVFS
+            MU02VFo1TE14YVpZRzNhVjZwRTZwd0UKLS0tIDFaMWF4NUZjZmhRdXRuZFJYNkg3
+            SUUzN1FnMlI4bkkzbVNlMWhUVnJZODAKQ+Iy86CIv/Oj9E9j4E/0y5hDbcctRuqB
+            OR825KRPRaTdF/G2hudE1R1JbM4wGbPLMzv078R4ZXHDPGvDttSEkA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-06T06:10:31Z"
+    mac: ENC[AES256_GCM,data:6wAGAwWL7jI/NEeswBnUiWoRty+bb5DEsGEAPXajKNNYCwtm5yuXBrcOI/q4w6H8FG+nqVAECkdpAOI3S9mjDfc2zlWAzfCJJdDic92RKPXOC0tOH/M1BBSGJ1fVdJGAZZT1yAVxmtYy9WohJpawv3kmsVv+JnD+TIqmZ1CJHXo=,iv:srs4o0olSwmraQQgmdPxb+4h/t6fHHeFKJ2z914kGgY=,tag:jjoIKhU8Tu9i5SJkvkfcCg==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

platform/vpn-ingress/wg-server-key.sops.yaml.example

@@ -0,0 +1,76 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
+# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
+# (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
+# daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
+# register (peers.conf) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
+#
+# WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
+# source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`
+# would put the single credential every enrolled device depends on OUTSIDE the GitOps tree, where a
+# cluster rebuild silently loses it — and losing it means RE-ENROLLING EVERY DEVICE, because each
+# client profile pins this key's public half.
+#
+# *** THE REAL FILE IS COMMITTED WITH `privatekey: REPLACE-ME`, ALREADY ENCRYPTED. ***
+# That is deliberate, so the wiring is complete and green before the key exists: the resource is
+# already listed in kustomization.yaml and both overlays already carry the `vpn-ingress` decryption
+# patch. Your job is ONLY to replace the value (step 3 below). Until you do, the DaemonSet
+# CrashLoops with an explicit FATAL naming this file — the init container validates the value with
+# `wg pubkey` before writing wg0.conf, so a placeholder fails CLOSED and LOUDLY rather than
+# producing a server that silently cannot complete a handshake.
+#
+# ---------------------------------------------------------------------------------------------
+# EVERYTHING RUNS IN THE DEV CONTAINER. `wg`, `sops` and `qrencode` are all baked into the image
+# (infra/devcontainer/Containerfile) because that image IS the toolchain — CLAUDE.md §5, and the
+# devcontainer README's "Don't install the toolchain on the host". One shell, one toolchain, so
+# `wg genkey | sops set` is just a pipeline.
+# ---------------------------------------------------------------------------------------------
+#
+# HAND-OFF STEPS (operator — an agent must not mint a long-lived private key, CLAUDE.md §6).
+# Full procedure with verification: runbooks/vpn-remote-access.md step 2.
+#
+#  1. Generate the key STRAIGHT INTO this file. The plaintext never lands on disk and never
+#     reaches a terminal; `sops set` edits the already-encrypted document in place:
+#         ./infra/devcontainer/dev.sh run bash -c '
+#           cd /work
+#           umask 077
+#           sops set platform/vpn-ingress/wg-server-key.sops.yaml \
+#             "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
+#         '
+#     (Or edit it: `./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml`.)
+#
+#  2. Record BOTH halves in your PASSWORD MANAGER. The encrypted file is in git and Velero captures
+#     this namespace to R2, so this is belt-and-braces — but it is the only copy that survives
+#     losing the cluster AND the backups, and the key cannot be regenerated without invalidating
+#     every client profile.
+#         # public half (safe to display; it is what every client profile carries)
+#         ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d \
+#           --extract "[\"stringData\"][\"privatekey\"]" \
+#           platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
+#
+#  3. Confirm the value is a real key without printing it — `wg pubkey` succeeds only on a valid
+#     32-byte curve25519 private key, which is exactly the check the pod's init container runs:
+#         ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d \
+#           --extract "[\"stringData\"][\"privatekey\"]" \
+#           platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null \
+#           && echo "valid WireGuard key"'
+#
+#  4. PR it. Nothing else to wire: the resource line and both overlay decryption patches already
+#     exist. Flux rolls the DaemonSet and the pods leave CrashLoop.
+#
+# NOTE infra/vpn-ingress/new-peer.sh reads THIS FILE to derive each client profile's
+# `[Peer] PublicKey`, so there is no separate copy of the key to keep in step with anything.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: wg-server-key
+  namespace: vpn-ingress
+type: Opaque
+stringData:
+  # The server's WireGuard private key: `wg genkey` output — 44 chars of base64 ending in '='.
+  # NOT a passphrase, and NOT the public key. The init container validates it with `wg pubkey`
+  # before writing wg0.conf, so a placeholder or a mangled value fails closed with a clear message.
+  privatekey: REPLACE-ME

runbooks/pm-agent.md

@@ -67,7 +67,9 @@ Work through these in order; the first three are by far the most likely.
 | Symptom in the log | Cause | Fix |
 |---|---|---|
 | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
-| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) |
+| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
+| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
+| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
 | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
 | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
 | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |

runbooks/vpn-remote-access.md

@@ -21,7 +21,7 @@ behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's
 DNS, and internet egress out the house WAN.
 
 Authorisation is one thing only: the device's **public key is in
-`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
+`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.
 
 > **An enrolled device is as privileged as a machine sitting in the house** — including the three
 > k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
@@ -34,10 +34,26 @@ deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module.
 `wg-quick` fails outright and the pod CrashLoops.
 
 ```bash
-# operator action (Ansible is not an agent action)
-cd infra/ansible && ./site.sh --tags system --limit k3s_prod
+# operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox,
+# and ansible.cfg is only picked up from infra/ansible, so cd there inside the container.
+./infra/devcontainer/dev.sh run bash -c \
+  'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
 ```
 
+Three details in that command are load-bearing, not style:
+
+- **`-e k3s_hosts=k3s_prod`.** `node-baseline.yml` targets `k3s_servers` by default, which is the
+  **dev** group — and dev was decommissioned, so an unqualified run does nothing useful. This is
+  the targeting pattern `infra/ansible/README.md` "Targeting prod" documents for every playbook.
+- **`--tags system`, and never an untagged run.** The two kernel-module tasks carry the `system`
+  tag. An untagged run would also execute the `disks` tasks, which assert on block-device variables
+  prod has deliberately left as placeholders (`infra/PROD-PROVISIONING.md` §2) — the playbook's own
+  header warns about this. `--tags system` is broader than the modules alone (it also re-asserts
+  sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and
+  already in its desired state on a live node.
+- **It is a playbook, not a wrapper script.** There is no `site.sh` in this repo; `playbooks/site.yml`
+  is the full bring-up chain and is the wrong tool here.
+
 Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded:
 
 ```bash
@@ -46,46 +62,109 @@ lsmod | grep '^wireguard'
 grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf
 ```
 
-## Step 2 — create the server key (once)
+## Step 2 — fill in the server key (once)
 
-This is the **only** secret in the component, and the one thing that must never be generated in an
-agent session or pasted into a transcript. Generate it on your own machine:
+The key is **SOPS-encrypted in this tree** (R-SEC-01, spec docs/07), not created by hand:
+`platform/vpn-ingress/wg-server-key.sops.yaml`. It is already committed — encrypted, with a
+`REPLACE-ME` placeholder — and it is already listed in the component's `resources:` with a
+`decryption:` patch in both overlays. **Nothing needs wiring. You are only replacing one value.**
+
+Why in git rather than `kubectl create secret`: an out-of-band Secret puts the one credential every
+enrolled device depends on outside the GitOps tree, where a cluster rebuild silently loses it — and
+losing it means re-enrolling every device, because each client profile pins this key's public half.
+
+Everything below runs **inside the dev container** — `wg`, `sops` and `qrencode` are all baked into
+the image (CLAUDE.md §5: the toolchain lives only in the pinned container; don't install it on the
+host). One shell, one toolchain.
+
+**1. Generate the key straight into the encrypted file.** One pipeline: `wg genkey` makes it,
+`sops set` writes it into the already-encrypted document in place. The plaintext never lands on
+disk and never reaches a terminal.
 
 ```bash
-mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
-umask 077
-wg genkey > ~/.config/wireguard-home/server.key          # keep this; see "Back it up" below
+./infra/devcontainer/dev.sh run bash -c '
+  cd /work
+  umask 077
+  sops set platform/vpn-ingress/wg-server-key.sops.yaml \
+    "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
+'
 ```
 
-Load it into the cluster. **Do not echo it, do not pass it on an interactive command line that
-lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE.md §6):
+**2. Record it in your PASSWORD MANAGER.** Derive both halves from the file you just wrote:
 
 ```bash
-./infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \
-     --from-file=privatekey=/work/.cache/server.key'
+# the PUBLIC half — safe to display, and what every client profile carries
+./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
+  platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
+
+# the PRIVATE half — paste into the password manager, then clear your scrollback
+./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
+  platform/vpn-ingress/wg-server-key.sops.yaml'
 ```
 
-…where `/work/.cache/server.key` is a path you copied the key to **inside the repo's gitignored
-cache** (or mount it however you prefer — the requirement is only that the bytes never pass through
-a log). Delete that copy afterwards: `shred -u .cache/server.key`.
+The second command is the one place this key is deliberately shown. The encrypted file is in git and
+Velero captures this namespace to R2, so the password-manager copy is belt-and-braces — but it is
+the only copy that survives losing the cluster *and* the backups, and the key cannot be regenerated
+without invalidating every client profile.
 
-Confirm it exists **by key name only**:
+**3. Confirm the value is a real key**, without printing it — if `wg pubkey` prints a key, the
+value is valid; if it errors, it is still wrong:
 
 ```bash
-./infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1'
-# expected: {"privatekey   -- a NAME. If you see base64 payload, you printed the key. Rotate it.
+./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
+  platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null && echo "valid WireGuard key"'
 ```
 
-**Why out of band and not SOPS.** An agent must not mint a long-lived private key, and a key that
-passed through a session transcript would have to be treated as compromised. The trade-off is that
-this Secret is **not reconstructible from git** — so:
+That is the same check the pod's init container runs, so a pass here predicts a pass there.
 
-**Back it up.** Keep `server.key` in your password manager. Velero's `daily` schedule excludes only
-`monitoring` and `unifi` (`platform/velero/helmrelease.yaml`), so this namespace *is* captured
-off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work
-again after a cluster rebuild — **losing it means re-enrolling every device.**
+**4. PR it, and merge.** Flux rolls the DaemonSet and the pods leave CrashLoop.
+
+If you would rather use an editor than `sops set`:
+`./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml` (replace
+`REPLACE-ME`, save, exit — it re-encrypts on write).
+
+### What the pod tells you if this went wrong
+
+The init container validates the value with `wg pubkey` before writing `wg0.conf`, and names the
+specific failure in its log — so you never get a server that silently cannot complete a handshake:
+
+| pod log says | meaning |
+|---|---|
+| `still the committed REPLACE-ME PLACEHOLDER` | step 3 not done (the expected fresh-install state) |
+| `It is EMPTY` | the value was blanked |
+| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
+| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
+| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
+| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |
+
+### What a healthy start looks like
+
+Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key:
+
+```
+wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820.
+```
+
+then the `wireguard` container brings the tunnel up:
+
+```
+**** Found WG conf /config/wg_confs/wg0.conf, adding to list ****
+**** Activating tunnel /config/wg_confs/wg0.conf ****
+[#] ip link add dev wg0 type wireguard
+[#] wg addconf wg0 /dev/fd/63
+[#] ip -4 address add 10.90.0.1/24 dev wg0
+[#] ip link set mtu 1420 up dev wg0
+```
+
+**Ignore the `chown: ... Operation not permitted` run before that**, and the image's own
+"Permissions could not be set ... we will not provide support for it". Capabilities are dropped to
+`NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs
+as root, and `wg-quick` proceeds. It is wrong about the cause and it does not matter.
+
+**A pod stuck `0/1 Running` with `Startup probe failed: Unable to access interface: No such device`
+means `wg-quick` tore the tunnel down** — read the `wireguard` container log for the `[#]` line it
+died on. That is the shape of a real failure; the probe is doing its job of refusing to call a
+server without a `wg0` "ready".
 
 ## Step 3 — forward udp/51820 on the router (once)
 
@@ -126,17 +205,37 @@ unauthenticated traffic either way. If the pod is Ready, the interface exists.
 ## Step 5 — enrol a device
 
 ```bash
-./infra/vpn-ingress/new-peer.sh pixel-8 2              # full tunnel  (LAN + WAN egress)
-./infra/vpn-ingress/new-peer.sh thinkpad 3 --split     # split tunnel (LAN only)
+# in the dev container, like everything else
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2'            # full tunnel
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh thinkpad 3 --split'   # LAN only
 ```
 
-It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout,
-prints a QR code if `qrencode` is installed, and prints the `[Peer]` stanza to add. Then:
+It derives the server's public key from `wg-server-key.sops.yaml` itself, so there is no local key
+copy to keep in step with anything. Profiles land in `.cache/wg-peers/` — gitignored, and persistent
+because the repo is the container's only read-write mount.
 
-1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
+It generates the device's keypair, writes a `0600` client profile, prints a QR code, and prints the
+`[Peer]` stanza to add. It refuses to write a profile anywhere git does not ignore.
+
+Then:
+
+1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
    `# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
    comment is the only record of which stanza belongs to which device.
-2. PR → merge. Flux rolls the DaemonSet (~1 min).
+2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
+   **content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
+   which changes the pod template, which forces the roll. (It did **not** work this way at first —
+   a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
+   keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:
+
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
+        -o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
+   ```
+
+   The generation must have **increased** since before the merge. If it is still `1`, the roll did
+   not happen and `wg show` will list no peer no matter how long you wait.
 3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
    Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
 4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
@@ -176,7 +275,7 @@ if the router is not delivering the packets, and a `handshake` with no completed
 
 ## Revoking a device
 
-Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
+Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
 next reconcile. **Never re-use that address octet**: an old profile would silently start working
 again as a different device.