fix(coredns): hold in-cluster DNS at 2 replicas via an HPA, because k3s owns the Deployment

MEASURED on prod 2026-10-18: kube-system/coredns had replicas: 1. In-cluster DNS was a single pod on a single node, so draining or rebooting that node was a CLUSTER-WIDE in-cluster DNS outage until the pod rescheduled -- every Service lookup, every CNPG replica finding its primary, every operator addressing the API by name. It was invisible because 1/1 Ready looks healthy. It is urgent rather than latent now: the scheduled reboot automation on feat/automated-upgrades drains and reboots one node at a time, unattended, overnight -- turning a single point of failure into a recurring one. WHY AN HPA AND NOT replicas: 2 -- the whole decision. k3s OWNS the coredns Deployment: AddOn/coredns, source /data/k3s/server/manifests/coredns.yaml, managedFields managers deploy@k3s-prod-1 + k3s, objectset.rio.cattle.io/hash label. k3s re-stages its bundled manifests on startup, so replicas: 2 written there -- or a kubectl scale -- is reverted on the next restart or upgrade. Rejected alternatives: * coredns.yaml.skip + our own templated copy (the roles/kube_vip pattern). Deterministic, but takes ownership of coredns from k3s permanently so k3s upgrades stop updating it. Too high a price for a replica count. * A Flux patch of the Deployment. Rejected outright: Flux and k3s's deploy controller would both own one object and fight -- the default-deny NetworkPolicy drift-loop shape (~2500 events/10min) this cluster already hit. An HPA is a separate object k3s does not own, driving /scale rather than the Deployment spec, so there is no shared-ownership fight; if k3s re-stages replicas: 1 it is re-asserted within a reconcile. Disclosed cost: a brief window at 1 replica after a k3s restart. It is a FLOOR-HOLDER, not an autoscaler -- the CPU metric exists only so the autoscaling/v2 object is valid. Measured: 5m/4m against a 100m request (~5%), so the 80% target is never approached. scaleDown stabilization 600s because DNS is the worst thing to flap; scaleUp immediate, since that restores redundancy. The kustomization edit is ADDITIVE: platform/coredns/ already existed and holds the coredns-custom split-horizon ConfigMap (R-DNS-01/03), which is untouched. UNAPPLIED. The HPA has never run, and the claim that k3s reverts replicas on restart is reasoned, not observed. After merge check `describe hpa coredns` for ScalingActive=True -- an inert HPA looks identical to a working one because the hand scale already left 2 replicas. See the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Merge pull request 'fix(oauth2-proxy): gate startup on Keycloak discovery, or a cold boot CrashLoops the proxy' (#70 ) from fix/oauth2-proxy-oidc-startup-race into main
2026-10-18 00:44:09 -05:00 · 2026-10-18 04:14:52 +00:00 · 2026-10-17 22:47:30 -05:00 · 2026-10-10 16:13:46 +00:00 · 2026-10-10 16:09:53 +00:00 · 2026-10-10 13:55:15 +00:00
41 changed files with 2356 additions and 341 deletions
@@ -37,6 +37,14 @@ coverage/
 infra/tofu/backend.hcl
 infra/tofu/.r2.env

+# Operator scratch, writable from inside the dev container (the repo is its only persistent
+# read-write mount). Holds things that must never be committed: a staged copy of the SOPS age or
+# WireGuard key, rendered VPN client profiles (infra/vpn-ingress/new-peer.sh writes here by
+# default). `infra/ansible/.gitignore` already does this for its own .cache/; this is the same
+# convention at the root. Do NOT rely on `*.key` alone to protect a staged key — that only covers
+# one filename shape, which is why this rule exists.
+.cache/
+
 # Claude Code local scratch: agent worktrees + session state. Nothing here is tracked;
 # ignoring it keeps `git add -A` from staging embedded worktree repos and keeps the lint
 # stage (which scans untracked files) from tripping on .claude/worktrees/.
@@ -77,7 +77,7 @@ spec:
  storageClassName: longhorn
  resources:
    requests:
-      storage: 8Gi
+      storage: 16Gi
 ---
 # PVC plex-media (R-STATE)
 apiVersion: v1
@@ -351,7 +351,7 @@ exposure: public              # edge port-forward + public DNS; clients + Seerr
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
  - name: config
-    size: 8Gi
+    size: 16Gi
    class: longhorn           # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
    mountPath: /config        # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
                              # renderer default /data/config. Without this the PVC sat unused and Plex wrote to the
@@ -359,7 +359,7 @@ storage:
                                # The live /config contents were copied onto each PVC before this landed,
                                # so API keys and credentials other apps reference were preserved.
  # THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
-  # (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
+  # (192.168.2.30:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
  # /downloads, which is precisely where this app is ALREADY configured to write.
  #
  # WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
@@ -434,7 +434,7 @@ dependencies: []                # default-deny: reaches only DNS; arr reach it n
                                # for weeks and neither has any CNP naming the NFS server or port 2049
                                # (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
                                # egress gateway off this traffic (its selector is a podSelector), and
-                                # 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
+                                # 192.168.2.30 is inside the 192.168.0.0/16 excludedCIDR besides — see
                                # platform/vpn-egress/egress-steering.yaml, "BULK NFS".
 ingressRoutes:
  - host: qbittorrent.${BASE_DOMAIN}
@@ -234,26 +234,28 @@ data:
  WG_CLIENT_CIDR: "10.90.0.0/24"             # remote VPN clients; .1 = server. NOT a LAN range

  # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
-  # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
-  # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
+  # THE address of the NAS whose ZFS datasets back every `class: bulk` volume: the media library
+  # (Storage/plex, 8.33 T — plex, sonarr, radarr, qbittorrent) and the legacy Nextcloud archive
  # (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
  # `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
  # declared in the ServiceDefinition, because the path is identical in every environment and only
  # the ADDRESS is not.
  #
-  # WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
-  # The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
-  #   dev  nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
-  #        NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
-  #        /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
-  #        per the convention of the LAN block above.
-  #   prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
-  #        22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
-  #        the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
-  #        DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
-  # So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
-  # blast radius of shipping it unset, which is NOT the usual "fails to apply".
-  BULK_NFS_SERVER: "192.168.1.9"
+  # IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read "the two clusters are on
+  # OPPOSITE SIDES of pfSense": the datasets lived on a Proxmox host whose only interface was
+  # vmbr0 192.168.1.9/24 on pfSense's LAN, which prod (192.168.2.0/24, pfSense's WAN) could reach
+  # only through a DNAT at 192.168.2.102. Both pools were physically moved onto a standalone
+  # TrueNAS at 192.168.2.30 — prod's OWN /24 — so there is no crossing to arrange and no candidate
+  # values to choose between. Measured 2026-10-07: 192.168.2.102:2049 no longer answers;
+  # 192.168.2.30:2049 and :111 do. There is now exactly ONE storage host and one correct value;
+  # this stays a knob only because the renderer reads it per-environment.
+  #
+  # *** CHANGING THIS VALUE IS NOT A PLAIN RECONCILE. *** A bound PersistentVolume's spec.nfs is
+  # immutable — measured by server dry-run 2026-10-07:
+  #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+  # so Flux's apply FAILS rather than updating, and the pods keep their old mount. The PVs must be
+  # deleted and recreated; the operator sequence is docs/BULK-NFS-STORAGE.md §10.
+  BULK_NFS_SERVER: "192.168.2.30"

  # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
  # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
@@ -8,8 +8,8 @@
 #     NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves
 #     (DECISIONS-LOG: "Headlamp gating stall").
 #   * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the
-#     operator creates the out-of-band `wg-server-key` Secret (runbooks/vpn-remote-access.md
-#     step 1). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
+#     operator replaces the `REPLACE-ME` placeholder in platform/vpn-ingress/wg-server-key.sops.yaml
+#     with a real key (runbooks/vpn-remote-access.md step 2). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
 #     the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the
 #     cause (a missing manual step) invisible from any Flux condition.
 #
@@ -51,6 +51,9 @@ spec:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
-  # NO `decryption:` stanza is needed in the overlays: this path carries no *.sops.yaml. The server
-  # key is an out-of-band Secret (platform/vpn-ingress/kustomization.yaml says what to do if that
-  # ever changes).
+  # SOPS decryption is layered per-environment by the overlay (clusters/dev and clusters/prod each
+  # patch THIS Kustomization by name), identical to headlamp/crafty/velero — this path carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml, the WireGuard server private key. The `sops-age`
+  # Secret is created out of band and referenced by NAME only. Without the overlay patch the Secret
+  # applies cleanly full of ENC[...] and only the tunnel is broken, so the patch is as load-bearing
+  # as the encrypted file itself.
@@ -122,6 +122,24 @@ patches:
          provider: sops
          secretRef:
            name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
  # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
  # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
  # account and the Anthropic subscription token from `claude setup-token`. Without this
@@ -261,15 +279,15 @@ patches:
      - op: replace
        path: /data/CRAFTY_L4_VIP
        value: "192.168.1.55"
-      # Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
-      # (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
-      # Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
-      # Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
-      # end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
-      # change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
+      # Bulk (external NFS) storage host. THIS VALUE IS INERT — dev is decommissioned, its nodes
+      # (192.168.1.40/.41/.42) no longer exist, and the Proxmox host this used to name (192.168.1.9)
+      # went with the pools it held. It is restated per this overlay's "supply every knob"
+      # convention, not because anything consumes it. It carries the real address so that no file
+      # in the tree renders a host that is gone; it has NOT been proven from a dev node, because
+      # there is no dev node to prove it from.
      - op: replace
        path: /data/BULK_NFS_SERVER
-        value: "192.168.1.9"
+        value: "192.168.2.30"
      # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
      # barmanObjectStore and Velero's backupStorageLocation write to.
      #
@@ -113,6 +113,24 @@ patches:
          provider: sops
          secretRef:
            name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
  # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
  # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
  # account and the Anthropic subscription token from `claude setup-token`. Without this
@@ -522,60 +540,48 @@ patches:
        path: /data/LAN_CIDR
        value: "192.168.2.0/24"
      # ------------------------------------------------------------------------------------------
-      # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
+      # BULK (EXTERNAL NFS) STORAGE HOST — the TrueNAS at 192.168.2.30.
      # ------------------------------------------------------------------------------------------
      # Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
-      # radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
+      # radarr, qbittorrent) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
      #
-      # THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
-      # vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
-      # 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
-      # 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
-      # outcomes give DIFFERENT values here:
-      #     port-forward 192.168.2.102:2049 -> 192.168.1.9:2049   =>  "192.168.2.102"
-      #     a WAN->LAN route to 192.168.1.0/24                    =>  "192.168.1.9"
-      # Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
-      # ever became reachable — mounts something unintended. So it stays a placeholder.
+      # THIS WAS THE REPO'S ONE LOAD-BEARING PLACEHOLDER, AND THE REASON IT COULD BE IS NOW GONE.
+      # The datasets lived on a Proxmox host with a single interface on pfSense's LAN
+      # (192.168.1.9), which prod could reach only through a DNAT at 192.168.2.102 — so the correct
+      # value here depended on an undecided operator choice, and guessing it would have authored a
+      # cluster that mounts nothing. Both pools were physically moved onto a standalone TrueNAS at
+      # 192.168.2.30, prod's OWN /24. No crossing to arrange, no candidates to choose between.
+      # Measured 2026-10-07: 192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and :111 do.
      #
-      # WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
-      # NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
-      # over v3 was in part this decision's cost: v3 would have needed four ports and three of them
-      # on ephemeral numbers.
-      #
-      # WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
-      # Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
-      # the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
-      # address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
-      # 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
-      # addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
-      #
-      # *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
-      # A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
-      # placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
-      # failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
-      # TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
-      # ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
-      # apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
-      # a revision containing the bulk volumes; the dev overlay renders a working value, which is
-      # where these manifests are meant to be proven first.
+      # ONE PORT IS STILL ALL IT NEEDS: tcp/2049. The PVs mount nfsvers=4.1 and NFSv4 carries mount
+      # and lock in-protocol — no rpcbind/111, no mountd, no statd. The NAS advertises
+      # `+3 +4 +4.1 +4.2` (/proc/fs/nfsd/versions, measured), so 4.1 negotiates.
      - op: replace
        path: /data/BULK_NFS_SERVER
-        # 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
-        # host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
-        # reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
-        # keeps the base value 192.168.1.9.
+        # VERIFIED ON THE NAS 2026-10-07, before any cluster touched it (`exportfs -v`):
+        #   /mnt/Storage/plex       192.168.2.10/.11/.12  rw,sec=sys,secure,root_squash,no_all_squash
+        #   /mnt/Storage/nextcloud  192.168.2.10/.11/.12  ro,sec=sys,secure,root_squash,no_all_squash
        #
-        # PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
-        #   - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
-        #   - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
-        #     carries no_root_squash, so a root write would have passed and proved nothing, while
-        #     LSIO images default to PUID 911 and would have gone Ready then failed every import
-        #     with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
-        #   - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
-        #     real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
-        # pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
-        # so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
-        value: "192.168.2.102"
+        # no_all_squash is the flag the entire permission model rests on: client uids pass through
+        # untouched, so the arrs' PUID=1000 arrives as the owner of a 0755 1000:3000 tree. sec=sys
+        # keeps that numeric rather than principal-mapped. The export deliberately does NOT carry
+        # no_root_squash, which the Proxmox one did: nothing here needs it, because every write to
+        # the tree happens after the LSIO drop to uid 1000 and the images only chown their Longhorn
+        # /config. Both subPath directories (plex/downloads, nextcloud/nextcloud) already exist at
+        # 0755 1000:3000, so a squashed root can still traverse them for subPath resolution.
+        #
+        # The ro= posture on the archive is now SERVER-enforced, which the old export also managed.
+        # The media tree's read-only half (plex) stays a CLIENT-side guarantee — all four media pods
+        # can land on any node, so the server sees the same three addresses for reader and writers
+        # alike and cannot tell them apart. Unchanged by the move; see docs/BULK-NFS-STORAGE.md §5.
+        #
+        # *** CHANGING THIS VALUE REQUIRES DELETING AND RECREATING THE PVs. *** A bound PV's
+        # spec.nfs is immutable — measured by server dry-run 2026-10-07:
+        #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+        # so Flux's apply FAILS rather than updating and the pods keep their old, dead mount. The
+        # operator sequence is docs/BULK-NFS-STORAGE.md §10. Retain + the renderer's pre-set
+        # claimRef make it safe: no data is touched and each recreated PV rebinds its own PVC.
+        value: "192.168.2.30"
      # ==========================================================================================
      # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
      # ==========================================================================================
@@ -0,0 +1,127 @@
+---
+date: 2026-10-05
+branch: docs/pm-agent-registry-pull-prereq
+title: pm-agent reconciled exactly as designed — and is the first workload ever to pull from the in-cluster registry, which does not work yet
+---
+
+## What changed
+
+Documentation only. No manifest, no policy, no code.
+
+PR #57 merged and `pm-agent` reconciled. I measured what landed, and the measurement turned
+up **two prerequisites for the next operator step that nothing in this repository
+declares** — and that the handoff entry for #57 could not have known, because it listed the
+image pull as unverified and that is exactly where it bit.
+
+So this entry records the post-merge measurement, and
+`platform/pm-agent/README.md`, `docs/plans/pm-agent-issue-to-pr.md` and
+`runbooks/pm-agent.md` now name both prerequisites where they say "build, push, pin the
+digest". Before this, that step read as one action; it is three, and the two new ones are
+not an agent's to perform.
+
+**The finding is not specific to pm-agent.** `conformance-runner` has the identical
+registry path (`gitea.home.keithjackson.dev/home-network/...`), the identical "operator
+pushes, a digest goes in a PR" model, and has never been pulled either — so the same two
+blockers sit in front of the L2/L3 conformance wiring. Measured: **no pod anywhere in the
+cluster references that registry** except the pm-agent dispatcher, which is therefore the
+first workload ever to try it.
+
+## What I verified
+
+Read-only, through the pinned toolbox, against the live prod cluster.
+
+**The merge reconciled, completely and exactly as designed.**
+
+- `GitRepository/flux-system` artifact revision: `main@sha1:95597bd6883637…` — the merge
+  commit. `platform-config`, `pm-agent` and `apps-wave-0` all report that same
+  `lastAppliedRevision` with `Ready=True`. Compared field to field, per CLAUDE.md §2b,
+  rather than assumed from the merge.
+- **The security boundary is live.** `ClusterPolicy/pm-agent-job-shape`:
+  `validationFailureAction=Enforce`, `failurePolicy=Fail`, `Ready=True`, and all six rules
+  present (`worker-must-be-the-committed-shape`,
+  `worker-must-be-one-container-with-no-args`,
+  `worker-volumes-must-be-workspace-or-tmp-emptydirs`,
+  `worker-may-not-declare-init-or-ephemeral-containers`,
+  `worker-env-names-must-be-the-committed-set`, `worker-may-not-use-envfrom`). This is the
+  check that mattered most: a ClusterPolicy added to the wrong kustomization list ships
+  green and never reaches the cluster, which is what happened to the conformance boundary
+  on 2026-09-29.
+- Namespace contents: both ServiceAccounts, `pm-agent-config` (17 keys), the
+  `pm-agent-credentials` Secret, the `default-deny` NetworkPolicy, and both
+  CiliumNetworkPolicies with `VALID=True`.
+- **SOPS decryption works through the new overlay patch.** Checked by KEY NAMES and VALUE
+  LENGTHS only, never values (CLAUDE.md §6): keys `gitea-token`, `claude-code-oauth-token`,
+  lengths 76 and 52 bytes — which are the base64 lengths of the committed placeholder
+  strings, so the Secret holds the decrypted placeholders and not ciphertext. A
+  ciphertext apply succeeds silently and would have left `ENC[AES256_GCM,…]` values an
+  order of magnitude longer.
+- **`pm-agent-worker` really is bound to nothing.** The namespace has exactly one
+  RoleBinding (`pm-agent` → Role `pm-agent`, subject `ServiceAccount/pm-agent`), and no
+  ClusterRoleBinding anywhere names either pm-agent ServiceAccount.
+- `gitea-allow-pm-agent-ingress` is applied in the `gitea` namespace, matching
+  `io.kubernetes.pod.namespace: pm-agent`.
+- The dispatcher is `ImagePullBackOff` on the all-zero placeholder digest — the designed,
+  visible failure.
+
+**The two prerequisites, measured.**
+
+1. **The nodes cannot verify the registry's TLS certificate.** containerd's error on the
+   pull is not about the digest:
+
+       failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:000…":
+       tls: failed to verify certificate: x509: certificate is not valid for any names,
+       but wanted to match gitea.home.keithjackson.dev
+
+   "not valid for **any** names" means the certificate presented to the node carries no
+   matching SAN at all. From the operator's workstation the same name is healthy: it
+   resolves to `192.168.2.20` and serves a Let's Encrypt certificate
+   `CN=*.home.keithjackson.dev` with SANs `*.home.keithjackson.dev, home.keithjackson.dev`
+   — and `Certificate gateway/wildcard-home` is `Ready=True` for exactly those names. So
+   the certificate that *should* be served is fine, and what the NODE reaches is not what
+   the workstation reaches. **Correcting the digest will not clear this.**
+2. **Anonymous pulls from this registry are refused.** Not inferred from a bare 401 — that
+   would have been wrong, because this Gitea answers 401 for *everything* anonymously:
+   `/v2/home-network/agent-runner/manifests/latest`, `/v2/home-network/conformance-runner/…`,
+   a randomly-named nonexistent repository, and `nosuchowner/nosuchimage` all return 401,
+   and `/api/v1/packages/Home-Network` returns `{"message":"token is required"}`. The
+   registry *does* issue a token to an anonymous client, but its `access` claim is `null` —
+   no pull scope. So the conclusion that holds is the simple one: **an unauthenticated
+   client gets nothing from this registry**, so a pull with no credential cannot succeed.
+   And `grep` over the tree finds **no `imagePullSecrets` anywhere** — not in
+   `platform/pm-agent/deployment.yaml`, not in `infra/agent/worker-job-template.yaml`, not
+   in `conformance/incluster/`.
+
+## What I did NOT verify
+
+- **Why the node sees a nameless certificate.** I measured that it does, and that the LAN
+  does not. I did not diagnose it: I did not read the nodes' resolver or
+  `/etc/rancher/k3s/registries.yaml` (node files, not reachable read-only through the API),
+  and I did not check whether the node resolves the name to the same `192.168.2.20` the
+  workstation does or to something else.
+
+  **A correction to a remark I first wrote here:** I cited
+  `cluster-vars`'s `GATEWAY_VIP: 192.168.1.53` as suspicious next to the nodes'
+  `192.168.2.x`. That was noise — the operator confirms those base `cluster-vars` values are
+  **dev-era leftovers for a cluster that is decommissioned**, so the 192.168.1.x addresses in
+  that file say nothing about prod and should not be read as evidence about anything. The
+  measured facts stand on their own and none of them depended on that knob.
+- **Whether the package would be pullable once authenticated.** Nothing has ever been
+  pushed to that path, so "a credential is sufficient" is untested. The only thing
+  established is that no credential is insufficient.
+- **Which fix the operator wants.** Two routes, and they are not equivalent:
+  (a) an `imagePullSecret` — a `dockerconfigjson` holding a Gitea token, SOPS-encrypted
+  like `pm-agent-credentials`, referenced from the Deployment *and* from the committed
+  worker Job template; note the Job template's shape is pinned by admission, and
+  `imagePullSecrets` is a field that policy does not currently constrain, so adding one is
+  a template change that needs a deliberate look at whether the policy should pin it too.
+  (b) allowing anonymous reads of that package in Gitea — no manifest change at all, but it
+  makes the image world-readable to anything that can reach the host.
+  I have implemented neither: both need an operator action I must not take, and (b) is a
+  posture decision. The docs now present both.
+- **Everything the #57 entry listed as unverified about RUNTIME remains unverified.** The
+  image has still never been built or pushed. No dispatcher has authenticated to Gitea, no
+  issue has been triaged, no worker Job has been created — so the admission policy, though
+  now provably applied and Enforce, has still never refused or admitted a real Job. The
+  network policies have not carried traffic.
+- **I did not re-run the merge gate against `main` after the merge.** This branch's own gate
+  run is the evidence for this change; the merged state's is CI run on PR #57.
@@ -0,0 +1,112 @@
+---
+date: 2026-10-05
+branch: fix/vpn-runbook-ansible-command
+title: vpn-remote-access runbook — step 1 named a script that does not exist, and step 2 cited the wrong reason the key is gitignored
+---
+
+## What changed
+
+Two defects in `runbooks/vpn-remote-access.md` (merged in #55), both authored by me, both reported
+by the operator the first time they tried to follow it.
+
+**1. Step 1 named a command that does not exist.** It said:
+
+    cd infra/ansible && ./site.sh --tags system --limit k3s_prod
+
+There is **no `site.sh` anywhere in this repo** (`find . -name site.sh` → nothing). The invocation
+was wrong in four separate ways at once, which is what makes it worth recording rather than just
+patching: wrong kind of thing (this repo drives Ansible with `ansible-playbook <playbook>`, never a
+wrapper script — `playbooks/site.yml` exists and is the full bring-up chain, not this); wrong
+playbook (the module work lives in `playbooks/node-baseline.yml`); wrong targeting flag (`--limit`
+instead of this repo's `-e k3s_hosts=...` convention, which `infra/ansible/README.md` "Targeting
+prod" documents for every playbook); and no toolbox wrapper, though Ansible exists only inside the
+pinned container and `ansible.cfg` is only read from `infra/ansible`.
+
+Replaced with the real command, plus the three things about it that are load-bearing:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
+
+- `-e k3s_hosts=k3s_prod` — the playbook defaults to `k3s_servers`, which is the **dev** group;
+  dev is decommissioned, so an unqualified run does nothing useful.
+- `--tags system`, never untagged — the two kernel-module tasks carry that tag, and an untagged run
+  would also hit the `disks` tasks, which assert on block-device variables prod deliberately leaves
+  as placeholders (the playbook's own header warns about this). `--tags system` is broader than the
+  modules alone (sysctls, iscsid, qemu-guest-agent, swap masking) but each is idempotent and
+  already in its desired state on a live node.
+
+**2. Step 2 gave the wrong reason the server key is safe from git — and the right reason is
+fragile.** It told the operator to stage the key at `/work/.cache/server.key` "inside the repo's
+gitignored cache". `.cache/` is **not** gitignored: `git check-ignore -v .cache/serverkey.txt`
+matches nothing. The file was ignored only incidentally, by `.gitignore` line 11 `*.key` — so the
+path happened to be safe for a reason I had not identified, and **any operator who renamed it
+(`server.pem`, `wg-server`, `privatekey`) would have had a committable WireGuard private key sitting
+in the checkout.** The gate would not have caught it either; `no-plaintext-secrets` does not
+recognise a bare WireGuard key.
+
+The step now states the real rule, marks the `.key` suffix as the thing doing the work, and gives
+the operator a verification they can run before trusting it (`git check-ignore -v` must print a
+rule, and `git status --short` must not list the file).
+
+Nothing outside `runbooks/vpn-remote-access.md` changed; no manifest, no behaviour.
+
+## What I verified
+
+- **`site.sh` does not exist**: `find . -name 'site.sh' -not -path './.git/*'` → no output.
+- **The real surface**: `infra/ansible/playbooks/` contains `node-baseline.yml` and `site.yml` (and
+  11 others); `node-baseline.yml`'s own header documents both
+  `-e k3s_hosts=k3s_prod` and the tag-scoping hazard. The `-e <playbook>_hosts=` convention is
+  confirmed by `infra/ansible/README.md` §"Targeting prod" and by the existing
+  `runbooks/node-memory-swap.md`, which invokes `ansible-playbook playbooks/node-health.yml -e
+  node_health_hosts=...`.
+- **`k3s_prod` is a real inventory group**: `[k3s_prod:children]` in `inventory/hosts.ini`, with
+  `[k3s_prod_primary]`/`[k3s_prod_secondary]`.
+- **`--tags system` covers the kernel-module tasks**: both "Persist kernel modules…" and "Load
+  kernel modules now" carry `tags: [system]` in `roles/node_baseline/tasks/main.yml`.
+- **The gitignore claim, measured both directions**:
+  `git check-ignore -v .cache/server.key` → `.gitignore:11:*.key`;
+  `git check-ignore -v .cache/serverkey.txt` → no match. So the suffix is the only thing ignoring it.
+- **Every path the component's docs reference resolves**: all six relative markdown links in
+  `platform/vpn-ingress/README.md` (checked from that file's own directory), and both Ansible
+  playbook paths.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; the rc 3 is the two
+  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no `CLUSTER_AVAILABLE`).
+  **Not green. Reported as rc 3, not as pass.**
+- **This branch touches NO sensitive path** (`runbooks/**` and `decisions/**` are not in
+  `infra/gitea/approval-policy.json`), so unlike #55 it needs no `/approve-override`.
+
+## What I did NOT verify
+
+- **The corrected command has not been RUN.** Ansible is an operator action, and this is the
+  decisive gap: I have verified the playbook, the group, the tag and the flag all exist and are
+  the documented convention, but not that the run succeeds end to end or that
+  `lsmod | grep '^wireguard'` then reports the module. It is a better-evidenced command, not a
+  proven one.
+- **Whether `--tags system` is entirely side-effect-free on a live prod node.** Each task it runs is
+  idempotent by construction, but I did not dry-run it (`--check`) to confirm it reports zero
+  changes beyond the module load.
+- **The rest of the runbook's commands past step 2** are unchanged and still unexercised — in
+  particular `kubectl exec ds/wireguard -- wg show`, which cannot be reached while the pods sit in
+  `Init` waiting on the server key.
+- **Whether any OTHER document I wrote in #55 contains an invented command.** I re-checked every
+  file path and markdown link in the component (all resolve) and every script the docs tell an
+  operator to run, but "the referenced file exists" is weaker than "the command is correct" — which
+  is exactly the gap that produced `site.sh`.
+
+## Post-merge state of #55, recorded here since its own entry predates the merge
+
+Measured on live prod after #55 merged as `ad3f56e`:
+
+- Flux applied it by the real path: Kustomization `vpn-ingress` `lastAppliedRevision:
+  main@sha1:ad3f56e`, equal to the GitRepository's artifact revision, `Ready=True`.
+- All **18** Flux Kustomizations `Ready=True` — the new tier disturbed nothing, and `wait: false`
+  behaved as intended (`vpn-ingress` is Ready while its own pods are not, so nothing downstream
+  stalled on it).
+- **`hostNetwork` is confirmed in effect, not merely configured**: the three pods report pod IPs
+  `192.168.2.10/.11/.12` — the node addresses, one per node.
+- **It fails closed exactly as designed**, with a legible cause: all three pods `Init:0/1`,
+  `MountVolume.SetUp failed for volume "wg-server-key" : secret "wg-server-key" not found`, and no
+  Secret in the namespace. The #55 entry's "the DaemonSet has never been reconciled" is superseded
+  by this paragraph; its remaining unverified claims (the tunnel carrying a packet, the
+  iptables/nftables backend question) still stand.
@@ -0,0 +1,187 @@
+---
+date: 2026-10-05
+branch: fix/vpn-server-key-in-sops
+title: VPN server key moves into SOPS-encrypted git — the out-of-band Secret broke R-SEC-01 and the DR story
+---
+
+## What changed
+
+**Operator correction (2026-10-05): "secrets are kept in source through sops".** They are right, and
+#55 shipped the WireGuard server private key the wrong way: as a Secret the operator was told to
+`kubectl create` out of band, referenced by name only. This branch moves it into the tree as
+`platform/vpn-ingress/wg-server-key.sops.yaml`, where every other secret in this repo lives.
+
+**Why the original choice was wrong, not merely unconventional.** Two concrete costs, both of which
+#55's own README half-admitted and then shipped anyway:
+
+1. **R-SEC-01 / spec docs/07 says secrets live in source, SOPS-encrypted, reaching the cluster only
+   through Flux.** An out-of-band Secret is outside the GitOps tree by construction.
+2. **It broke DR for the one credential that cannot be re-derived.** A cluster rebuild restores
+   everything in git; it does not restore a hand-made Secret. Losing this key means **re-enrolling
+   every device**, because each client profile pins its public half. #55 papered over this with
+   "keep a copy in your password manager", which is a workaround for a self-inflicted gap.
+
+Note `platform/kyverno/admission/secret-provenance.yaml` did **not** forbid the original shape —
+it scopes to `app-namespace=true` namespaces and `vpn-ingress` is deliberately a platform namespace,
+so admission would have accepted the hand-applied Secret. The rule it violated was the written
+requirement, not an enforced control. Recording that because "the gate let it through" was never
+evidence it was correct.
+
+**How the unfillable-key problem is solved.** An agent still must not mint a long-lived private key
+(CLAUDE.md §6), so the file is committed **already encrypted, with `privatekey: REPLACE-ME`**. That
+makes the wiring complete and the gate green from the first commit — the resource line and both
+overlay decryption patches are present — and leaves the operator exactly one act: replace the value.
+The alternative, a commented-out resource line to uncomment later, is precisely the "file in git with
+no generator entry: reviewed, merged, apparently done, still broken" shape this repo keeps getting
+bitten by (`alert-runbooks-resolvable` exists because of it).
+
+**The init-container guard is now a validation, not an emptiness check.** It was `[ -z "$key" ]`.
+Because the committed placeholder is non-empty, the likely failure is now a value that is present
+and *wrong*, so the guard pipes the value through `wg pubkey` — which accepts only a real 32-byte
+curve25519 private key — and names which of four cases it hit:
+
+| case | why it matters |
+|---|---|
+| `REPLACE-ME` placeholder | the expected fresh-install state |
+| empty | blanked value |
+| **starts with `ENC[`** | **the Secret reached the cluster UNDECRYPTED.** A missing overlay decryption patch applies cleanly and leaves every Flux condition green — this is the silent failure the guard exists for |
+| rejected, none of the above | mangled value, or a *public* key pasted by mistake |
+
+Files:
+
+- `platform/vpn-ingress/wg-server-key.sops.yaml` (new) — SOPS-encrypted to the repo's age recipient,
+  `stringData.privatekey` only, placeholder value.
+- `platform/vpn-ingress/wg-server-key.sops.yaml.example` (new) — plaintext template + operator
+  hand-off, following the `crafty-oauth2-proxy-credentials.sops.yaml.example` convention.
+- `platform/vpn-ingress/kustomization.yaml` — the resource added; the "DELIBERATELY ABSENT: a
+  Secret" block replaced with the overlay-patch warning.
+- `platform/vpn-ingress/daemonset.yaml` — the `wg pubkey` guard; volume comment rewritten.
+- `clusters/dev/kustomization.yaml`, `clusters/prod/kustomization.yaml` — **the `vpn-ingress`
+  `decryption:` patch, in both.** This is the load-bearing half: #55's own kustomization comment
+  predicted that forgetting it applies ciphertext cleanly.
+- `clusters/base/vpn-ingress.yaml` — its "NO decryption stanza is needed" comment was now false.
+- `platform/vpn-ingress/README.md`, `runbooks/vpn-remote-access.md` — step 2 rewritten.
+- `infra/vpn-ingress/new-peer.sh` — stale "step 1" pointer.
+
+**A second finding: `wg` and `sops` were not in the same place — and the first fix for that was
+also wrong.** Measured: `wg` existed only on the host (`/usr/bin/wg`), `sops` only in the dev
+container, `qrencode` on neither. So `sops -d … | wg pubkey` could not run in one shell, and #55's
+step 2 — which assumed it could — would have failed.
+
+My first fix was to **split every step by where it runs**, and to tell the operator to
+`dnf install qrencode` on their workstation. **Operator correction: "sop, wg, and any other tool
+requirement is supposed to be baked into the devcontainer."** That is CLAUDE.md §5 and the
+devcontainer README's "Don't install the toolchain on the host", and the split fix broke the
+invariant rather than honouring it — it also produced a procedure that could never be reproduced in
+CI, and which no one should have to follow.
+
+**The real fix:** `wireguard-tools` and `qrencode` are now distro packages in
+`infra/devcontainer/Containerfile`, alongside ansible/openssh/diffutils. Both are in Fedora 44's
+repos (`wireguard-tools` 1.0.20260223 — the same upstream version as the container image this
+component runs — and `qrencode` 4.1.1), unpinned like every other dnf package there, because the
+pinned thing is the Fedora base and neither is a gate tool whose version changes what green means.
+
+Three things got *simpler* rather than merely relocated, which is the tell that the operator's rule
+was the right one:
+
+1. **Step 2 is now one pipeline in one shell:** `sops set … "\"$(wg genkey)\""` generates the key
+   straight into the already-encrypted document. The plaintext never lands on disk, never reaches a
+   terminal, and there is no staged copy to shred. The previous version needed a copy in
+   `.cache/server.key`, a `git check-ignore` warning about the `*.key` suffix being the only thing
+   protecting it, and a sha256 comparison to prove the right bytes arrived. All of that is gone.
+2. **`new-peer.sh` now derives the server's public key from `wg-server-key.sops.yaml` itself**
+   (`sops -d --extract … | wg pubkey`), so there is no local key copy to keep in step with the
+   encrypted one. It previously required the operator to keep the private key at a magic host path.
+3. **`.cache/` is now gitignored at the repo root** (mirroring `infra/ansible/.gitignore`), so the
+   container has a persistent, non-committable scratch directory. `new-peer.sh` writes client
+   profiles to `.cache/wg-peers/` by default — necessary because the repo bind-mount is the
+   container's ONLY persistent read-write mount, so a profile written to `/tmp` in there is gone
+   before it can be moved to a desktop. Its old blanket "refuse to write inside the checkout" guard
+   is replaced by `git check-ignore`, which tests the actual property instead of a proxy.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two overlays) AND adds a `**/*.sops.yaml`, so
+`ci/approval-policy.sh` will BLOCK it.** It needs the operator's `/approve-override <reason>` as a
+PR **review** of type "Comment". I have deliberately not posted it (CLAUDE.md §1).
+
+## What I verified
+
+- **The encrypted file is real and correctly targeted.** `sops -e -i` in the pinned toolbox
+  produced `stringData.privatekey: ENC[AES256_GCM,...]` with cleartext keys, `encrypted_regex
+  ^(data|stringData)$`, and exactly one age recipient:
+  `age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg` — the recipient in the root
+  `.sops.yaml`, i.e. the key Flux holds as `sops-age`.
+- **Round-trip, without exposing anything**: `sops -d` then a Python read of key names only →
+  `stringData keys: ['privatekey']`, and `value is still the placeholder: True`.
+- **Both overlay patches are present and name the right Kustomization** (`name: vpn-ingress`,
+  `namespace: flux-system`, `decryption.secretRef.name: sops-age`), inserted after each overlay's
+  `velero` stanza, mirroring headlamp/crafty/velero.
+- **`kubectl kustomize platform/vpn-ingress` builds** and now emits five objects — Namespace,
+  ConfigMap, **Secret**, DaemonSet, NetworkPolicy — with `${WG_CLIENT_CIDR}` still an unexpanded
+  token for Flux.
+- **The tool gap, measured on both host and container before the fix**: `wg` host-only, `sops`
+  container-only, `qrencode` on neither.
+- **Both packages exist in the image's repos**, queried from inside the running container:
+  `wireguard-tools` 1.0.20260223 (repo `updates`) and `qrencode` 4.1.1 (repo `fedora`).
+- **The image REBUILDS with them and both tools resolve inside it** —
+  `./infra/devcontainer/dev.sh build` succeeded, and `wg --version` / `qrencode --version` /
+  `sops --version` all answer in the rebuilt container. This is the claim the change rests on, so
+  it was executed rather than reasoned.
+- **`new-peer.sh`'s new SOPS path fails correctly against the placeholder.** Run in the rebuilt
+  container it refused with "could not derive the server public key … it still holds the REPLACE-ME
+  placeholder", which is both the expected pre-key behaviour and a live test of the
+  `sops -d | wg pubkey` pipeline the whole design now depends on.
+- **`.cache/` is ignored regardless of filename** now: `git check-ignore -v .cache/anything.conf`
+  and `.cache/serverkey.txt` both match `.gitignore:46:.cache/`. Before this change only `*.key`
+  matched, which is the fragility the previous runbook had to warn about.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
+  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no cluster).
+  **Not green. Reported as rc 3, not as pass.**
+- **The L0 gate caught THREE real defects in this branch before it could merge — including one
+  in the fix for another — and I had asserted one of them was fine without running it.** Both are recorded because the gate's value here is the
+  whole point of the exercise:
+  1. **`no-plaintext-secrets` FAILED**: "`wg-server-key.sops.yaml.example`: an unencrypted .example
+     template is referenced by a kustomization — it would be applied as a plaintext Secret". My
+     kustomization named the template in a *comment*, and the check matches comments too (correctly:
+     naming one is how a plaintext Secret gets applied by accident). `platform/crafty/kustomization.yaml`
+     never names its own example either — the convention was there to copy and I missed it. Fixed by
+     not naming the file; the hand-off lives inside the example, where it belongs.
+     **An earlier draft of this entry claimed this check passed. It did not. The claim was written
+     from expectation rather than from a run.**
+  2. **`flux-postbuild-substitution` FAILED** on `${key#ENC[}` in my new guard
+     (`daemonset.yaml:155`), flagged `[SILENT-EMPTY]`: envsubst consumes every `${...}`, so my
+     shell prefix-strip would have rendered to the empty string — the ciphertext branch of the guard
+     would never have fired, which is precisely the silent failure the guard was added to catch.
+     Rewritten with `case`, which uses no braces and so has nothing to escape.
+  3. **And then it failed a THIRD time, on the fix's own comment** — `[FATAL-PARSE]
+     daemonset.yaml:156 ${...}`, the prose explaining the rule, written with the literal form. That
+     class is worse than it looks: a `FATAL-PARSE` hazard fails the **entire** Kustomization, every
+     resource in the path, not just the line. Rewritten as "dollar-brace expansion". This is the
+     second time in this component that writing the hazard literally inside a comment about the
+     hazard has been the defect (the first was in #55), so the habit to build is: **in any
+     substituted path, describe the construct in words, never type it.**
+- **After both fixes, `no-plaintext-secrets` and `flux-postbuild-substitution` pass** (446 and 198
+  assertions respectively), and `l0` is green.
+
+## What I did NOT verify
+
+- **That Flux decrypts this file on prod.** The overlay patches have never reconciled. This is the
+  whole risk of the change and it has exactly one observable outcome: after merge the pods should
+  CrashLoop reporting the **`REPLACE-ME` placeholder** (proving decryption worked), *not* reporting
+  **`ENC[`** (which would prove a patch is wrong). I built the guard to tell those apart because I
+  cannot test them.
+- **That `wg pubkey` accepts the operator's real key through this path.** The guard is reasoned from
+  `wg pubkey`'s behaviour and has never run — `wg` is absent from both this workstation's container
+  and the gate, so the first execution will be in the pod.
+- **The `sops set` command in the hand-off.** I verified `sops -e -i` and `sops -d` in the toolbox;
+  I did **not** run `sops set` with the quoting shown, and nested quoting inside
+  `dev.sh run bash -c` is exactly where a documented command tends to break. If it misbehaves, the
+  editor form (`sops <file>`) is the fallback and does not depend on quoting.
+- **The sha256 comparison in step 2 step 4** is untested end to end (it needs the real key).
+- **Whether anything else in #55 assumed a tool is available where it is not.** I checked `wg`,
+  `sops` and `qrencode` because step 2 depended on them. I did not audit every command in the
+  runbook the same way — and that class of defect has now produced two corrections in a row
+  (`site.sh`, then `sops | wg`), so assume more remain until each step has actually been run.
+- **The live VPN still carries no traffic**, and the earlier gaps stand unchanged: the
+  iptables-vs-nftables question, `allowPrivilegeEscalation: false` versus the image's s6 init, and
+  any off-LAN reachability.
@@ -0,0 +1,73 @@
+---
+date: 2026-10-06
+branch: fix/new-peer-ignores-template-comments
+title: new-peer.sh refused the first VPN address because its duplicate check matched its own commented-out example
+---
+
+## What changed
+
+The operator's first real enrolment failed:
+
+```
+$ ./infra/vpn-ingress/new-peer.sh keith-phone 2
+new-peer.sh: 10.90.0.2/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet.
+```
+
+**Nothing was allocated.** `peers.yaml` had zero peers — the pod's own init container reported
+`wg0.conf assembled: 0 peer(s)`. The duplicate-address check grepped the WHOLE file, and
+`peers.yaml` line 66 is a **commented-out example stanza** I wrote, containing literally
+`#   AllowedIPs = 10.90.0.2/32`. So `.2` — the first address anybody would ask for, and the one my
+own runbook uses in its example — was permanently un-allocatable, with a message asserting the
+opposite of the truth.
+
+Both halves fixed, because either alone leaves the trap half-set:
+
+- **`infra/vpn-ingress/new-peer.sh`**: the check strips comment lines before matching
+  (`grep -vE '^[[:space:]]*#' | grep -q …`). That is the correct predicate rather than a patch: a
+  real stanza's `AllowedIPs` line is never commented, while both the template and every peer's
+  `# <name> — <owner>` label always are.
+- **`platform/vpn-ingress/peers.yaml`**: the example now uses placeholders that cannot match
+  anything real — `# <device-name> — <owner>, enrolled <YYYY-MM-DD>` and
+  `AllowedIPs = 10.90.0.<octet>/32`. A future grep cannot be fooled by it either.
+
+The bug class is worth naming: **a documentation example that is byte-identical to real data will
+eventually be parsed as real data.** The same shape caused two earlier defects in this component —
+`flux-postbuild-substitution` flagged a `${...}` written inside a comment *explaining* that hazard,
+and `no-plaintext-secrets` flagged the `.sops.yaml.example` because the kustomization named it in a
+comment. Three instances now; the rule is to make examples structurally unmatchable, not merely
+commented out.
+
+No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `decisions/**`), so this
+merges on a green gate with no override.
+
+## What I verified
+
+Run in the pinned toolbox against the live tree, 2026-10-06.
+
+- **`.2` now works**, and it went through the REAL key path: `new-peer.sh keith-phone 2` rendered a
+  profile with `Address 10.90.0.2/32`, `AllowedIPs = 10.90.0.2/32` in the printed stanza, full
+  tunnel, at `.cache/wg-peers/keith-phone.conf` (0600). The throwaway keypair was deleted after.
+- **This incidentally closes a gap left open in #60**: the profile could only have been rendered by
+  deriving the server's public key via `sops -d --extract … | wg pubkey` from
+  `wg-server-key.sops.yaml` — so that pipeline is now proven **against the operator's real key**,
+  not just against a placeholder it was expected to reject.
+- **A real duplicate is still refused.** Appending an uncommented
+  `AllowedIPs = 10.90.0.5/32` stanza and asking for octet 5 produced the correct refusal; the tree
+  was restored afterwards and `git diff --stat` showed only the intended template edit.
+- **Zero active allocations remain**: `grep -vE '^[[:space:]]*#' peers.yaml | grep -c AllowedIPs`
+  → `0`.
+- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
+  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips (`gitea-settings`,
+  conformance L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **No peer is enrolled by this branch.** The keypair I generated was a throwaway and was deleted;
+  the operator must run the script themselves, since the device's private key must exist only on
+  the device. So the end of the enrolment path — a stanza merged into `peers.yaml`, Flux rolling
+  the DaemonSet, and `wg show` listing a peer — is still unexercised.
+- **The duplicate-NAME warning** (`grep -qi "^ *# *${NAME} "`) deliberately still inspects
+  comments, because per-peer labels are comments. It is no longer tripped by the template, but I
+  did not test it against a real enrolled peer.
+- **Nothing about client traffic.** The WAN port-forward is still not configured and there is still
+  no off-LAN vantage point, so the tunnel has carried no packet.
@@ -0,0 +1,103 @@
+---
+date: 2026-10-06
+branch: fix/peers-configmap-content-hash
+title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
+---
+
+## What changed
+
+The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
+2026-10-06, with the peer already on `main`:
+
+| observation | value |
+|---|---|
+| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
+| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
+| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
+| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
+| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
+| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
+| init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
+
+The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
+fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
+no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
+into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
+would have had **no effect at all** until something unrelated happened to restart the pods. For a
+*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
+see a green PR, and the device would keep working.
+
+My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
+never true.
+
+**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
+*"The generated name carries kustomize's content hash, so editing any runbook rolls the server
+automatically and a stale projection is impossible."*
+
+So the register becomes a **content-hashed generator**:
+
+- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
+  plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
+  carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
+  documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
+  stripped.
+- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
+  ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
+  `nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
+  the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
+  `wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
+  by building the output and reading the volumes block.
+- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
+  register is now **fatal** instead of skipped. The old guard was
+  `[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
+  duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
+  as a defect in its own right.
+- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
+  `networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
+  operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
+- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
+  `generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
+  merge.
+
+Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
+
+No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
+`decisions/**`), so this merges on a green gate with no override.
+
+## What I verified
+
+Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
+
+- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
+  namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
+  `namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
+  bearing, measured both ways.
+- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
+  append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
+  and reversible, so an edit necessarily changes the pod template.
+- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
+  `peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
+  `main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
+- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
+  NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
+  transformer did not rewrite it).
+- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
+  "append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
+  real `.2` now occupies it — the first time the duplicate check has been exercised against real
+  data rather than against its own example.
+- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
+  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
+  as rc 3.**
+
+## What I did NOT verify
+
+- **That the roll actually happens on the cluster.** The whole point of the change is unproven
+  until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
+  `wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
+- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
+  port-forward is reportedly configured but has never been exercised by a packet I can see, and I
+  have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
+- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
+  should now roll the pods and drop the peer; untested.
+- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
+  `prune: true` on the Kustomization, not observed.
@@ -0,0 +1,126 @@
+---
+date: 2026-10-06
+branch: docs/pm-agent-registry-pull-prereq
+title: The nodes resolve gitea to the WAN, so the failing certificate is the router's — the second node process to need the issuer pin
+---
+
+## What changed
+
+The image pull for `pm-agent` failed TLS verification, and the previous entry on this branch
+honestly recorded it as **not diagnosed**. The operator asked for the diagnosis before any
+registry design. This is it, and the fix turns out to be a pattern this repository already
+has.
+
+`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins
+`gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the
+Keycloak issuer pin that is there for the identical reason.
+
+**It reaches the nodes the way every other node setting in this project does — through
+Ansible, from the inventory.** Node configuration is Ansible's (and, for the dev VMs,
+OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the
+*only* surface: `infra/PROD-PROVISIONING.md` records that prod is 3× bare metal and that the
+dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no
+cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command:
+
+    cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
+      -e k3s_hosts=k3s_prod --tags hosts
+
+`--tags hosts` runs ONLY the `/etc/hosts` task, so applying this pin cannot drift any other
+baseline setting. Running the play is an operator action (it has become/root on prod), but
+the configuration itself is declared, reviewed and version-controlled — not typed on a node.
+
+The docs that described this as an undiagnosed blocker now describe the cause and the fix:
+`platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and
+the play), and `docs/plans/pm-agent-issue-to-pr.md`.
+
+Also recorded, in the two places an operator reads when creating the credential: **the agent
+authenticates to Gitea with a local account and a scoped PAT, by decision** (operator,
+2026-10-06), not with a federated identity. Written down because the alternative looks
+obviously better until you check whether it is possible — see below.
+
+## What I verified
+
+Every step measured, read-only, in this order.
+
+**The failing certificate is the router's, not the Gateway's.**
+
+- containerd's error names TLS, not the digest: `tls: failed to verify certificate: x509:
+  certificate is not valid for any names, but wanted to match gitea.home.keithjackson.dev`.
+  "not valid for **any** names" means the presented certificate carries no SANs at all.
+- `192.168.2.20` is the shared Gateway's own LoadBalancer VIP
+  (`gateway/cilium-gateway-shared`, from the Cilium pool `192.168.2.20-30`), and its `https`
+  listener serves `*.home.keithjackson.dev` with `wildcard-home-tls`. From the operator's
+  workstation the name resolves there and serves the real Let's Encrypt
+  `CN=*.home.keithjackson.dev` with correct SANs. **So the certificate and the Gateway are
+  healthy.**
+- **The node asks a different resolver and gets a different answer.** Prod's
+  `node_baseline_dns_servers` is the LAN gateway, not Pi-hole — deliberately, as that file
+  explains. Public DNS for this zone is a CNAME to `wan.home.keithjackson.dev`: measured via
+  both `1.1.1.1` and `8.8.8.8`, `gitea.home.keithjackson.dev` →
+  `wan.home.keithjackson.dev.` → `136.51.115.252`, and the apex answers the same.
+- **That address serves a certificate with no names.** Measured from the workstation against
+  the WAN address with the right SNI:
+
+      subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
+      issuer=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
+      No extensions in certificate
+
+  No extensions means no `subjectAltName`, which is exactly "not valid for any names". The
+  `CN=%D` is the router's placeholder.
+
+**This repository had already measured the same thing, for Keycloak, and I did not need to
+re-derive it.** `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` says, above the pin:
+"Since platform/gateway/gateway.yaml set an external-dns `target`, every public name is a
+CNAME to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from
+the LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
+k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18)." Same cause, same symptom,
+one name later. The pin exists precisely for node processes that must reach the Gateway
+directly; containerd is one, and nobody had added it.
+
+**The parity the pin has to satisfy:** `192.168.2.20` equals the live
+`GATEWAY_VIP`, and `k3s_base_domain` equals the live `BASE_DOMAIN` — both read from the
+cluster, not from the comment that claims prod serves `prod.home.keithjackson.dev` (it does
+not; `flux-system/cluster-vars` reports `BASE_DOMAIN=home.keithjackson.dev` and
+`GATEWAY_VIP=192.168.2.20`, and Ansible's `k3s_base_domain` is `home.keithjackson.dev`).
+That comment is stale history.
+
+**The credential decision, and why the obvious alternative is not available.** Gitea's API
+accepts basic auth, PATs, SSH signatures and its own OAuth2 bearer tokens; as an OAuth2
+provider Gitea implements **only the Authorization Code Grant** (plus PKCE/OIDC), so there
+is no client-credentials grant, and nothing in Gitea validates a token issued by a
+third-party OIDC provider. A Keycloak service-account token therefore cannot authenticate to
+Gitea's API or to git-over-HTTP. Read from Gitea's own documentation
+(`docs.gitea.com/development/api-usage`, `/development/oauth2-provider`), not assumed.
+
+**Gate:** `./ci/gate.sh` in the pinned toolbox from a §0 worktree — see the branch's run; rc 3
+with `gitea-settings` the only stage that did not run.
+
+## What I did NOT verify
+
+- **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the
+  nodes is written by the `node_baseline` play's `hosts`-tagged task, which is an operator
+  action. I did confirm the mechanism is not inert — `roles/node_baseline/tasks/main.yml`
+  renders `node_baseline_hosts_pins` into a marked `blockinfile` with `backup: true`, and
+  `state: absent` when the list empties, so a removed entry really leaves the node. Until that runs
+  and an image exists, "the node now reaches the Gateway" is a design claim. The evidence
+  that it *will* is the Keycloak pin working for the same cause — which I also did not
+  re-measure.
+- **Why public DNS answers at all for an `internal` service.** `catalog/gitea.yaml` sets no
+  `exposure`, so it inherits `internal`, and the renderer contract says an internal service's
+  HTTPRoute carries `external-dns.alpha.kubernetes.io/controller: none` and publishes no
+  public record. Yet `gitea.home.keithjackson.dev` resolves publicly to the WAN CNAME. I did
+  not chase that: it may be a zone-level wildcard rather than a published record, and it did
+  not change the diagnosis. It is worth someone's attention, because "internal services
+  publish no public DNS" is an L0-checked claim and this looks like a case where the *name*
+  resolves regardless of what the renderer withheld.
+- **Registry authentication.** Unchanged and still open: an unauthenticated client gets
+  nothing from this registry, and no `imagePullSecret` exists anywhere in the tree. With the
+  credential model now decided (local Gitea accounts with PATs) the shape is a
+  `dockerconfigjson` for an account whose token carries `read:package`, referenced from the
+  Deployment and the committed worker Job template — and the worker Job template's shape is
+  pinned by admission, so whether the policy should also pin `imagePullSecrets` deserves a
+  deliberate look rather than a quiet addition. Not built: minting that token is an operator
+  action, and I did not want to guess at a second credential's scope.
+- **Anything about runtime.** The image still does not exist. No dispatcher has
+  authenticated, no issue has been triaged, no worker Job has been created, and the admission
+  policy — live, `Enforce`, six rules — has still never admitted or refused a real Job.
@@ -0,0 +1,113 @@
+---
+date: 2026-10-06
+branch: fix/vpn-ingress-ip-forward
+title: The VPN tunnel never came up — a hostNetwork pod cannot write net.ipv4.ip_forward, and wg-quick treats a failed PostUp as fatal
+---
+
+## What changed
+
+The remote-access VPN reconciled for the first time with a real server key (#60 then #62) and
+**the tunnel did not come up**. One line in my own manifest was the whole cause.
+
+Measured in the `wireguard` container on prod, 2026-10-06:
+
+```
+[#] ip link add dev wg0 type wireguard
+[#] wg addconf wg0 /dev/fd/63
+[#] ip -4 address add 10.90.0.1/24 dev wg0
+[#] ip link set mtu 1420 up dev wg0
+[#] sysctl -q -w net.ipv4.ip_forward=1
+sysctl: permission denied on key "net.ipv4.ip_forward"
+[#] ip link delete dev wg0
+**** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! ****
+```
+
+Everything up to that point worked. **The cause is not a missing capability.** With
+`hostNetwork: true` the runtime mounts `/proc/sys` **read-only** in the container — confirmed in the
+pod's own `/proc/mounts`: `proc /proc/sys proc ro,nosuid,nodev,noexec,relatime` — precisely because
+those sysctls are the *host's* globals and a container must not rewrite them. `CAP_NET_ADMIN` cannot
+lift a read-only mount, so this can never succeed from here. And `wg-quick` treats any failed
+`PostUp` as fatal, so it deleted the interface it had just finished building.
+
+**The line was never needed.** `node_baseline` writes `net.ipv4.ip_forward=1` to
+`/etc/sysctl.d/99-k3s-cilium.conf`, and reading the value from inside the pod returns `1`. I had
+written the header comment *correctly* — "ip_forward is ALREADY 1 on these nodes … so no pod sysctl
+is needed" — and then emitted the sysctl anyway, justified in its own comment as "re-asserted so
+this component does not silently depend on another role's sysctl file surviving". That defensive
+re-assertion is the entire outage. **The lesson is narrow and reusable: in a hostNetwork pod,
+verify host state, never write it.**
+
+Fix:
+
+- **`platform/vpn-ingress/daemonset.yaml`**: the `PostUp = sysctl …` line is gone, replaced by a
+  **read** in the init container — `grep -qx 1 /proc/sys/net/ipv4/ip_forward` — which fails closed
+  with a message naming the `node_baseline` role. Reading `/proc/sys` is permitted; only writing is
+  not. Fail-closed still matters: a tunnel that comes up but cannot forward would satisfy every
+  probe in this component and silently drop every client's traffic.
+- The header bullet that said "no pod sysctl is needed" now also says **do not attempt one**, with
+  the mechanism, so the next author does not re-add it for the same reason I did.
+- The `securityContext` comment claimed NET_ADMIN was for "create the wg0 link, set the host sysctl,
+  and write the netfilter rules". The middle clause was false; corrected.
+- The image's `chown: … Operation not permitted` startup run — plus its own "Permissions could not
+  be set … we will not provide support for it" — is now documented as **expected and harmless**, in
+  both the manifest and the runbook, so nobody chases it. Capabilities are dropped to `NET_ADMIN`
+  only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs as root,
+  and `wg-quick` proceeds past it (as it did here — the failure was 30 lines later).
+- **`runbooks/vpn-remote-access.md`**: a "what a healthy start looks like" block with the real log
+  lines; `ip_forward is 0` added to the init-failure table; and the note that `0/1 Running` with
+  `Startup probe failed: Unable to access interface: No such device` is the signature of `wg-quick`
+  tearing the tunnel down — read the `[#]` line it died on.
+- **`platform/vpn-ingress/README.md`**: a table of what the first reconcile settled (below).
+
+No sensitive paths: `platform/vpn-ingress/**`, `runbooks/**` and `decisions/**` only. This merges on
+a green gate with no override.
+
+## What I verified
+
+Live prod, read-only, 2026-10-06. **Four questions this component's design rested on are now
+measured rather than assumed — three went the right way, one was the defect above.**
+
+- **The GitOps path is proven end to end.** `Kustomization vpn-ingress` `lastAppliedRevision:
+  main@sha1:cf7d5e4` equals the GitRepository artifact revision. The Secret carries
+  `kustomize.toolkit.fluxcd.io/name: vpn-ingress`, so Flux delivered it.
+- **SOPS decryption works, proven by digest rather than by eye.** The value in
+  `platform/vpn-ingress/wg-server-key.sops.yaml` on `main`, decrypted, is 44 bytes, is **not** the
+  placeholder, and **`wg pubkey` accepts it**. Its SHA-256 equals the SHA-256 of the live Secret's
+  decoded `data.privatekey` (`d7c8f8de6f252649…` both sides). No key material was printed at any
+  point — lengths, booleans and digests only.
+- **The `wg pubkey` guard fired correctly, and its `ENC[` branch was never needed.** During the
+  kubelet window before the updated Secret propagated, all three pods reported *"It is still the
+  committed REPLACE-ME PLACEHOLDER"* — the guard distinguishing a stale/unfilled value from
+  ciphertext, exactly as designed. It cleared on its own once the mount refreshed; **a transient
+  `REPLACE-ME` CrashLoop for a minute or two after the key PR merges is expected, not a fault.**
+- **The kernel module reaches the pod**: `ip link add dev wg0 type wireguard` succeeded and
+  `/proc/modules` lists `wireguard`. So the `node_baseline` addition works and `CAP_SYS_MODULE` is
+  correctly not required.
+- **`allowPrivilegeEscalation: false` does NOT break the image's s6 init** — the container reaches
+  `wg-quick` normally. The flag stays; the fear recorded in #55 is closed.
+- **The iptables backends MATCH, which was my single biggest documented doubt.** The image ships
+  `iptables v1.8.13 (nf_tables)` via `/usr/sbin/xtables-nft-multi`, and Debian 13 is nf_tables. The
+  nightmare case — rules accepted into a table the kernel never consults, so handshakes succeed and
+  every forwarded connection hangs — cannot occur on this pairing.
+- **`net.ipv4.ip_forward` is `1`** in the host netns the pod shares, and `/proc/sys` is `ro` there.
+  Both read directly from the pod.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
+  environmental skips (`gitea-settings`, conformance L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **That the tunnel now comes up.** This fix has not reconciled. The next reconcile is the test, and
+  its success criterion is specific: pods `1/1 Running`, and `wg show` reporting `interface: wg0`
+  with `listening port: 51820`.
+- **That the MASQUERADE and FORWARD rules actually apply.** They are the next statements after the
+  one that failed, so **they have never executed**. The backends matching (above) removes the worst
+  failure mode but proves nothing about the rules themselves. First check after the pods go Ready:
+  `iptables -t nat -S POSTROUTING | grep 10.90.0.0/24` on a node.
+- **Anything about client traffic.** Still zero peers enrolled, still no WAN port-forward, still no
+  off-LAN vantage point. The VPN has carried no packet.
+- **Whether the transient `REPLACE-ME` window has an upper bound.** It cleared within ~2 minutes
+  here; I did not determine the kubelet refresh behaviour, so a longer stall after a key change is
+  uncharacterised.
+- **The `sops set` hand-off command**, still. The operator filled the key in by their own route
+  (#62), so the exact quoting I documented inside `dev.sh run bash -c` remains unexercised.
@@ -0,0 +1,85 @@
+---
+date: 2026-10-07
+branch: fix/bulk-nfs-truenas
+title: Repoint bulk NFS at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
+---
+
+## What changed
+
+Both ZFS pools were physically moved off the Proxmox host onto a standalone TrueNAS Community
+Edition at **192.168.2.30**, which sits on prod's own `192.168.2.0/24`. `BULK_NFS_SERVER` moves from
+`192.168.2.102` (the pfSense DNAT prod used to mount through) to `192.168.2.30`.
+
+- `clusters/prod/kustomization.yaml` — `BULK_NFS_SERVER` → `192.168.2.30`. The comment block that
+  justified the DNAT, and the "NOT YET DECIDED / THIS PLACEHOLDER IS LOAD-BEARING" banner, are
+  replaced with the measured export configuration and the recreate requirement below.
+- `clusters/base/cluster-vars.yaml` — base default `192.168.1.9` → `192.168.2.30`. That host no
+  longer exists.
+- `clusters/dev/kustomization.yaml` — same; marked explicitly **inert**, since dev is decommissioned
+  and its nodes are gone. It carries the real address only so nothing in the tree renders a dead one.
+- `docs/BULK-NFS-STORAGE.md` — §1 and §5 rewritten for TrueNAS (exports, service settings, the ZFS
+  `sharenfs` properties that had to be cleared, and the ACL hazard); §7 rewritten now that the
+  placeholder is gone; §9 retargeted from dev to prod; **new §10** documents the PV
+  delete-and-recreate. §2/§3/§4/§6 are unchanged by the move.
+- `catalog/qbittorrent.yaml`, `platform/monitoring/node-pressure-dashboard.yaml` — two comment/prose
+  references to `192.168.2.102` as the live export address.
+
+**The headline operational fact: repointing this knob is NOT a plain reconcile.** A bound PV's NFS
+source is immutable, so Flux's apply fails rather than updating and the pods keep their old, dead
+mount. The five PVs must be deleted and recreated. That sequence is §10 and is an operator action;
+it is **not** performed by this PR.
+
+## What I verified
+
+**On the NAS (192.168.2.30), measured 2026-10-07 by the operator at my request:**
+
+- `zfs list` — `Storage` → `/mnt/Storage`, `Storage/plex` → `/mnt/Storage/plex`,
+  `Storage/nextcloud` → `/mnt/Storage/nextcloud`. The PV paths resolve unchanged.
+- Ownership intact: `/mnt/Storage/plex`, `tv/`, `downloads/`, `movies/`,
+  `downloads/complete/`, `nextcloud/nextcloud/` all `drwxr-xr-x 1000:3000`. `movies/` 1171 entries
+  and `tv/` 107, against 1165 / 107 measured on Proxmox.
+- `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`), `aclinherit=passthrough`, and no
+  non-trivial ACLs — mode bits still govern.
+- Three stale `sharenfs` dataset properties (`plex`, `nextcloud`, `docker-volumes`) carried over from
+  Proxmox; cleared with `zfs inherit`, re-checked empty.
+- `exportfs -v`: both shares to `192.168.2.10/.11/.12` with `sec=sys,secure,root_squash,no_all_squash`,
+  `rw` on plex and `ro` on nextcloud.
+- `/proc/fs/nfsd/versions` → `+3 +4 +4.1 +4.2`, so the PVs' `nfsvers=4.1` negotiates.
+- **A defect was found and fixed during setup:** the plex share listed `192.168.2.13` — an address no
+  node holds — and so omitted `192.168.2.12` (k3s-prod-3). Corrected and re-verified.
+
+**From the cluster:**
+
+- Node InternalIPs are `192.168.2.10/.11/.12` (`kubectl get nodes`), confirming `.13` was wrong.
+- All five NFS PVs currently name `192.168.2.102`.
+- `192.168.2.30:2049` and `:111` accept connections; `192.168.2.102:2049` does not.
+- **`spec.nfs` immutability, by server-side dry-run** — this is the finding that shapes the change:
+  `spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation`.
+- The pods report `Ready` while their mounts are dead: a `stat /media` answers from attribute cache,
+  but a `readdir` blocks indefinitely (the PVs mount `hard`). Hence "scale to zero first" in §10.
+
+**Gate:** see the PR; run in the pinned toolbox from this worktree.
+
+## What I did NOT verify
+
+- **Nothing has mounted 192.168.2.30.** No pod has touched the new server — the PVs still point at
+  the old address and cannot be updated in place. Every §9 probe (numeric-id passthrough, the uid-1000
+  write, the hardlink/EXDEV check, Plex's read-only posture) is **outstanding**, not passing. The
+  equivalents passed against the old host on 2026-09-12; that is not evidence about this one.
+- **The §10 sequence is reasoned, not executed.** Retain + the renderer's pre-set `claimRef` are read
+  from the manifests and from `catalog/render.py`; I have not watched a PVC rebind after a PV delete
+  in this cluster.
+- **`root_squash` is a behaviour change from the Proxmox export**, which carried `no_root_squash`. The
+  argument that nothing needs root on the tree (LSIO drops to 1000; the images chown only their
+  Longhorn `/config`; both subPath directories already exist at 0755) is reasoned from the images and
+  the measured permissions. It has not been exercised. If a mount or subPath resolution fails with
+  EACCES, Maproot `root:root` reproduces the old export and is the first thing to try.
+- **Whether NFSv3 can be disabled.** Enabled Protocols is `NFSv3, NFSv4`; the PVs only need v4.1, so
+  narrowing would close the rpcbind/mountd/statd surface. Not done — it is a change to make after the
+  cutover is proven, not during it.
+- **Whether pfSense / the legacy stack is actually retired.** I measured only that
+  `192.168.2.102:2049` stopped answering. The other legacy consumers of these datasets
+  (`192.168.1.6`, `10.123.1.10`, `192.168.1.18`) were dropped from the new exports; if any is still
+  live, it has lost access. Not checked.
+- **L2/L3 conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline subset
+  only. Nothing in the gate exercises an NFS mount.
@@ -0,0 +1,99 @@
+---
+date: 2026-10-17
+branch: fix/oauth2-proxy-oidc-startup-race
+title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
+---
+
+## What changed
+
+Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
+discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
+serving yet, so the shared Gateway answers 503 and the proxy dies:
+
+```
+[provider.go:55] Performing OIDC Discovery...
+[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
+             failed to discover OIDC configuration: unexpected status "503": no healthy upstream
+```
+
+It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
+replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
+retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
+is already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+
+`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
+polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
+accrue CrashLoopBackOff backoff and retries promptly, which is the point.
+
+**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
+case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
+for a pod wedged in `Init` forever, which is not an improvement.
+
+## Why this shape and not the alternatives
+
+* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
+  example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
+  pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
+  at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
+  "reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
+* **No new base image.** The chart already runs an `alpine` init container for its built-in
+  `initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
+  `3.22` (verified to exist upstream) rather than the chart's own `latest`.
+* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
+  this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
+  the chart already has a hook for.
+* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
+  more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
+  oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
+  the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
+  the operator out of all of them, so the reversible, additive option wins.
+
+The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
+with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
+and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
+silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
+the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.
+
+## What I verified
+
+* **The failure is measured, not theorised.** The log above is from the live pod
+  (`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
+  with `--previous` after the cold boot. The surviving replica's log shows the same startup
+  succeeding once Keycloak was up, which is what identifies this as a race rather than a config
+  fault.
+* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
+  clean and both replicas have been healthy since.
+* **The chart key is real**, at the pinned version range: `helm search repo` resolves
+  `>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
+  `extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
+* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
+  assumed from memory.
+* The edited HelmRelease parses as YAML.
+
+## What I did NOT verify
+
+* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
+  seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
+  replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
+  seven apps, so **do it when you can watch it**, not unattended.
+* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
+  oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
+  from the measured failure, not demonstrated against it.
+* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
+  `ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
+  failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
+  degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
+  anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
+  init container's log on first apply**: it should print "Keycloak discovery reachable" within
+  seconds on a warm cluster, not the give-up message.
+* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
+  re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
+  good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
+  (`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
+  Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
+  this should hold — unverified.
+* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
+  not a measured bound.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
+  subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
@@ -0,0 +1,108 @@
+---
+date: 2026-10-18
+branch: fix/coredns-ha
+title: "coredns ran at one replica, so any node drain was a cluster-wide DNS outage"
+---
+
+## What changed
+
+**MEASURED on prod 2026-10-18: `kube-system/coredns` had `replicas: 1`.** In-cluster DNS was a
+single pod on a single node, so draining or rebooting that node was a **cluster-wide in-cluster DNS
+outage** until the pod rescheduled and became ready — every Service lookup, every CNPG replica
+finding its primary, every operator addressing the API by name. It was invisible because `1/1
+Ready` looks perfectly healthy.
+
+It also became urgent rather than latent: the scheduled reboot automation on
+`feat/automated-upgrades` drains and reboots one node at a time, unattended, overnight — turning a
+single point of failure into a recurring one.
+
+New `platform/coredns/hpa.yaml`: a `HorizontalPodAutoscaler` on the coredns Deployment with
+`minReplicas: 2`, `maxReplicas: 3`, registered in the existing `platform/coredns/kustomization.yaml`
+alongside the split-horizon ConfigMap already there.
+
+**Why an HPA and not `replicas: 2` — this is the whole decision.** k3s owns the coredns Deployment
+and we cannot durably edit it: it is applied by k3s's own deploy controller from a bundled manifest
+(`AddOn/coredns`, source `/data/k3s/server/manifests/coredns.yaml`, `managedFields` managers
+`deploy@k3s-prod-1` and `k3s`, plus an `objectset.rio.cattle.io/hash` label). k3s re-stages its
+bundled manifests on startup, so a `replicas: 2` written there — or a `kubectl scale` — is reverted
+on the next k3s restart or upgrade. Two alternatives were weighed and rejected:
+
+* **`coredns.yaml.skip` + our own templated copy** (the pattern `roles/kube_vip` already uses for a
+  file in `server/manifests`). Deterministic and never flaps, but it takes ownership of coredns away
+  from k3s permanently — k3s upgrades would stop updating it and the version would need refreshing
+  by hand forever. Too high a price for a replica count.
+* **A Flux-managed patch of the Deployment.** Rejected outright: Flux and k3s's deploy controller
+  would both own one object and fight continuously — the exact shape of the `default-deny`
+  NetworkPolicy drift loop this cluster already suffered (~2500 events/10min).
+
+An HPA avoids both: a separate object k3s does not own, driving the `/scale` subresource rather than
+the Deployment spec, so there is no shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA
+sees it below `minReplicas` and scales back within one reconcile.
+
+It is a **floor-holder, not an autoscaler**: the CPU metric exists only because an `autoscaling/v2`
+HPA needs one to be valid. `scaleDown.stabilizationWindowSeconds: 600` because DNS is the worst
+thing in the cluster to flap; `scaleUp` is immediate, since that is the path that restores redundancy
+after a k3s restart.
+
+**Immediate action taken outside this PR:** `kubectl -n kube-system scale deploy coredns
+--replicas=2` was run at the operator's explicit request, so the cluster has two replicas **now**
+rather than after merge. That hand scale is what this PR makes durable.
+
+## What I verified
+
+* **The defect:** `kubectl -n kube-system get deploy coredns` → `replicas=1 available=1`.
+* **k3s ownership:** the `AddOn/coredns` object exists with source
+  `/data/k3s/server/manifests/coredns.yaml`; the Deployment has no `ownerReferences` but carries
+  `objectset.rio.cattle.io/hash` and `managedFields` managers `deploy@k3s-prod-1` / `k3s`. So it is
+  k3s-managed, not Flux-managed — independently corroborated by another agent, which confirmed
+  `platform/coredns/` contains **only** the `coredns-custom` ConfigMap and does not touch the
+  Deployment or its replica count.
+* **The hand scale works and holds:** coredns went to `2/2 Running` on two different nodes, and
+  in-cluster DNS kept answering throughout (`nslookup kubernetes.default.svc.cluster.local
+  10.43.0.10` → `10.43.0.1`). Still `spec.replicas=2 ready=2` after ~2 minutes, with the AddOn
+  checksum **unchanged** — so k3s's deploy controller does not revert on a drift timer; it
+  re-applies on manifest change and on restart.
+* **The HPA's preconditions hold:** coredns ships `requests: {cpu: 100m, memory: 70Mi}` (so a
+  Resource/Utilization metric is meaningful), and metrics-server is installed as a k3s addon and
+  verified serving — `kubectl top pod -l k8s-app=kube-dns` returns **5m and 4m** against that 100m
+  request, i.e. ~5% utilisation, confirming the 80% target will never be approached and the HPA will
+  sit at `minReplicas`.
+* **The kustomization edit is additive:** `git diff --stat` is `1 file changed, 4 insertions(+)`,
+  and `- coredns` was already registered in `platform/kustomization.yaml`, so no change was needed
+  there.
+* All three touched/added YAML files parse.
+
+## What I did NOT verify
+
+* **The HPA has never been applied.** I did not reconcile it, so I have **not** observed it create,
+  reach `ScalingActive=True`, resolve its metric against real coredns data, or hold `minReplicas`.
+  Applying is an operator action (CLAUDE.md §5). The failure mode to watch for is
+  `ScalingActive=False` with `FailedGetResourceMetric` — the HPA would then be present and inert,
+  which looks identical to working because the hand scale already left 2 replicas in place. **Check
+  `kubectl -n kube-system describe hpa coredns` after merge**, not just the replica count.
+* **The central claim that k3s reverts `replicas` on restart is REASONED, NOT OBSERVED.** It rests
+  on k3s re-staging bundled manifests at startup plus the measured ownership evidence above. I did
+  **not** restart k3s to watch the revert happen — doing so is a rolling control-plane restart, an
+  operator action. If k3s in fact preserves a modified replica count, this HPA is unnecessary
+  (though harmless).
+* **I did not confirm the HPA wins the race against k3s's deploy controller after a restart.** Both
+  write the scale; the expectation is the HPA re-asserts within a reconcile, leaving a brief window
+  at 1 replica. Neither the window's length nor the outcome has been measured.
+* **I nearly shipped a destructive change.** My first write of
+  `platform/coredns/kustomization.yaml` replaced the file wholesale, listing only `hpa.yaml` — which
+  would have pruned the existing `coredns-custom` ConfigMap and **broken in-cluster split-horizon
+  DNS (R-DNS-01/03)**. It was caught by a guard assertion before commit and restored from `HEAD`, and
+  the diff is now additive. Recorded because the near-miss is the useful part: `platform/coredns/`
+  already existed and I assumed it did not. **No conformance check would obviously have caught the
+  removal** — it is valid YAML and a legitimate-looking prune.
+* **Whether 2 is enough.** Three nodes, and coredns' anti-affinity only *prefers* spreading, so both
+  replicas could in principle land on one node and a single drain would still take DNS down. I did
+  not add a `topologySpreadConstraint` or a PodDisruptionBudget — both arguably belong here and
+  neither is in this change.
+* **No alert fires if coredns drops to 1 replica**, so a future silent regression (k3s upgrade
+  changing the manifest, the HPA going inert) would not be reported. That gap is not closed here.
+* **No conformance test asserts this**, so a future edit could remove the HPA with no gate stage
+  noticing.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
+  subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
@@ -1,61 +1,43 @@
 # Bulk NFS storage — the media library and the legacy Nextcloud archive

-The operator-facing companion to the `class: bulk` storage entries in `catalog/{plex,sonarr,radarr,nextcloud}.yaml`.
-Everything here was **measured** on the storage host on 2026-09-07, over read-only SSH; nothing is
-assumed. It resolves the long-standing **OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
+The operator-facing companion to the `class: bulk` storage entries in
+`catalog/{plex,sonarr,radarr,qbittorrent,nextcloud}.yaml`.
+
+**Rewritten 2026-10-07 for the move off Proxmox onto a standalone TrueNAS at 192.168.2.30.** The
+layout (§2), the mount design (§3, §4) and the uid model (§6) are unchanged by the move and were
+measured on the old host on 2026-09-07. Everything about the storage host itself — §1, §5, §10 —
+was re-measured on the NAS on 2026-10-07. Nothing here is assumed. It resolves the long-standing
+**OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
 case that actually exists here — attaching datasets that already hold data — and leaves dynamic bulk
 provisioning unimplemented, because nothing needs it.

 ---

-## 1. The storage host, and why prod cannot reach it
+## 1. The storage host

-The bulk data lives on the **Proxmox host's ZFS pool**. That host has exactly one interface:
+The bulk data lives on a **standalone TrueNAS** (Community Edition) at **192.168.2.30** — the same
+`192.168.2.0/24` as prod's nodes (`192.168.2.10/.11/.12`), so it is reached directly, with no NAT
+and no routing decision.

-```
-vmbr0  192.168.1.9/24        default via 192.168.1.1   (pfSense LAN)
-```
-
-| | nodes | reaches 192.168.1.9? |
-|---|---|---|
-| **dev** | 192.168.1.40 / .41 / .42 | **Yes** — same L2 segment, and it already holds an NFS export from this host (`/mnt/Storage/k8s-volumes`, in `/etc/exports`) |
-| **prod** | 192.168.2.10 / .11 / .12 | **No** — 192.168.2.0/24 is pfSense's **WAN**. Ports 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. 2049 and 111 do not. |
-
-Opening the storage path for prod is a **pending operator decision**. Its two candidate outcomes give
-different values for the `BULK_NFS_SERVER` cluster-var:
-
-| outcome | `BULK_NFS_SERVER` for prod |
-|---|---|
-| port-forward `192.168.2.102:2049` → `192.168.1.9:2049` | `192.168.2.102` |
-| a WAN→LAN route to 192.168.1.0/24 | `192.168.1.9` |
-
-Until it lands, `clusters/prod` holds `TBD-PROD-BULK-NFS-SERVER`. **See §7 for why that placeholder
-behaves differently from the repo's other ones.**
+**This section used to be the hard part of this page.** The datasets were on the Proxmox host, whose
+only interface was `vmbr0 192.168.1.9/24` on pfSense's **LAN**, while prod sits on pfSense's **WAN**.
+tcp/2049 did not traverse that, so prod mounted through a DNAT at `192.168.2.102`, and which address
+belonged in `BULK_NFS_SERVER` depended on an undecided operator choice. Both pools were physically
+moved on 2026-10-07 (`docs/plans/proxmox-to-n5-air-migration.md`, Phase C/D) and all of that is
+simply gone. Measured 2026-10-07: **192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and
+:111 do.**

 ### It only needs ONE port

 The PVs mount `nfsvers=4.1`. **NFSv4 carries mount and lock in-protocol**, so the only thing that
 must be reachable is **tcp/2049** — no rpcbind/111, no mountd, no statd. NFSv3 would have needed
-four ports, three of them on ephemeral numbers, which is a materially worse firewall story. The
-server advertises `+3 +4 +4.1 +4.2` (`/proc/fs/nfsd/versions`), so v4.1 is available and is the
-conservative floor.
+four ports, three of them on ephemeral numbers. The NAS advertises `+3 +4 +4.1 +4.2`
+(`/proc/fs/nfsd/versions`, measured 2026-10-07), so 4.1 is available and is the conservative floor.

-### The export's client list is the SAME under both outcomes — measured, not assumed
-
-**pfSense DNATs the existing port-forward WITHOUT source NAT.** Connecting from this workstation
-(192.168.2.132) to `192.168.2.102:22`, the storage host logged the peer as `192.168.2.132`:
-
-```
-SSH_CONNECTION=192.168.2.132 39768 192.168.1.9 22
-ss -tn  ->  192.168.1.9:22   192.168.2.132:39768
-```
-
-The original WAN-side source address survives the translation (return traffic finds its way back via
-the host's default route, 192.168.1.1). So under **either** outcome the NFS server will see prod's
-**real** node addresses — 192.168.2.10/.11/.12 — and the `sharenfs` change in §5 is the same either
-way. **It does not have to wait for the routing decision.**
-
---
+**NFSv4 is not on by default and its absence does not look like a storage problem.** TrueNAS ships
+NFSv3-only until NFSv4 is added to *Services → NFS → Enabled Protocols*. Without it the mount never
+completes and the pods sit in `ContainerCreating` — no permission error, nothing in the app logs,
+just no volume.

 ## 2. The layout, as found

@@ -107,16 +89,18 @@ why it is mounted read-only; see §6.
 | plex | `/mnt/Storage/plex` | `/media` | — | **ro** | ReadOnlyMany |
 | sonarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
 | radarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
+| qbittorrent | `/mnt/Storage/plex` | `/downloads` | `downloads` | rw | ReadWriteMany |
 | nextcloud | `/mnt/Storage/nextcloud` | `/mnt/legacy-nextcloud` | `nextcloud` | **ro** | ReadOnlyMany |

 In-app paths that follow: Plex libraries at `/media/movies`, `/media/tv`, `/media/photos`; Sonarr
 Root Folder `/media/tv`; Radarr Root Folder `/media/movies`; both importing from
 `/media/downloads/complete/{tv-sonarr,radarr}`.

-**All three media consumers use the identical mountPath on purpose.** When qbittorrent is eventually
-given the same export (it is not in this change), the arrs and the download client will already agree
-on absolute paths, which is the second precondition for hardlinked imports — otherwise every import
-needs a Remote Path Mapping and quietly falls back to copying when one is missing.
+**The three library consumers use the identical mountPath on purpose**, and qbittorrent has since
+been given the same export (narrowed by `subPath: downloads`, presented at `/downloads` to match its
+existing `Session\DefaultSavePath`). The arrs and the download client therefore agree on absolute
+paths, which is the second precondition for hardlinked imports — otherwise every import needs a
+Remote Path Mapping and quietly falls back to copying when one is missing.

 **`/media` and not `/data`, and that is not cosmetic.** The arrs' Longhorn config PVC renders at the
 renderer default `/data/config`. A media mount at `/data` would be mounted **over** it and mask the
@@ -147,62 +131,123 @@ Narrowing that would cost hardlinks, which is the worse trade.

 ---

-## 5. The export change the operator must apply — STRICTLY ADDITIVE
+## 5. The exports, as configured on TrueNAS — and the settings that decide permissions

-**Do not apply this from the repo.** It is an operator step on the storage host, recorded here so the
-exact value is reviewable. The existing consumers **must keep working**: 192.168.1.6, 10.123.1.10 and
-192.168.1.18 on the media tree, and 192.168.1.6 and 192.168.1.5 on the Nextcloud dataset, are the
-live non-k3s prod stack.
+**Do not apply this from the repo.** It is operator configuration on the NAS, recorded here so the
+exact values are reviewable. Everything below was **measured on 192.168.2.30 on 2026-10-07**, after
+the move.

-Current values (`zfs get sharenfs Storage`):
+### The exports

 ```
-Storage/plex       rw=@192.168.1.6:@10.123.1.10:@192.168.1.18,no_subtree_check,no_root_squash
-Storage/nextcloud  [email protected]:@192.168.1.5,no_subtree_check,no_root_squash
+/mnt/Storage/plex       192.168.2.10 192.168.2.11 192.168.2.12
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)
+/mnt/Storage/nextcloud  192.168.2.10 192.168.2.11 192.168.2.12
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,root_squash,no_all_squash)
 ```

-### `Storage/plex` — append the six k3s node addresses to the existing `rw=` list
+Per-node addresses rather than `@192.168.2.0/24`, following the precedent of the old host. A `/24`
+would work and would survive adding a node; it also grants the whole subnet.
+
+**All three nodes must appear on BOTH shares.** Any media pod can be scheduled on any node, so a
+missing node is a latent failure, not an immediate one: it surfaces the first time a pod reschedules
+there, as `access denied by server while mounting`, with the cause weeks stale. One was caught
+exactly this way during setup — the plex share listed `192.168.2.13`, an address no node holds, and
+so was missing `192.168.2.12` (k3s-prod-3). Nothing was failing at the time.
+
+### The option flags ARE the permission contract
+
+| flag | why it matters here |
+|---|---|
+| **`no_all_squash`** | The one the whole model rests on. Client uids pass through untouched, so the arrs' `PUID=1000` arrives as the owner of a `0755 1000:3000` tree. In the UI this is **Mapall User/Group left EMPTY** — setting either squashes every client to one identity and erases the reader/writer distinction entirely. |
+| **`sec=sys`** | AUTH_SYS. Numeric uid/gid, not Kerberos principals. §6 is only true because of this. |
+| **`root_squash`** | Maproot left empty. The Proxmox export carried `no_root_squash`; **nothing here needs it** — every write to the tree happens after the LSIO drop to uid 1000, and the images only chown their Longhorn `/config`. This is a deliberate tightening over the old export, not parity. |
+| **`rw` / `ro`** | The archive's read-only posture is **server-enforced**, as it was before. The media tree cannot get the same treatment — see below. |
+| **`secure`** | Reserved source ports required; "Allow non-root mount" off. The kubelet mounts as root from a reserved port and does not need the exception. |
+
+**Plex's read-only posture is still a CLIENT-side guarantee.** Plex is ro and the arrs are rw, but
+all four can land on any node, so the server sees the same three source addresses for all of them
+and cannot distinguish the reader from the writers. `spec.nfs.readOnly: true` on `plex-media` plus
+the volumeMount flag is the whole enforcement. Unchanged by the move; an honest limitation, not an
+oversight. If it ever needs to be server-enforced, the lever is a dedicated node pool for Plex.
+
+### Service settings that are NOT obvious from the export list
+
+- **Enabled Protocols must include NFSv4** (above).
+- **Require Kerberos for NFSv4: off.** The numeric AUTH_SYS pass-through is the model.
+- **Manage Groups Server-side: off.** This is `rpc.mountd --manage-gids`: the server *discards the
+  client's supplied group list* and looks that uid's groups up in TrueNAS's own user database
+  instead. With no local uid 1000 defined, the arrs' `PGID=3000` would simply vanish. Writes would
+  survive anyway — uid 1000 **owns** the tree, so the owner bits grant access without the group —
+  but it is a free way to lose group semantics.
+- **NFSv4 DNS Domain: blank.** It only feeds `idmapd.conf`, which numeric mapping bypasses.
+- **Bind IP 192.168.2.30** means loopback does not answer: check with `showmount -e 192.168.2.30`,
+  not `localhost`.
+
+There is **no** "NFSv3 ownership model for NFSv4" control on current TrueNAS — it was an older
+TrueNAS/CORE option, and no replacement is needed: on a Linux client against a Linux server over
+AUTH_SYS, both the `nfs` and `nfsd` modules default to `nfs4_disable_idmapping=Y`, so NFSv4 carries
+raw numeric ids rather than `user@domain` principals. §9's `ls -n` probe is how you confirm it.
+
+### The ZFS `sharenfs` properties must be CLEARED, not edited
+
+The pools arrived with the old host's exports still set as dataset properties:
+
+```
+Storage/plex              [email protected]:@10.123.1.10:@192.168.1.18:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
+Storage/nextcloud         [email protected]:@192.168.1.5,[email protected]:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
+Storage/docker-volumes    [email protected]:@192.168.1.5:@192.168.1.10:@192.168.1.18,no_subtree_check,no_root_squash
+```
+
+TrueNAS generates `/etc/exports` from its own middleware database and will fight a property it did
+not set — the failure mode is two differently-configured exports of one path, with no indication
+which is in force. Cleared 2026-10-07:

 ```sh
-zfs set sharenfs='[email protected]:@10.123.1.10:@192.168.1.18:@192.168.1.40:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/plex
+zfs inherit sharenfs Storage/plex
+zfs inherit sharenfs Storage/nextcloud
+zfs inherit sharenfs Storage/docker-volumes
 ```

-Every pre-existing host is preserved verbatim and no option changes; only clients are added.
+`Storage` is `off`, so each resolves to `off`.

-### `Storage/nextcloud` — add the k3s nodes to a **`ro=`** list, not to `rw=`
+### Do NOT touch the dataset's permissions or ACLs

-```sh
-zfs set sharenfs='[email protected]:@192.168.1.5,[email protected]:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/nextcloud
-```
+The tree's ownership **is** the access story (§6), and TrueNAS offers two ways to destroy it
+irreversibly across 8.33 T.

-This is the one place read-only can be enforced **server-side**, and it is worth taking: only
-Nextcloud consumes this dataset from k3s, so the export itself can refuse writes. Even a
-misconfigured client cannot then damage the archive.
+`Storage/plex` carries `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`) and
+`aclinherit=passthrough` (inherited). Measured 2026-10-07: the directories carry **trivial ACLs
+only** — `nfs4xdr_getfacl` shows just `owner@`/`group@`/`everyone@` matching the mode — so `0755
+1000:3000` governs exactly as it did under Proxmox.

-**The media tree cannot get the same treatment.** Plex is read-only and the arrs are read-write, but
-all three may be scheduled on any node, so the server sees the same source addresses for all of
-them and cannot distinguish the reader from the writers. **Plex's read-only posture is therefore a
-client-side guarantee** (`spec.nfs.readOnly: true` on `plex-media`, plus the volumeMount flag) — an
-honest limitation, not an oversight. If it ever needs to be server-enforced, the lever is a
-dedicated node pool for Plex, which is a much larger change.
+- **Stay out of the dataset's Edit Permissions / Edit ACL screen.** On an `acltype=nfsv4` dataset
+  that is a full ACL editor, and its presets write *real* NFSv4 ACLs, recursively if permitted.
+  There is no undo.
+- `aclinherit=passthrough` means a single ACL written at `/mnt/Storage/plex` would be **inherited by
+  everything created under it afterwards** — the blast radius is the whole future library, not one
+  directory.
+- Note GNU `ls`'s `+` marker is **not** a reliable ACL indicator here: it looks for a POSIX ACL
+  xattr, and this dataset stores NFSv4 ACLs elsewhere. Use `nfs4xdr_getfacl`.

-Per-node addresses rather than `@192.168.2.0/24` follows the precedent already on this host
-(`/etc/exports` lists the dev nodes individually, as `/32`). A `/24` would work and would survive
-adding a node; it also grants the whole subnet. Per-node is the tighter default.
+`xattr=on` rather than `sa` is a performance detail, not a permissions one, and changing it would
+not affect existing files.

 ### Verifying, without changing anything

 ```sh
-exportfs -v                  # the new client lines appear, existing ones unchanged
-zfs get sharenfs Storage/plex Storage/nextcloud
+cat /proc/fs/nfsd/versions          # +4.1 must be present
+exportfs -v                         # the two shares, three nodes each, flags as tabulated above
+showmount -e 192.168.2.30           # NOT localhost -- the service is bound to one address
+zfs get -r -o name,property,value,source sharenfs Storage    # every row off or -
 ```

---
-
 ## 6. uid / gid — the failure this design exists to avoid

-The export uses `sec=sys` and `no_root_squash`, so **the client's numeric uid is passed through
-verbatim** and the server's on-disk ownership is the entire access story. The media tree is
+The export uses `sec=sys` and `no_all_squash`, so **the client's numeric uid is passed through
+verbatim** and the server's on-disk ownership is the entire access story. (It no longer carries
+`no_root_squash`, which the Proxmox export did — see §5. Nothing here needs it, and the uid that
+matters is 1000, not 0.) The media tree is
 `0755 1000:3000` throughout (some directories `1000:1000`), so **only uid 1000 can write it**.

 **`fsGroup` cannot fix this, and reaching for it is how this gets misdiagnosed.** The in-tree NFS
@@ -233,24 +278,27 @@ Nextcloud needs no uid work at all, which is one more reason its mount is read-o

 ---

-## 7. The prod placeholder does NOT fail at apply time — read this before merging
+## 7. Why this knob is unvalidated at create AND immutable at update
+
+**The `TBD-PROD-BULK-NFS-SERVER` placeholder is gone** (2026-10-07) — prod now carries a real
+address, because the NAS is on prod's own subnet and there is no longer a decision to defer. The
+hazard that made it load-bearing is worth keeping on the page, because it is the half that explains
+why §10 exists.

 Every other `TBD-PROD-*` in `clusters/prod/kustomization.yaml` is rejected loudly by CRD validation
-(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one is not.**
+(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one was not.**

-A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepts
-`TBD-PROD-BULK-NFS-SERVER`, the PV is created, and the PVC **binds** to it — binding is pure
-control-plane and never touches the server. The failure appears only when a pod mounts:
+A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepted
+`TBD-PROD-BULK-NFS-SERVER`, the PV was created, and the PVC **bound** to it — binding is pure
+control-plane and never touches the server. The failure appeared only when a pod mounted:

 ```
 MountVolume.SetUp failed ... Failed to resolve server TBD-PROD-BULK-NFS-SERVER
 ```

-so **plex, sonarr, radarr and nextcloud would sit in ContainerCreating on prod.**
-
-**Consequently: set `BULK_NFS_SERVER` in `clusters/prod/kustomization.yaml` BEFORE prod reconciles a
-revision that carries these bulk volumes.** Dev renders a working value and is where these manifests
-are meant to be proven first.
+**The same field is immutable on UPDATE** (§10), which is the other half of the same awkwardness: a
+wrong value is accepted silently at create, and then cannot be corrected in place. Together they are
+why repointing this knob is a delete-and-recreate rather than a reconcile.

 ### The static guard this convention leans on is missing from `main` — but the gate now refuses the value anyway

@@ -287,10 +335,9 @@ Without the arm, HALF A is what holds the line, by design.
   EPHEMERAL layer"). Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it
   means those apps start from an empty config volume, so it belongs in its own change with the
   operator present. **This change does not make it worse** — that is why the media mount is `/media`.
-2. **qbittorrent has no bulk mount.** Today the arrs' import source is written by the **legacy
-   non-k3s** download client, which is coherent while the legacy stack stays live. Giving the k3s
-   qbittorrent the same export at the same `/media` path is the natural follow-up, and the paths were
-   chosen so it needs no Remote Path Mapping.
+2. **RESOLVED — qbittorrent now has the bulk mount** (`subPath: downloads` at `/downloads`), so the
+   import source is written by the k3s download client rather than the legacy non-k3s one. See
+   `catalog/qbittorrent.yaml`.
 3. **`/mnt/Storage/plex/downloads/downloads` is a symlink to `/shared/downloads`**, which is outside
   the export. Over NFS it is a dangling symlink on every client. Harmless, but expect Sonarr to log
   it while scanning `downloads/`.
@@ -300,10 +347,10 @@ Without the arm, HALF A is what holds the line, by design.
 5. **`nconnect` was not set.** It would likely help throughput on the media tree, but it is a tuning
   change that should be measured, not guessed, and it changes client behaviour under server
   failure.
-6. **Nothing was verified against a live cluster.** No `kubectl` context is usable from this
-   workstation (browser-OIDC only), and this change is manifest-authoring under an explicit
-   do-not-apply constraint. Every claim about the *host* is measured; every claim about *binding and
-   mounting* is reasoned from the manifests.
+6. **Every claim about the NAS in §5 is measured (2026-10-07); the post-move MOUNT is not.** The PVs
+   still have to be recreated (§10) before any pod mounts 192.168.2.30, so the §9 probes are the
+   outstanding verification, not a formality. The pre-move equivalents did pass against the old host
+   on 2026-09-12.
 7. **The bulk-tier requirement ID is deliberately not typed on any non-`.md` line of this change.**
   `ci/touched-requirements.sh` attributes requirements from R-IDs appearing in added lines, and the
   in-cluster conformance test for the bulk tier is still a gap, so typing it would select a
@@ -312,36 +359,95 @@ Without the arm, HALF A is what holds the line, by design.

 ---

-## 9. Validating on dev, once the export is extended
+## 9. Validating, after the PVs are recreated

-Dev can prove all of this before prod's network path exists. Read-only checks first:
+These run against **prod**; dev is decommissioned. They are the outstanding verification for the
+move — nothing below has been run against 192.168.2.30. Read-only checks first:

 ```sh
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pv
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pvc -A | grep -E 'media|legacy-archive'
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr get pod -o wide
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pv -o custom-columns=\
+NAME:.metadata.name,SERVER:.spec.nfs.server,PATH:.spec.nfs.path,STATUS:.status.phase'
 ```

-Expect **four** PVs — `plex-media`, `sonarr-media`, `radarr-media`, `nextcloud-legacy-archive` (three
-of them naming the *same* export, one per namespace, which is the cross-namespace model) — each
-`Bound` to the same-named PVC, with `RECLAIM POLICY Retain` and an empty `STORAGECLASS`.
+Expect **five** NFS PVs — `plex-media`, `sonarr-media`, `radarr-media`, `qbittorrent-downloads`,
+`nextcloud-legacy-archive` (four of them naming the *same* export, one per namespace, which is the
+cross-namespace model) — each `Bound`, `RECLAIM POLICY Retain`, empty `STORAGECLASS`, and
+**`SERVER 192.168.2.30`**. A PV still showing `192.168.2.102` means the recreate in §10 did not
+happen; Flux will NOT have updated it in place.

-Then the two things that actually matter, from inside a pod:
+Then the four things that actually matter, from inside a pod:

 ```sh
-# (a) the writer can write — this is the uid check, and it is the one that fails in practice
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
-  sh -c 'id; touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+# (a) numeric ids survived the wire -- the idmapping check, new since the TrueNAS move
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- sh -c 'ls -n /media | head -3'
+#     expect uid 1000 / gid 3000 on the entries. 65534 / nobody means ids are being mapped, not
+#     passed through, and every import will fail EACCES while the pod still reports Ready.

-# (b) hardlinks work BETWEEN the library and the download path — the root-mount check
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
+# (b) the writer can write -- the uid check, and the one that fails in practice
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
+  sh -c 'touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+
+# (c) hardlinks work BETWEEN the library and the download path -- the root-mount check
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
  sh -c 'set -e; d=/media/downloads/complete/tv-sonarr/.probe; t=/media/tv/.probe-link;
         echo x > "$d"; ln "$d" "$t"; ls -i "$d" "$t"; rm -f "$d" "$t"; echo HARDLINK-OK'

-# (c) the reader really is read-only
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n plex exec deploy/plex -- \
-  sh -c 'touch /media/.probe 2>&1 | head -1'      # expect "Read-only file system"
+# (d) the reader really is read-only
+kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c 'touch /media/.probe 2>&1 | head -1'
+#     expect "Read-only file system"
 ```

-`(b)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
+`(c)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
 prints `EXDEV`, something has split the single mount back into two.
+
+Reading the failures together is what localises the fault: **(b) EACCES with (a) showing `nobody`**
+is idmapping (§5 service settings). **(b) EACCES with (a) showing `1000`** is on-disk ownership —
+stop and check `ls -ldn` on the NAS before changing anything else.
+
+---
+
+## 10. Repointing BULK_NFS_SERVER means DELETING AND RECREATING THE PVs
+
+**A bound PersistentVolume's NFS source is immutable.** Measured by server-side dry-run on
+2026-10-07:
+
+```
+$ kubectl patch pv sonarr-media --type=json --dry-run=server \
+    -p '[{"op":"replace","path":"/spec/nfs/server","value":"192.168.2.30"}]'
+The PersistentVolume "sonarr-media" is invalid: spec.persistentvolumesource: Forbidden:
+  spec.persistentvolumesource is immutable after creation
+```
+
+Note the pairing with §7: `spec.nfs.server` is **unvalidated at create** (any string is accepted and
+the PVC binds) and **immutable at update**. So editing `BULK_NFS_SERVER` alone does not fail loudly
+and does not work either — Flux's apply errors on the five PVs and the pods keep their previous,
+now-dead mount.
+
+This is safe to do because `persistentVolumeReclaimPolicy: Retain` means deleting a PV never touches
+the data, and `catalog/render.py` emits a **pre-set `claimRef`** on each one, so a recreated PV
+rebinds its own PVC with no manual step.
+
+**Operator sequence** (mutating `kubectl` is an operator action, not an agent action):
+
+```sh
+# 1. Stop the consumers. This also releases the hung hard-mounts against the old server:
+#    the pods report Ready, but any real I/O on /media blocks indefinitely once 2049 stops
+#    answering, because the PVs mount `hard`.
+for ns in plex sonarr radarr qbittorrent nextcloud; do
+  kubectl --context k3s-prod -n $ns scale deploy --all --replicas=0
+done
+
+# 2. Delete the five PVs. Retain + no live claim means nothing is reclaimed or erased.
+kubectl --context k3s-prod delete pv plex-media sonarr-media radarr-media \
+  qbittorrent-downloads nextcloud-legacy-archive
+
+# 3. Let Flux reconcile the revision carrying BULK_NFS_SERVER=192.168.2.30, which recreates them.
+#    Confirm .status.lastAppliedRevision matches the GitRepository artifact before continuing.
+
+# 4. Bring the consumers back, then run every probe in §9.
+```
+
+The PVCs pass through `Lost` between steps 2 and 3; that is expected and resolves when the PV with
+the matching `claimRef` reappears. If a PVC is still `Lost` after Flux reconciles, the PV name or
+its `claimRef` namespace does not match — do **not** delete the PVC to "fix" it.
+
@@ -233,9 +233,21 @@ about runtime behaviour in this document is a design claim, not a measurement.

 ## Follow-ups, in the order they are worth doing

-1. **Build, push, pin.** Operator runs `infra/agent/build-image.sh`, pushes, and one PR
-   replaces both placeholder digests. Expect the build to need iteration: the nodejs module
-   stream and the npm install have never run against this base.
+1. **Make the registry pullable, THEN build, push, pin.** Measured after the merge, and
+   neither half is declared anywhere in this repository
+   (`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`):
+   (i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the
+   router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning
+   `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same
+   pin the Keycloak issuer already carries, applied by
+   `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`;
+   (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
+   the credential decision now being local Gitea accounts with PATs, that means a
+   `dockerconfigjson` for an account whose token carries `read:package`. Only then does the
+   digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
+   placeholder digests. Expect the build itself to need iteration too: the nodejs module
+   stream and the npm install have never run against this base. The same two blockers sit in
+   front of `conformance-runner`, which shares the registry path and has never been pulled.
 2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the
   bot's permission is `write`, not `admin`** — that measurement is the control that keeps
   the agent out of its own approval gate.
@@ -90,9 +90,19 @@ esac
 # pointing at an external host would be REACHABLE: it could clone a repository that is not
 # this one and push this repository's work somewhere else. The allowlist is baked in here,
 # where changing it requires rebuilding the image whose digest admission pins.
+#
+# ONE ENTRY, AND IT IS THE CLUSTER-LOCAL SERVICE NAME. This agent runs INSIDE the cluster, so
+# it reaches Gitea the way every other in-cluster consumer does — `*.svc.cluster.local` —
+# which is by design a different path from how a client outside the cluster reaches the same
+# service. The external hostname `gitea.home.keithjackson.dev` used to be a second accepted
+# value here; it is removed deliberately. It was never needed (nothing sets it), it routes
+# in-cluster traffic out through the Gateway and back for no reason, and as an ACCEPTED value
+# it widened this allowlist to a host reachable over the `world:443` egress rule — which is
+# the exact shape of injection the allowlist exists to refuse. The cluster-local name also
+# depends on none of public DNS, Cloudflare, the Gateway or the certificate.
 case "${PM_GITEA_URL}" in
-  http://gitea.gitea.svc.cluster.local:3000|https://gitea.home.keithjackson.dev) ;;
-  *) die "PM_GITEA_URL '${PM_GITEA_URL}' is not an allowed git host (see this script's header)" ;;
+  http://gitea.gitea.svc.cluster.local:3000) ;;
+  *) die "PM_GITEA_URL '${PM_GITEA_URL}' is not the allowed in-cluster git host (see this script's header)" ;;
 esac
 [[ "${PM_GITEA_OWNER}" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,38}$ ]] || die "PM_GITEA_OWNER is malformed"
 [[ "${PM_GITEA_REPO}"  =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$ ]] || die "PM_GITEA_REPO is malformed"
@@ -86,12 +86,22 @@ refuses "branch equal to the base branch"       "must not be the base branch" \
 # --- where code comes FROM and goes TO -------------------------------------
 # The namespace's egress policy permits tcp/443 to the world (the model needs it), so an
 # injected host would be REACHABLE. These are the cases that matter most.
-refuses "a redirected git host"                 "is not an allowed git host" \
+refuses "a redirected git host"                 "is not the allowed in-cluster git host" \
        PM_GITEA_URL=https://evil.invalid
-refuses "a real, reachable, WRONG git host"     "is not an allowed git host" \
+refuses "a real, reachable, WRONG git host"     "is not the allowed in-cluster git host" \
        PM_GITEA_URL=https://github.com
-refuses "the right host on a smuggled path"     "is not an allowed git host" \
+refuses "the right host on a smuggled path"     "is not the allowed in-cluster git host" \
        PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:3000/../..
+# This agent is IN the cluster, so it reaches Gitea by its cluster-local service name. The
+# EXTERNAL hostname is refused too — it used to be a second accepted value, and as an
+# accepted value it widened this allowlist to a host reachable over the `world:443` egress
+# rule, which is the injection shape the allowlist exists to refuse.
+refuses "this Gitea's EXTERNAL hostname"        "is not the allowed in-cluster git host" \
+        PM_GITEA_URL=https://gitea.home.keithjackson.dev
+refuses "the cluster-local name over https"     "is not the allowed in-cluster git host" \
+        PM_GITEA_URL=https://gitea.gitea.svc.cluster.local:3000
+refuses "the cluster-local name, wrong port"    "is not the allowed in-cluster git host" \
+        PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:8080
 refuses "a malformed owner"                     "PM_GITEA_OWNER is malformed" \
        'PM_GITEA_OWNER=../other-org'
 refuses "a malformed repository"                "PM_GITEA_REPO is malformed" \
@@ -327,6 +327,29 @@ node_baseline_hosts_pins:
  - ip: "192.168.2.20"
    names:
      - "keycloak.{{ k3s_base_domain }}"
+      # gitea: the SECOND node process to need this pin, for the identical reason, found
+      # 2026-10-06. containerd pulls images as a NODE process, so it resolves through the
+      # public upstreams above and gets the CNAME to wan.home.keithjackson.dev — the WAN IP,
+      # which this router does not hairpin. The pull then fails on the router's own
+      # placeholder certificate rather than on anything to do with the image:
+      #
+      #   failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:…":
+      #   tls: failed to verify certificate: x509: certificate is not valid for any names,
+      #   but wanted to match gitea.home.keithjackson.dev
+      #
+      # Measured from the workstation the same day, against the WAN address with that SNI:
+      #   subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
+      #   No extensions in certificate          <- no SANs at all, hence "not valid for ANY names"
+      # while the same name via PiHole answers 192.168.2.20 and serves the real
+      # Let's Encrypt *.home.keithjackson.dev. So the certificate and the Gateway are fine;
+      # only the node's ANSWER is wrong. Same root cause, same fix, as the issuer pin above.
+      #
+      # WHO NEEDS IT: any workload whose image lives in the in-cluster Gitea registry —
+      # today platform/pm-agent (agent-runner) and conformance/incluster
+      # (conformance-runner), neither of which has ever been pulled. A pull is still
+      # UNAUTHENTICATED after this pin, which is a separate thing to settle; this only makes
+      # the node reach the right endpoint.
+      - "gitea.{{ k3s_base_domain }}"

 # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
 # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
@@ -32,9 +32,23 @@ ARG TARGETARCH=amd64
 # Distro packages: libvirt client libs (harmless disk-image utilities), ansible, ssh,
 # disk-image + cloud-init seed tooling, python3-pip (for the pip-installed test tools),
 # diffutils (provides `diff`, used by L1 golden render checks), and common utilities.
+#
+# wireguard-tools + qrencode serve the REMOTE-ACCESS VPN (platform/vpn-ingress):
+#   * wireguard-tools gives `wg genkey` / `wg pubkey`, needed to mint the server key and to derive
+#     every client profile's `[Peer] PublicKey` (infra/vpn-ingress/new-peer.sh).
+#   * qrencode renders a client profile as a QR code, which is how a PHONE gets enrolled.
+# They are here rather than on the operator's host because this image is THE toolchain
+# (CLAUDE.md §5; README "Don't install the toolchain on the host"). That rule was briefly broken
+# by a runbook that told the operator to run `wg` on the host and `sops` in here, and then split
+# every step across the boundary to work around it — a procedure nobody should have to follow, and
+# one that cannot be reproduced in CI. Both tools in one image means `sops -d … | wg pubkey` is
+# just a pipeline again.
+# Unpinned on purpose, like every other dnf package here: the Fedora base is what is pinned
+# (FEDORA_VERSION), and these are not gate tools whose version changes what green means.
 RUN dnf -y install --setopt=install_weak_deps=False \
      libvirt-client qemu-img cloud-utils genisoimage openssh-clients \
      ansible-core python3-libvirt python3-pip diffutils \
+      wireguard-tools qrencode \
      git make jq unzip tar gzip xz which ca-certificates curl \
 && dnf clean all

@@ -14,6 +14,7 @@ baseline, and the *same* binaries CI uses.
 | Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
 | GitOps / cluster | `flux`, `kubectl`, `helm` |
 | Secrets | `sops`, `age` + `age-keygen` |
+| VPN (`platform/vpn-ingress`) | `wg` (wireguard-tools) — mint the server key, derive client public keys; `qrencode` — render a client profile as a phone-scannable QR |
 | **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
 | Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |

@@ -2,10 +2,24 @@
 # =================================================================================================
 # new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
 #
-# Generates a WireGuard keypair ON THIS MACHINE, renders the device's client profile, optionally
-# prints a QR code for a phone, and prints the one `[Peer]` stanza to paste into
-# platform/vpn-ingress/peers.yaml. The private half NEVER leaves this machine: it is not sent to the
-# cluster, not committed, and not written to any log.
+# Generates a WireGuard keypair, renders the device's client profile, prints a QR code for a phone,
+# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.conf. The device's
+# private half is written to ONE 0600 file and nowhere else: not sent to the cluster, not committed,
+# never logged.
+#
+# RUN IT IN THE DEV CONTAINER — that is where the toolchain lives (CLAUDE.md §5):
+#
+#     ./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2'
+#
+# It needs `wg` (wireguard-tools), `qrencode` and `sops`, all three baked into the image. It will
+# run on a host that happens to have them too, but the container is the supported way and the only
+# one that is reproducible.
+#
+# THE SERVER'S PUBLIC KEY IS DERIVED FROM THE ENCRYPTED FILE, not from a copy you keep lying around:
+# `sops -d` the private half out of platform/vpn-ingress/wg-server-key.sops.yaml and pipe it
+# straight into `wg pubkey`. The plaintext never touches disk and never reaches a terminal. (This
+# is why both tools being in ONE image matters — see the Containerfile comment.) Override with
+# WG_SERVER_PUBKEY or WG_SERVER_KEY_FILE if you are working off-tree.
 #
 # WHY A SCRIPT AND NOT A RUNBOOK STEP. Three of these fields are easy to get wrong in ways that are
 # either silently insecure or silently broken, and all three are mechanical:
@@ -25,24 +39,30 @@
 #   <last-octet>    2-254, the device's address inside WG_CLIENT_CIDR. Must be unused.
 #   --split         split tunnel: route ONLY the LAN and the tunnel subnet. Default is a FULL
 #                   tunnel (0.0.0.0/0) so the device also egresses the house WAN.
-#   --out DIR       where to write the profile (default: a 0700 dir under $TMPDIR).
+#   --out DIR       where to write the profile. Default: `.cache/wg-peers/` in the repo, which is
+#                   GITIGNORED (`.gitignore` `.cache/`) and, unlike /tmp inside the container,
+#                   survives the container exiting — the repo is its only persistent read-write
+#                   mount, so a profile written anywhere else is gone before you can move it to a
+#                   desktop. The script refuses any destination git does not ignore.
 #
 # ENVIRONMENT
-#   WG_SERVER_KEY_FILE  server PRIVATE key (default ~/.config/wireguard-home/server.key); its public
-#                       half is derived from it. Alternatively set WG_SERVER_PUBKEY directly.
+#   WG_SERVER_PUBKEY    the server's public key, if you want to skip the SOPS decrypt entirely.
+#   WG_SERVER_KEY_FILE  a local copy of the server PRIVATE key to derive the public half from,
+#                       instead of reading the SOPS file.
+#   WG_SERVER_SOPS_FILE default platform/vpn-ingress/wg-server-key.sops.yaml
 #   WG_ENDPOINT_HOST    default wan.home.keithjackson.dev
 #   WG_CLIENT_CIDR      default 10.90.0.0/24   (must match clusters/base/cluster-vars.yaml)
 #   WG_LAN_CIDR         default 192.168.2.0/24 (prod LAN_CIDR)
 #   WG_DNS              default 192.168.2.23   (prod PIHOLE_DNS_VIP)
 #
-# REQUIRES: wireguard-tools (`wg`). `qrencode` is optional and only used for the QR code.
+# REQUIRES: `wg`, `qrencode`, `sops` — all three in the dev container.
 # =================================================================================================
 set -euo pipefail

 die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; }

 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
+PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"

 # --- defaults, each mirroring a value that is DECLARED elsewhere in this repo -------------------
 # These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to
@@ -51,7 +71,7 @@ PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
 : "${WG_CLIENT_CIDR:=10.90.0.0/24}"                 # clusters/base/cluster-vars.yaml
 : "${WG_LAN_CIDR:=192.168.2.0/24}"                  # clusters/prod LAN_CIDR
 : "${WG_DNS:=192.168.2.23}"                         # clusters/prod PIHOLE_DNS_VIP
-: "${WG_SERVER_KEY_FILE:=${HOME}/.config/wireguard-home/server.key}"
+: "${WG_SERVER_SOPS_FILE:=${REPO_ROOT}/platform/vpn-ingress/wg-server-key.sops.yaml}"
 # wan.<base-domain> is an UNPROXIED A record that platform/externaldns/ddns.yaml keeps pointed at
 # the current WAN IP (checked every 5 min). Use it, never a literal address: this house has a
 # DYNAMIC WAN IP, and a profile pinned to today's address stops working without warning and with no
@@ -83,14 +103,28 @@ printf '%s' "$NAME" | grep -Eq '^[a-z0-9][a-z0-9-]{0,30}$' \
 printf '%s' "$OCTET" | grep -Eq '^[0-9]+$' || die "octet must be a number: got '$OCTET'"
 [ "$OCTET" -ge 2 ] && [ "$OCTET" -le 254 ] || die "octet must be 2-254 (.1 is the server): got '$OCTET'"

-command -v wg >/dev/null || die "wireguard-tools not installed ('wg' not on PATH). Debian/Ubuntu: apt install wireguard-tools"
+command -v wg >/dev/null || die "'wg' not on PATH. Run this inside the dev container:
+  ./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh $NAME $OCTET'"

 # --- the server's public key --------------------------------------------------------------------
+# Order: an explicit pubkey, then a local private-key file, then the SOPS file in the tree. The
+# last is the normal path and needs no local copy of anything.
 if [ -z "${WG_SERVER_PUBKEY:-}" ]; then
-  [ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read server key '$WG_SERVER_KEY_FILE'.
-  Either point WG_SERVER_KEY_FILE at it, or set WG_SERVER_PUBKEY to the public half.
-  If the server key does not exist yet, see runbooks/vpn-remote-access.md step 1."
-  WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
+  if [ -n "${WG_SERVER_KEY_FILE:-}" ]; then
+    [ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read WG_SERVER_KEY_FILE '$WG_SERVER_KEY_FILE'"
+    WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
+  else
+    command -v sops >/dev/null || die "'sops' not on PATH. Run this inside the dev container (see --help)."
+    [ -r "$WG_SERVER_SOPS_FILE" ] || die "cannot read '$WG_SERVER_SOPS_FILE'"
+    # The decrypted private key goes straight into `wg pubkey` — never to a file, never to a tty.
+    # `wg pubkey` fails on the committed REPLACE-ME placeholder, which is the useful error here.
+    WG_SERVER_PUBKEY="$(sops -d --extract '["stringData"]["privatekey"]' "$WG_SERVER_SOPS_FILE" \
+                          | tr -d ' \r\n' | wg pubkey 2>/dev/null)" || WG_SERVER_PUBKEY=""
+    [ -n "$WG_SERVER_PUBKEY" ] || die "could not derive the server public key from
+  $WG_SERVER_SOPS_FILE
+  Either it still holds the REPLACE-ME placeholder (see runbooks/vpn-remote-access.md step 2),
+  or this shell cannot decrypt it (the age keyring is mounted at ~/.config/sops in the container)."
+  fi
 fi

 CLIENT_BASE="${WG_CLIENT_CIDR%/*}"; CLIENT_BASE="${CLIENT_BASE%.*}"
@@ -99,22 +133,41 @@ CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
 # --- refuse a collision, because the failure is otherwise invisible -----------------------------
 # Two devices sharing an address does not error anywhere: the server routes that address to
 # whichever peer handshook most recently, so BOTH devices appear to work intermittently.
-if [ -r "$PEERS_FILE" ] && grep -q "AllowedIPs *= *${CLIENT_ADDR}/32" "$PEERS_FILE"; then
-  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
+# COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the
+# COMMENTED-OUT EXAMPLE STANZA in the register, which made `.2` — the first address anyone would
+# ask for — permanently un-allocatable with a message claiming it was already taken. Strip
+# comments first: a real stanza's `AllowedIPs` line is never commented, while both the template
+# and every peer's `# <name> — <owner>` label are.
+# FAIL if the register is missing rather than skipping the check. The path was silently wrong once
+# (it still said peers.yaml after the file became peers.conf), and because the old guard was
+# `[ -r "$PEERS_FILE" ] && ...` the only symptom would have been duplicate detection quietly
+# never firing again — the exact silent-skip shape this repo treats as a defect.
+[ -r "$PEERS_FILE" ] || die "cannot read the peer register '$PEERS_FILE'.
+  Duplicate-address detection cannot run, so this script refuses to issue a profile."
+if grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
+  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.conf. Pick another octet."
 fi
-if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
-  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2
+if grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
+  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.conf.\n' "$NAME" >&2
 fi

 # --- output location: 0700, and never inside the repo -------------------------------------------
-if [ -z "$OUT_DIR" ]; then
-  OUT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/wg-peer-XXXXXX")"
-else
-  mkdir -p "$OUT_DIR"
-fi
+[ -n "$OUT_DIR" ] || OUT_DIR="${REPO_ROOT}/.cache/wg-peers"
+mkdir -p "$OUT_DIR"
 chmod 700 "$OUT_DIR"
-case "$(cd "$OUT_DIR" && pwd)" in
-  "$REPO_ROOT"|"$REPO_ROOT"/*) die "refusing to write a private key inside the git checkout ($OUT_DIR)" ;;
+OUT_ABS="$(cd "$OUT_DIR" && pwd)"
+# A profile holds a private key, so the destination must not be committable. Inside the repo that
+# means git must IGNORE it — test the real property rather than a proxy, because the repo is the
+# container's only persistent read-write mount and so the only sane place to write. Outside the
+# repo, anywhere goes.
+case "$OUT_ABS" in
+  "$REPO_ROOT"|"$REPO_ROOT"/*)
+    if ! git -C "$REPO_ROOT" check-ignore -q "$OUT_ABS/probe.conf" 2>/dev/null; then
+      die "refusing to write a private key to '$OUT_ABS': it is inside the git checkout and NOT
+  gitignored, so the profile would be committable. Use the default (.cache/wg-peers) or add an
+  ignore rule first. Verify with: git check-ignore -v <path>"
+    fi
+    ;;
 esac
 PROFILE="${OUT_DIR}/${NAME}.conf"

@@ -162,7 +215,7 @@ cat <<EOF
  Profile     : ${PROFILE}   (0600; contains the private key)

  ------------------------------------------------------------------------------------------
-  STEP 1 — add this to platform/vpn-ingress/peers.yaml, under data.peers.conf, then PR it:
+  STEP 1 — append this to platform/vpn-ingress/peers.conf, then PR it:

    # ${NAME} — <owner>, enrolled $(date -u +%Y-%m-%d)
    [Peer]
@@ -181,7 +234,7 @@ cat <<EOF
  STEP 3 — delete the profile from this machine once the device has it:
    shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE}

-  The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet
+  The tunnel will not work until the peers.conf PR is MERGED and Flux has rolled the DaemonSet
  (~1 min). Until then the device handshakes and gets silence, exactly like an unknown key.
  ------------------------------------------------------------------------------------------
 EOF
@@ -190,6 +243,8 @@ if command -v qrencode >/dev/null; then
  printf '  QR code (this IS the private key — your terminal scrollback now holds a credential):\n\n'
  qrencode -t ansiutf8 < "$PROFILE"
 else
-  printf '  qrencode not installed, so no QR code was printed (Debian/Ubuntu: apt install qrencode).\n'
-  printf '  Transfer %s to the device by another route instead.\n' "$PROFILE"
+  # Should not happen in the dev container: qrencode is baked into the image. Kept as a real
+  # fallback for a host run, rather than failing after the profile has already been written.
+  printf '  qrencode not on PATH, so no QR code was printed. It is in the dev container —\n'
+  printf '  rerun there, or transfer %s to the device by another route.\n' "$PROFILE"
 fi
@@ -0,0 +1,112 @@
+---
+# =================================================================================================
+# coredns HA floor — hold in-cluster DNS at >= 2 replicas, which k3s does not.
+# =================================================================================================
+# MEASURED on prod 2026-10-18: `kube-system/coredns` ran with **replicas: 1**. In-cluster DNS is a
+# single pod on a single node, so **draining or rebooting that one node is a cluster-wide
+# in-cluster DNS outage** for as long as the pod takes to reschedule and become ready — every pod
+# that looks up a Service, every CNPG replica finding its primary, every operator talking to the
+# API by name. It is the single cheapest availability defect in the cluster, and it was invisible
+# because `1/1 Ready` looks perfectly healthy.
+#
+# This matters more now than it did: the reboot automation (kured + unattended-upgrades) drains and
+# reboots one node at a time on a schedule, which turns a latent single point of failure into a
+# recurring one, unattended, in the middle of the night.
+#
+# -------------------------------------------------------------------------------------------------
+# WHY AN HPA, AND NOT JUST `replicas: 2` — this is the whole decision
+# -------------------------------------------------------------------------------------------------
+# **k3s owns the coredns Deployment and we cannot durably edit it.** It is applied by k3s's own
+# deploy controller from a bundled manifest, not by Flux:
+#
+#   $ kubectl -n kube-system get addon coredns
+#   NAME      SOURCE                                        CHECKSUM
+#   coredns   /data/k3s/server/manifests/coredns.yaml        8cd91595...
+#   $ kubectl -n kube-system get deploy coredns -o jsonpath='{.metadata.managedFields[*].manager}'
+#   deploy@k3s-prod-1 k3s
+#
+# and the Deployment carries an `objectset.rio.cattle.io/hash` label — a wrangler objectset. k3s
+# RE-STAGES its bundled manifests on startup, so a `replicas: 2` written into that file (or a
+# `kubectl scale`) is reverted the next time k3s restarts or is upgraded. MEASURED the same day: a
+# hand `kubectl scale` to 2 was NOT actively reverted (it held, and the AddOn checksum did not
+# change) — the deploy controller re-applies on manifest change and on restart, not on a drift
+# timer. So the hand fix survives until exactly the event we are protecting against: a reboot.
+#
+# The two durable alternatives were weighed and rejected:
+#   * **`coredns.yaml.skip` + our own templated copy** (the pattern roles/kube_vip already uses for
+#     a manifest in server/manifests). Fully deterministic and never flaps — but it takes OWNERSHIP
+#     of coredns away from k3s permanently, so k3s upgrades would stop updating it and the version
+#     would have to be refreshed by hand forever. Too high a price for a replica count.
+#   * **A Flux-managed patch of the Deployment itself.** Rejected outright: Flux and k3s's deploy
+#     controller would both own one object and fight continuously. That is the shape of the
+#     `default-deny` NetworkPolicy drift loop this cluster already suffered (Kyverno generate + Flux
+#     both owning one resource, ~2500 events per 10 minutes).
+#
+# An HPA avoids both. It is a SEPARATE object that k3s does not own and will never reconcile away,
+# and it drives the Deployment's `/scale` subresource rather than its spec — so there is no
+# shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA observes it under `minReplicas`
+# and scales back to 2 within one reconcile (~15s by default).
+#
+# THE COST, STATED PLAINLY: there is a brief window — seconds, after a k3s restart — where coredns
+# is back at 1 replica until the HPA notices. That is strictly better than the status quo, where it
+# is at 1 replica permanently, but it is not zero. Accepting that window is the trade being made
+# here; the `.skip` option above is what to reach for if it ever proves unacceptable.
+#
+# -------------------------------------------------------------------------------------------------
+# THIS IS A FLOOR-HOLDER, NOT A REAL AUTOSCALER
+# -------------------------------------------------------------------------------------------------
+# `minReplicas: 2` is the entire point. The CPU metric exists because an autoscaling/v2 HPA needs a
+# metric to be a valid object, not because coredns is expected to scale on load: measured via
+# metrics-server, the two pods use **5m and 4m** against a **100m** request — ~5% utilisation, so
+# the 80% target will never be approached and the HPA will sit at `minReplicas` indefinitely.
+#
+# The CPU request is what makes this legal: coredns ships `requests: {cpu: 100m, memory: 70Mi}`
+# (measured), and a Resource/Utilization metric is meaningless without a request. metrics-server is
+# installed as a k3s addon and verified serving (`kubectl top pod -l k8s-app=kube-dns` returns
+# figures), so the metric resolves rather than leaving the HPA with `ScalingActive=False`.
+#
+# `maxReplicas: 3` = the node count. coredns' pod anti-affinity prefers one pod per node, so a
+# fourth replica would add no failure-domain coverage; the ceiling exists so a metrics glitch can
+# never balloon DNS pods across the cluster.
+apiVersion: autoscaling/v2
+kind: HorizontalPodAutoscaler
+metadata:
+  name: coredns
+  # kube-system, because that is where k3s puts coredns. This is the only object this component
+  # places outside a namespace it owns, and it is additive: it creates nothing in kube-system and
+  # mutates only the coredns Deployment's scale subresource.
+  namespace: kube-system
+  labels:
+    app.kubernetes.io/name: coredns
+spec:
+  scaleTargetRef:
+    apiVersion: apps/v1
+    kind: Deployment
+    name: coredns
+  minReplicas: 2
+  maxReplicas: 3
+  metrics:
+    - type: Resource
+      resource:
+        name: cpu
+        target:
+          type: Utilization
+          # Deliberately high. See "floor-holder" above: this must not scale on normal DNS load.
+          averageUtilization: 80
+  behavior:
+    scaleDown:
+      # 10 minutes. DNS is the worst thing in the cluster to flap: a scale-down races in-flight
+      # queries and every client's negative cache. The HPA should be reluctant in this direction.
+      stabilizationWindowSeconds: 600
+      policies:
+        - type: Pods
+          value: 1
+          periodSeconds: 600
+    scaleUp:
+      # Prompt in the direction that restores redundancy — this is the path taken after a k3s
+      # restart has re-staged `replicas: 1`.
+      stabilizationWindowSeconds: 0
+      policies:
+        - type: Pods
+          value: 2
+          periodSeconds: 60
@@ -7,3 +7,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
  - coredns-custom.yaml
+  # HA floor for the coredns Deployment itself — a separate concern from the split-horizon server
+  # block above, and deliberately an HPA rather than a replica edit: k3s owns that Deployment via
+  # its own deploy controller and re-stages `replicas: 1` on restart. hpa.yaml has the reasoning.
+  - hpa.yaml
@@ -199,7 +199,7 @@ data:
          "id": 15,
          "type": "timeseries",
          "title": "CAUSE 4 — NFS RPC retransmission ratio, per node",
-          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.102, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
+          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.30, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
          "gridPos": {"h": 8, "w": 12, "x": 12, "y": 39},
          "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
            {"color": "green", "value": null}, {"color": "red", "value": 0.01}
@@ -63,6 +63,75 @@ spec:
      # gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
      skip-provider-button: "true"
      scope: "openid email profile"
+    # ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ──────────
+    # MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at
+    # startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so
+    # the Gateway answers 503 and the proxy dies:
+    #
+    #   [provider.go:55] Performing OIDC Discovery...
+    #   [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC
+    #                configuration: unexpected status "503": no healthy upstream
+    #
+    # It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was
+    # still down minutes later and only a manual pod delete cleared it. There is no retry flag; the
+    # process simply does not survive a cold start. A readinessProbe cannot help (the container is
+    # already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+    #
+    # `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in
+    # values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init
+    # container for its built-in waitForRedis gate, so this adds no new base image to the platform.
+    # A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole
+    # point.
+    #
+    # BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst
+    # case is then exactly today's behaviour, never something new. An unbounded wait would trade a
+    # CrashLoop for a pod wedged in Init forever, which is not an improvement.
+    #
+    # SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is
+    # rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which
+    # CONSUMES braced expansions -- so a braced shell variable renders to the empty string and
+    # silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars
+    # knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud
+    # OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.)
+    extraInitContainers:
+      - name: wait-for-idp
+        image: docker.io/library/alpine:3.22
+        imagePullPolicy: IfNotPresent
+        command: ["/bin/sh", "-c"]
+        args:
+          - |
+            set -u
+            url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
+            i=0
+            while [ "$i" -lt 60 ]; do
+              # busybox wget validates against alpine's ca-certificates-bundle, so a real TLS
+              # failure is a non-zero exit here rather than a false pass.
+              if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then
+                echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy"
+                exit 0
+              fi
+              i=$(( i + 1 ))
+              sleep 5
+            done
+            echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2
+            echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2
+            exit 0
+        securityContext:
+          allowPrivilegeEscalation: false
+          runAsNonRoot: true
+          runAsUser: 65534
+          runAsGroup: 65534
+          readOnlyRootFilesystem: true
+          capabilities:
+            drop:
+              - ALL
+        resources:
+          requests:
+            cpu: 10m
+            memory: 16Mi
+          limits:
+            memory: 32Mi
+
    # Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
    metrics:
      enabled: true
@@ -22,6 +22,25 @@ requires.

 ---

+## Why the agent authenticates to Gitea with a local account and a PAT
+
+Not Keycloak, and that is a decision rather than an omission (operator, 2026-10-06): the
+agents only ever talk to Gitea, so a local Gitea account with a scoped personal access token
+is the whole requirement, and it is drastically simpler than federating a machine identity.
+
+It is also what Gitea can actually do. Keycloak service accounts were considered and are not
+available for this: Gitea's API accepts basic auth, PATs, SSH signatures and **its own**
+OAuth2 bearer tokens, and as an OAuth2 provider Gitea implements **only the Authorization
+Code Grant** — there is no client-credentials grant, and nothing in Gitea validates a token
+issued by a third-party OIDC provider. So a Keycloak service-account token cannot
+authenticate to Gitea's API or to git-over-HTTP at all. Keycloak remains what it is for human
+access: the browser login source registered on Gitea (`catalog/gitea.yaml`).
+
+What still matters, and is enforced by the hand-off rather than by Keycloak: **the account
+must not be an administrator** — `infra/gitea/approval-policy.json` authorises
+`/approve-override` on live repository permission alone, so an admin-permission bot could
+waive the sensitive-path human review on its own pull request.
+
 ## It does nothing at all until an operator does two things

 Both are deliberate, and both fail **visibly** rather than silently:
@@ -31,6 +50,36 @@ Both are deliberate, and both fail **visibly** rather than silently:
   `infra/agent/build-image.sh` (which verifies it and refuses to push), push it as the
   operator, then update **both** places with the digest `podman push --digestfile` returned,
   in one pull request. Until then: `ImagePullBackOff`.
+
+   **Two things have to be true before any digest will pull, and neither is declared in this
+   repository.** Both were measured after PR #57 merged — this service is the first workload
+   in the cluster ever to reference the in-cluster registry, so nothing had exercised the
+   path. Details and the exact commands:
+   [`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`](../../decisions/2026-10-05-pm-agent-registry-pull-prereq.md).
+
+   - **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is
+     the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a
+     *node* process, so it resolves through the node's public upstreams
+     (`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public
+     answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN
+     IP — which this router does not hairpin; it answers :443 with its own placeholder
+     certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely
+     `x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
+     and the digest were never implicated. The fix is the pin this repository already uses
+     for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
+     in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the
+     inventory, never typed on a node. Apply it scoped to that one task:
+     `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
+     --tags hosts`.
+   - **The pull is still unauthenticated** once the node reaches the right endpoint. An
+     unauthenticated client gets nothing from this Gitea's registry, and there is no
+     `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
+     (local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea
+     account whose token carries `read:package`, referenced from the Deployment and the
+     committed worker Job template — not built yet, because minting that token is yours.
+
+   The same two apply to `conformance-runner`: same registry path, same push-then-pin model,
+   never pulled either.
 2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the
   repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off —
   which Gitea account to create, which scopes, and why it must *not* be an administrator —
@@ -96,7 +145,7 @@ Job from a committed template. **The model's output is data, never a command.**
 | **cannot** | merge anything — the gate is the approver, and the service makes no merge call |
 | **cannot** | reach the Kubernetes API: no ServiceAccount token is mounted, the `pm-agent-worker` identity is bound to nothing, and the egress policy does not include the API server. `kubectl` is on its `PATH` (the gate's toolchain needs it) and is inert |
 | **cannot** | reach anything except this Gitea and tcp/443 — no package registry, no GitHub |
-| **cannot** | clone from or push to any host but this Gitea: `pm-worker` validates `PM_GITEA_URL` against an allowlist baked into the image |
+| **cannot** | clone from or push to anywhere but `gitea.gitea.svc.cluster.local:3000`: `pm-worker` validates `PM_GITEA_URL` against a one-entry allowlist baked into the image. In-cluster consumers reach a service by its cluster-local name — a different path by design from how a client outside the cluster reaches it — so the external hostname is not an accepted value |
 | **cannot** | authenticate a `git push` **during the model session**: `GIT_ASKPASS` is unset for its duration, so the restriction is structural rather than a request in a prompt |
 | **cannot** | be created in any shape but the committed one — see the admission policy |

@@ -51,6 +51,15 @@
 # (2) MINT THE GITEA TOKEN as that user (log in as `pm-agent`, or use the admin API to
 #     create a token for it): Settings -> Applications -> Generate New Token.
 #
+#     A LOCAL GITEA ACCOUNT WITH A PAT IS THE DECIDED MODEL (operator, 2026-10-06), not
+#     Keycloak. The agents only ever talk to Gitea, so this is the whole requirement and it
+#     is far simpler than federating a machine identity. It is also the only thing that
+#     works: Gitea's API accepts basic auth, PATs, SSH signatures and its OWN OAuth2 bearer
+#     tokens, and as a provider Gitea implements only the Authorization Code Grant — no
+#     client-credentials grant, and nothing that validates a third-party OIDC token. A
+#     Keycloak service-account token therefore cannot authenticate to Gitea's API or to
+#     git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that.
+#
 #       Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT
 #       `write:admin`, NOT `write:organization`, NOT `write:user`.
 #
@@ -13,8 +13,8 @@ egress out the house WAN.
 | Transport | WireGuard, **udp/51820**, kernel implementation |
 | Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
 | Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
-| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
-| The one secret | the server's private key, Secret `wg-server-key`, **created out of band** |
+| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
+| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
 | Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
 | Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
 | Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
@@ -129,12 +129,18 @@ translations win here.

 | half | where | why |
 |---|---|---|
-| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
-| the server's **private** key | Secret `wg-server-key`, **out of band** | the only real secret; an agent must not mint a long-lived private key (CLAUDE.md §6) |
+| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
+| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |

 The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
 so the assembled config holding the private key never touches any node's disk and dies with the pod.

+Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with
+everything else. An out-of-band `kubectl create secret` — the first shape this component shipped in,
+corrected after review — would have put the one credential every enrolled device depends on *outside*
+the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this
+key's public half.
+
 Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
 new line granting a new device access to the whole LAN.

@@ -160,21 +166,40 @@ NetworkPolicy applies.
 ## Operating it

 * **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
-  stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
-* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
+  stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
+* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
+  stops working as soon as the pods roll (the generator's content hash guarantees they do).
 * **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
 * **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
  (`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
 * **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.

+## Measured on prod, 2026-10-06 — what is now known rather than assumed
+
+The first real reconcile settled four things this component's design rested on. Three went the
+right way; one was a defect in my own manifest.
+
+| question | answer |
+|---|---|
+| Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. |
+| Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. |
+| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
+| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
+
 ## Prerequisites outside this directory

 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
   (`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
   `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
   CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
-2. **The `wg-server-key` Secret.** Runbook step 1. Until it exists the DaemonSet cannot start —
-   the correct fail-closed state for an unenrolled VPN.
+2. **The server private key.** `wg-server-key.sops.yaml` is committed, encrypted, with a
+   `REPLACE-ME` placeholder, and both overlays already carry its decryption patch — so the wiring
+   is complete and the only outstanding act is replacing the value (runbook step 2). Until then the
+   DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the
+   value with `wg pubkey`, which rejects the placeholder, an empty value, a public key pasted by
+   mistake, **and** a Secret that arrived as undecrypted `ENC[...]` ciphertext. That last case is
+   the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves
+   every Flux condition green.
 3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
   and the half CI can never check.

@@ -187,7 +212,7 @@ NetworkPolicy applies.
  covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
  kube-prometheus-stack. **This is the top follow-up.**
 * **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
-  handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
+  handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
  reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
  `wg-server-key` Secret.
 * **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
@@ -37,6 +37,10 @@
 #     /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
 #     is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
 #     required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change).
+#     COROLLARY LEARNED THE HARD WAY (prod, 2026-10-06): this pod must not merely avoid NEEDING
+#     the sysctl, it must not ATTEMPT it. hostNetwork implies /proc/sys is mounted read-only, so
+#     `sysctl -w` fails with EPERM regardless of capabilities, and wg-quick treats a failed PostUp
+#     as fatal and deletes the interface. Verify the value; never write it.
 #   * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost
 #     this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all.
 #   * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a
@@ -104,7 +108,7 @@ spec:
      dnsPolicy: ClusterFirstWithHostNet
      automountServiceAccountToken: false
      # ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
-      # The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
+      # The two halves are deliberately stored apart (peers.conf explains why) and are only ever
      # joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
      # key is therefore never written to any node's disk, never committed, and never logged.
      initContainers:
@@ -136,9 +140,51 @@ spec:
              # The key arrives as a Secret file; strip any trailing newline the operator's shell
              # added, or wg-quick rejects the config with "Key is not the correct length or format".
              key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey)
-              if [ -z "$key" ]; then
-                echo "FATAL: /run/wg-server-key/privatekey is empty." >&2
-                echo "       See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2
+              # VALIDATE, don't just check for empty. The Secret ships committed with a REPLACE-ME
+              # placeholder so the GitOps wiring is complete before the key exists, so the common
+              # failure is a value that is present and WRONG — a placeholder, a public key pasted by
+              # mistake, or a SOPS document that reached the cluster undecrypted (ENC[...], which
+              # happens if an overlay is missing its `vpn-ingress` decryption patch). `wg pubkey`
+              # accepts a value only if it is a real 32-byte curve25519 private key, so it rejects
+              # all four. Without this the bad value would be written into wg0.conf and the pod would
+              # fail later and less legibly, or come up with a server nobody can handshake with.
+              if ! printf '%s' "$key" | wg pubkey >/dev/null 2>&1; then
+                echo "FATAL: the server private key in Secret 'wg-server-key' is not a valid" >&2
+                echo "       WireGuard private key (wg pubkey rejected it)." >&2
+                if [ -z "$key" ]; then
+                  echo "       It is EMPTY." >&2
+                elif [ "$key" = "REPLACE-ME" ]; then
+                  echo "       It is still the committed REPLACE-ME PLACEHOLDER — the key has not" >&2
+                  echo "       been generated yet. This is the expected state on a fresh install." >&2
+                else
+                  # `case`, NOT a dollar-brace prefix-strip. This manifest is rendered through
+                  # envsubst, which consumes every dollar-brace expansion it sees, so a shell
+                  # parameter expansion here renders to the empty string and silently disables this
+                  # very branch. `case` uses no braces, so there is nothing to escape.
+                  # (ci: flux-postbuild-substitution caught both the original expansion AND the
+                  # first version of this comment, which spelled the hazard out literally.)
+                  case "$key" in
+                    ENC\[*)
+                      echo "       It is still SOPS CIPHERTEXT (starts with ENC[). The Secret" >&2
+                      echo "       reached the cluster undecrypted — check that this environment's" >&2
+                      echo "       overlay carries the 'vpn-ingress' decryption patch." >&2
+                      ;;
+                  esac
+                fi
+                echo "       Fix: runbooks/vpn-remote-access.md step 2." >&2
+                exit 1
+              fi
+
+              # The pod shares the HOST netns, so this reads the node's real setting. Reading
+              # /proc/sys is permitted even though writing is not (see the PostUp note below).
+              # Fail closed: a tunnel that comes up but cannot forward would look healthy to every
+              # probe here and simply drop every client's traffic.
+              if ! grep -qx 1 /proc/sys/net/ipv4/ip_forward; then
+                echo "FATAL: net.ipv4.ip_forward is 0 on this node, so forwarded VPN traffic would" >&2
+                echo "       be dropped. This pod CANNOT set it: with hostNetwork the runtime mounts" >&2
+                echo "       /proc/sys read-only, because it is the host's global setting." >&2
+                echo "       Fix on the NODE: run the node_baseline role (it writes" >&2
+                echo "       /etc/sysctl.d/99-k3s-cilium.conf) — runbooks/vpn-remote-access.md step 1." >&2
                exit 1
              fi

@@ -155,9 +201,20 @@ spec:
                echo 'MTU = 1420'
                printf 'PrivateKey = %s\n' "$key"
                # ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ──
-                # ip_forward is already 1 from node_baseline; re-asserted so this component does not
-                # silently depend on another role's sysctl file surviving.
-                echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1'
+                # NO `sysctl -w net.ipv4.ip_forward=1` HERE. It is IMPOSSIBLE from this pod and it was
+                # the only thing stopping the tunnel coming up — MEASURED on prod 2026-10-06:
+                #     [#] sysctl -q -w net.ipv4.ip_forward=1
+                #     sysctl: permission denied on key "net.ipv4.ip_forward"
+                #     [#] ip link delete dev wg0
+                # and wg-quick then tore the whole tunnel down. The cause is not a missing
+                # capability: with `hostNetwork: true` the runtime mounts /proc/sys READ-ONLY in the
+                # container (confirmed in /proc/mounts: `proc /proc/sys proc ro,...`), precisely
+                # because those sysctls are the HOST's globals and a container must not rewrite
+                # them. CAP_NET_ADMIN does not and cannot lift a read-only mount.
+                # It was never needed anyway: node_baseline writes net.ipv4.ip_forward=1 to
+                # /etc/sysctl.d/99-k3s-cilium.conf, and reading it from this pod returns 1. The
+                # init container now VERIFIES that (a read, which /proc/sys does allow) and fails
+                # closed with a pointed message, which is what the write was pretending to achieve.
                # SNAT client traffic to the node's address. REQUIRED: without it a LAN host would
                # reply to 10.90.0.x, find no route, hand the packet to the router, and the router
                # would drop it — so every connection would hang rather than fail. Matching on
@@ -217,8 +274,10 @@ spec:
            # `kubectl get pod -o wide` shows what this listener owns.
            - { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP }
          securityContext:
-            # NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and
-            # write the two netfilter rules above. Everything else is dropped, and the container is
+            # NET_ADMIN is the whole privilege budget: create the wg0 link and write the netfilter
+            # rules above. (It does NOT let the pod set a host sysctl — /proc/sys is read-only under
+            # hostNetwork — which is why nothing here tries.) Everything else is dropped, and the
+            # container is
            # NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule.
            # SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node
            # (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container
@@ -232,6 +291,16 @@ spec:
                - NET_ADMIN
            # readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and
            # /var during startup. Same deferral as crafty/headlamp.
+            #
+            # EXPECTED, HARMLESS STARTUP NOISE — do not "fix" it. Because capabilities are dropped
+            # to NET_ADMIN only, the image's s6 init cannot chown /config (no CAP_CHOWN) and logs a
+            # run of `chown: ... Operation not permitted` plus "Permissions could not be set. This
+            # is probably because your volume mounts are remote or read-only" and a scary "we will
+            # not provide support for it". It is wrong about the cause and it does not matter:
+            # /config is a writable tmpfs emptyDir, everything here runs as root (uid 0) anyway, and
+            # wg-quick proceeds normally — verified on prod 2026-10-06, where the tunnel's only
+            # failure was the sysctl above, well past these warnings. Granting CAP_CHOWN to silence
+            # log noise would widen the privilege set of an internet-facing pod for nothing.
          resources:
            requests:
              cpu: 25m
@@ -272,14 +341,22 @@ spec:
            # during startup rather than at a point anyone would connect to the volume size. Still
            # counted against the container's 128Mi limit, and still far more than wg0.conf needs.
            sizeLimit: 8Mi
-        # The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by
-        # name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this
-        # tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key
-        # that passed through a session transcript would have to be treated as compromised.
-        # DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace
-        # and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy
-        # in the operator's password manager as well — restoring it is what makes every already
-        # enrolled client work again without re-enrolling.
+        # The ONLY secret in this component. SOPS-encrypted IN THIS TREE (R-SEC-01, spec docs/07) as
+        # wg-server-key.sops.yaml, delivered by Flux, referenced here by name only (R-IDP-04).
+        #
+        # It ships committed with a REPLACE-ME placeholder: an agent must not mint a long-lived
+        # private key (CLAUDE.md §6), so the operator generates it and `sops set`s it in
+        # (runbooks/vpn-remote-access.md step 2). Shipping the encrypted placeholder rather than no
+        # file at all is what lets the resource line and both overlay decryption patches be present
+        # and GREEN from the first commit — the alternative, a commented-out resource to uncomment
+        # later, is exactly the "file in git with no generator entry: reviewed, merged, apparently
+        # done, still broken" shape this repo keeps getting bitten by.
+        #
+        # DR: in git means a cluster rebuild restores it with everything else — the property an
+        # out-of-band `kubectl create secret` would NOT have had, and losing this key means
+        # re-enrolling every device because each client profile pins its public half. Velero also
+        # captures this namespace (its `daily` schedule excludes only `monitoring` and `unifi`).
+        # Keep a password-manager copy too: that is the one that survives losing cluster AND backups.
        - name: wg-server-key
          secret:
            secretName: wg-server-key
@@ -4,27 +4,66 @@
 # same placement as crafty/headlamp/homepage/runbooks, for the same reason: a leaf whose pod can be
 # legitimately unhealthy must never sit in the gating `platform` tier, where one component's
 # unhealth stalls platform-config and every apps wave behind it (DECISIONS-LOG: "Headlamp gating
-# stall"). That rationale is unusually concrete here: until the operator creates the `wg-server-key`
-# Secret (runbooks/vpn-remote-access.md step 1) this DaemonSet CANNOT start, so putting it in the
-# gating tier would have taken the whole platform down on merge.
+# stall"). That rationale is unusually concrete here: until the operator replaces the `REPLACE-ME`
+# placeholder in wg-server-key.sops.yaml (runbooks/vpn-remote-access.md step 2) this DaemonSet
+# CANNOT start, so putting it in the gating tier would have taken the whole platform down on merge.
 #
 # NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
+# REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated
+# ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's
+# nameReference transformer will not rewrite a reference across a namespace boundary — so without
+# this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the
+# pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1).
+# Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own
+# content generator.
+namespace: vpn-ingress
 resources:
  - namespace.yaml
-  # The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
-  - peers.yaml
  # The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
  - daemonset.yaml
  # Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
  # says so at length rather than implying a protection it cannot provide.
-  - networkpolicy.yaml
+  - networkpolicy.yaml  # The server's WireGuard private key — the ONE secret here. SOPS-encrypted (R-SEC-01, spec
+  # docs/07), decrypted per-env by the `vpn-ingress` Kustomization, referenced by name only from
+  # daemonset.yaml (R-IDP-04). Its plaintext `.example` sibling carries the operator hand-off --
+  # deliberately NOT named here: conformance/tests/l0-static/no-plaintext-secrets fails any
+  # kustomization that mentions an unencrypted template, because naming one is how a plaintext
+  # Secret gets applied by accident. (It matches comments too. That is correct, not overzealous.)
+  #
+  # SHIPS WITH A `REPLACE-ME` PLACEHOLDER, already encrypted. An agent must not
+  # mint a long-lived private key (CLAUDE.md §6), and the init container validates the value with
+  # `wg pubkey`, so the unfilled state is a loud CrashLoop naming this file, never a server that
+  # quietly cannot handshake.
+  - wg-server-key.sops.yaml
 #
-# DELIBERATELY ABSENT: a Secret. The server's private key (`wg-server-key`) is created out of band
-# by the operator and referenced BY NAME from daemonset.yaml. There is no `*.sops.yaml` here, so
-# this component needs NO SOPS decryption stanza in clusters/dev or clusters/prod — unlike
-# crafty/headlamp/velero. If that key is ever migrated into SOPS, BOTH halves must land together:
-# the file here, and a `decryption:` patch naming the `vpn-ingress` Kustomization in each overlay.
-# Without the patch, kustomize hands Flux the raw ciphertext and the pod mounts a key full of
-# ENC[...] — which applies cleanly and fails only at handshake time.
+# THIS COMPONENT CARRIES AN ENCRYPTED SECRET, so BOTH overlays must patch the `vpn-ingress`
+# Kustomization with a `decryption:` stanza — clusters/dev and clusters/prod both do, added by the
+# same change as this file. That patch is as load-bearing as the file itself: without it kustomize
+# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
+# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
+# that into a named error rather than a silently non-working tunnel.
+
+# ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ─────────────────────────────────────────
+# peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a
+# GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic:
+# kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's
+# volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE
+# PODS automatically.
+#
+# A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux
+# updates it in place, the pod template is byte-identical, no rollout is triggered — and because
+# the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps
+# the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the
+# ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having
+# never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this.
+#
+# DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism.
+configMapGenerator:
+  - name: wg-peers
+    files:
+      - peers.conf
+    options:
+      labels:
+        app.kubernetes.io/name: vpn-ingress
@@ -19,7 +19,7 @@
 # protects this port, and it would be enforcing nothing. So there isn't one.
 #
 # WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
-#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
+#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.conf gets
 #      NO REPLY — see README "Why this is safe to expose". This is the real control.
 #   2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
 #   3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement
@@ -1,4 +1,16 @@
---
+# THIS FILE IS THE MOUNTED CONTENT. It is turned into the `wg-peers` ConfigMap by
+# `configMapGenerator` in kustomization.yaml, whose generated name carries a CONTENT HASH — so
+# editing this file changes the ConfigMap name, which changes the DaemonSet's pod template, which
+# ROLLS THE PODS. That is the whole reason it is a generator and not an inline ConfigMap manifest:
+# a fixed-name ConfigMap updates in place, the pod template never changes, no rollout happens, and
+# the init container's wg0.conf (built once at pod start, in tmpfs) keeps the OLD peer list
+# forever. Measured 2026-10-06: a merged peer sat in the ConfigMap while `wg show` listed none and
+# the DaemonSet reported generation=1 — it had never rolled. Same mechanism and same fix as
+# runbooks/kustomization.yaml.
+#
+# Every line here is WireGuard config. `#` is a legal comment in it, so the documentation below
+# travels inside the file that is actually mounted, rather than in a wrapper that gets stripped.
+#
 # ============================================================================================
 # THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
 # ============================================================================================
@@ -41,33 +53,40 @@
 # the Noise handshake and would be a genuine secret — which would force this register into SOPS
 # and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
 # belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: wg-peers
-  namespace: vpn-ingress
-  labels:
-    app.kubernetes.io/name: vpn-ingress
-data:
-  # Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
-  #
-  # *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
-  # An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
-  # authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
-  # DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
-  # The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
-  # (`wg show` reports an interface with no peers).
-  #
-  # COPY THIS TEMPLATE, uncomment, one block per device:
-  #
-  #   # pixel-8 — Keith's phone, enrolled 2026-10-05
-  #   [Peer]
-  #   PublicKey = <the public key new-peer.sh printed>
-  #   AllowedIPs = 10.90.0.2/32
-  #
-  # Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
-  # tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
-  # it an incident responder cannot tell which stanza to delete.
-  peers.conf: |
-    # No peers enrolled yet. See the template in this file's comments and
-    # ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
+#
+# name: wg-peers
+# namespace: vpn-ingress
+# labels:
+# app.kubernetes.io/name: vpn-ingress
+# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
+#
+# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
+# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
+# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
+# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
+# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
+# (`wg show` reports an interface with no peers).
+#
+# COPY THIS TEMPLATE, uncomment, one block per device:
+#
+#   # <device-name> — <owner>, enrolled <YYYY-MM-DD>
+#   [Peer]
+#   PublicKey = <the public key new-peer.sh printed>
+#   AllowedIPs = 10.90.0.<octet>/32
+#
+# The placeholders above are deliberately NOT valid values. An earlier version of this template
+# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
+# the whole file — matched it, so `.2` could never be allocated and the script insisted it was
+# already taken. The check now ignores comments; keeping the example unmatchable too means a
+# future grep cannot be fooled either.
+#
+# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
+# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
+# it an incident responder cannot tell which stanza to delete.
+#
+# No peers enrolled yet. See the template in this file's comments and
+# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
+# keith-phone — <owner>, enrolled 2026-10-06
+[Peer]
+PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
+AllowedIPs = 10.90.0.2/32
@@ -0,0 +1,34 @@
+# WireGuard SERVER private key for platform/vpn-ingress — SOPS-ENCRYPTED (R-SEC-01, spec docs/07).
+# Keys stay cleartext for reviewable diffs; only `stringData` values are encrypted (.sops.yaml
+# `encrypted_regex`). Plaintext template + operator hand-off: wg-server-key.sops.yaml.example.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: wg-server-key
+    namespace: vpn-ingress
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:fUoPkfWvasisdTtwdhjeOVOlmdQkTl6Bp9YxYyji8g6k+ClATds3BnHJWOj2iplaGpT2XaNakq1+D4YyOi7QbFvNJui+e4qRpqcY8hWnazvZIHJYxjxfAfzW,iv:xtKd6oMZQumAQJAsNLHqJB+o0wdQJ+vL1kI1mc9jL7w=,tag:aImqLgbSmhsRI36/t5kpyg==,type:comment]
+    #ENC[AES256_GCM,data:npDsCW6rQleolIpur5gFkX2jSRrIQzBlGaIPs2Schv0YU3q8AanG39z4QqT3ASDmskV5YuH29Bo0ujEAgBf52YIjQCp58PDDETofrVKOJzhHMa7bV+HCxUFE,iv:7E7thyFBleyqZpqJRGApKvIq3ubCFOkshF/mxim3YPc=,tag:joIrhMe2I6kUnx1ouOPdZw==,type:comment]
+    #ENC[AES256_GCM,data:wHUFx09FgP/2+LTyw61EFbKA5UaqfjU7CeGk6Xc5av2y4r4AvQJRefDM5EGlahKecDFEFsf7fSQRFRHtUJqrQTtw+6tcj3ubSCPW4iJvhyJJv0xdb/0puRmcgj+2NfgF,iv:xgDOSDLqMQ1wcI9vrkKqfNMgIDFEusFnnO7Rzq/aOa4=,tag:m1m/fgjkBsG91c+3daqOpA==,type:comment]
+    privatekey: ENC[AES256_GCM,data:ML8CPNm4JwhQHUTUUxZO/I+9X15BCIGqOb1Uw9EmX5RTAsxQ9N97YvkYlbY=,iv:5ZUsFdMMmeGsy4fNbk6qbJkzweMVWXZ5g6E7NciAGUg=,tag:pAHv3ieLjoEyFbqbWXsVgg==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMays2OXY0VkpRc2FQQmhU
+            Q0owM3N0VkhYR0lYTUdNZWRqeTFIMjNHOVJ3CnJSMnc5NFEyU0IzWTFtUkw2YVFS
+            MU02VFo1TE14YVpZRzNhVjZwRTZwd0UKLS0tIDFaMWF4NUZjZmhRdXRuZFJYNkg3
+            SUUzN1FnMlI4bkkzbVNlMWhUVnJZODAKQ+Iy86CIv/Oj9E9j4E/0y5hDbcctRuqB
+            OR825KRPRaTdF/G2hudE1R1JbM4wGbPLMzv078R4ZXHDPGvDttSEkA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-06T06:10:31Z"
+    mac: ENC[AES256_GCM,data:6wAGAwWL7jI/NEeswBnUiWoRty+bb5DEsGEAPXajKNNYCwtm5yuXBrcOI/q4w6H8FG+nqVAECkdpAOI3S9mjDfc2zlWAzfCJJdDic92RKPXOC0tOH/M1BBSGJ1fVdJGAZZT1yAVxmtYy9WohJpawv3kmsVv+JnD+TIqmZ1CJHXo=,iv:srs4o0olSwmraQQgmdPxb+4h/t6fHHeFKJ2z914kGgY=,tag:jjoIKhU8Tu9i5SJkvkfcCg==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4
@@ -0,0 +1,76 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
+# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
+# (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
+# daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
+# register (peers.conf) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
+#
+# WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
+# source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`
+# would put the single credential every enrolled device depends on OUTSIDE the GitOps tree, where a
+# cluster rebuild silently loses it — and losing it means RE-ENROLLING EVERY DEVICE, because each
+# client profile pins this key's public half.
+#
+# *** THE REAL FILE IS COMMITTED WITH `privatekey: REPLACE-ME`, ALREADY ENCRYPTED. ***
+# That is deliberate, so the wiring is complete and green before the key exists: the resource is
+# already listed in kustomization.yaml and both overlays already carry the `vpn-ingress` decryption
+# patch. Your job is ONLY to replace the value (step 3 below). Until you do, the DaemonSet
+# CrashLoops with an explicit FATAL naming this file — the init container validates the value with
+# `wg pubkey` before writing wg0.conf, so a placeholder fails CLOSED and LOUDLY rather than
+# producing a server that silently cannot complete a handshake.
+#
+# ---------------------------------------------------------------------------------------------
+# EVERYTHING RUNS IN THE DEV CONTAINER. `wg`, `sops` and `qrencode` are all baked into the image
+# (infra/devcontainer/Containerfile) because that image IS the toolchain — CLAUDE.md §5, and the
+# devcontainer README's "Don't install the toolchain on the host". One shell, one toolchain, so
+# `wg genkey | sops set` is just a pipeline.
+# ---------------------------------------------------------------------------------------------
+#
+# HAND-OFF STEPS (operator — an agent must not mint a long-lived private key, CLAUDE.md §6).
+# Full procedure with verification: runbooks/vpn-remote-access.md step 2.
+#
+#  1. Generate the key STRAIGHT INTO this file. The plaintext never lands on disk and never
+#     reaches a terminal; `sops set` edits the already-encrypted document in place:
+#         ./infra/devcontainer/dev.sh run bash -c '
+#           cd /work
+#           umask 077
+#           sops set platform/vpn-ingress/wg-server-key.sops.yaml \
+#             "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
+#         '
+#     (Or edit it: `./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml`.)
+#
+#  2. Record BOTH halves in your PASSWORD MANAGER. The encrypted file is in git and Velero captures
+#     this namespace to R2, so this is belt-and-braces — but it is the only copy that survives
+#     losing the cluster AND the backups, and the key cannot be regenerated without invalidating
+#     every client profile.
+#         # public half (safe to display; it is what every client profile carries)
+#         ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d \
+#           --extract "[\"stringData\"][\"privatekey\"]" \
+#           platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
+#
+#  3. Confirm the value is a real key without printing it — `wg pubkey` succeeds only on a valid
+#     32-byte curve25519 private key, which is exactly the check the pod's init container runs:
+#         ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d \
+#           --extract "[\"stringData\"][\"privatekey\"]" \
+#           platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null \
+#           && echo "valid WireGuard key"'
+#
+#  4. PR it. Nothing else to wire: the resource line and both overlay decryption patches already
+#     exist. Flux rolls the DaemonSet and the pods leave CrashLoop.
+#
+# NOTE infra/vpn-ingress/new-peer.sh reads THIS FILE to derive each client profile's
+# `[Peer] PublicKey`, so there is no separate copy of the key to keep in step with anything.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: wg-server-key
+  namespace: vpn-ingress
+type: Opaque
+stringData:
+  # The server's WireGuard private key: `wg genkey` output — 44 chars of base64 ending in '='.
+  # NOT a passphrase, and NOT the public key. The init container validates it with `wg pubkey`
+  # before writing wg0.conf, so a placeholder or a mangled value fails closed with a clear message.
+  privatekey: REPLACE-ME
@@ -67,7 +67,9 @@ Work through these in order; the first three are by far the most likely.
 | Symptom in the log | Cause | Fix |
 |---|---|---|
 | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
-| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) |
+| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
+| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
+| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
 | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
 | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
 | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
@@ -21,7 +21,7 @@ behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's
 DNS, and internet egress out the house WAN.

 Authorisation is one thing only: the device's **public key is in
-`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
+`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.

 > **An enrolled device is as privileged as a machine sitting in the house** — including the three
 > k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
@@ -34,10 +34,26 @@ deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module.
 `wg-quick` fails outright and the pod CrashLoops.

 ```bash
-# operator action (Ansible is not an agent action)
-cd infra/ansible && ./site.sh --tags system --limit k3s_prod
+# operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox,
+# and ansible.cfg is only picked up from infra/ansible, so cd there inside the container.
+./infra/devcontainer/dev.sh run bash -c \
+  'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
 ```

+Three details in that command are load-bearing, not style:
+
+- **`-e k3s_hosts=k3s_prod`.** `node-baseline.yml` targets `k3s_servers` by default, which is the
+  **dev** group — and dev was decommissioned, so an unqualified run does nothing useful. This is
+  the targeting pattern `infra/ansible/README.md` "Targeting prod" documents for every playbook.
+- **`--tags system`, and never an untagged run.** The two kernel-module tasks carry the `system`
+  tag. An untagged run would also execute the `disks` tasks, which assert on block-device variables
+  prod has deliberately left as placeholders (`infra/PROD-PROVISIONING.md` §2) — the playbook's own
+  header warns about this. `--tags system` is broader than the modules alone (it also re-asserts
+  sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and
+  already in its desired state on a live node.
+- **It is a playbook, not a wrapper script.** There is no `site.sh` in this repo; `playbooks/site.yml`
+  is the full bring-up chain and is the wrong tool here.
+
 Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded:

 ```bash
@@ -46,46 +62,109 @@ lsmod | grep '^wireguard'
 grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf
 ```

-## Step 2 — create the server key (once)
+## Step 2 — fill in the server key (once)

-This is the **only** secret in the component, and the one thing that must never be generated in an
-agent session or pasted into a transcript. Generate it on your own machine:
+The key is **SOPS-encrypted in this tree** (R-SEC-01, spec docs/07), not created by hand:
+`platform/vpn-ingress/wg-server-key.sops.yaml`. It is already committed — encrypted, with a
+`REPLACE-ME` placeholder — and it is already listed in the component's `resources:` with a
+`decryption:` patch in both overlays. **Nothing needs wiring. You are only replacing one value.**
+
+Why in git rather than `kubectl create secret`: an out-of-band Secret puts the one credential every
+enrolled device depends on outside the GitOps tree, where a cluster rebuild silently loses it — and
+losing it means re-enrolling every device, because each client profile pins this key's public half.
+
+Everything below runs **inside the dev container** — `wg`, `sops` and `qrencode` are all baked into
+the image (CLAUDE.md §5: the toolchain lives only in the pinned container; don't install it on the
+host). One shell, one toolchain.
+
+**1. Generate the key straight into the encrypted file.** One pipeline: `wg genkey` makes it,
+`sops set` writes it into the already-encrypted document in place. The plaintext never lands on
+disk and never reaches a terminal.

 ```bash
-mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
-umask 077
-wg genkey > ~/.config/wireguard-home/server.key          # keep this; see "Back it up" below
+./infra/devcontainer/dev.sh run bash -c '
+  cd /work
+  umask 077
+  sops set platform/vpn-ingress/wg-server-key.sops.yaml \
+    "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
+'
 ```

-Load it into the cluster. **Do not echo it, do not pass it on an interactive command line that
-lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE.md §6):
+**2. Record it in your PASSWORD MANAGER.** Derive both halves from the file you just wrote:

 ```bash
-./infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \
-     --from-file=privatekey=/work/.cache/server.key'
+# the PUBLIC half — safe to display, and what every client profile carries
+./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
+  platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
+
+# the PRIVATE half — paste into the password manager, then clear your scrollback
+./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
+  platform/vpn-ingress/wg-server-key.sops.yaml'
 ```

-…where `/work/.cache/server.key` is a path you copied the key to **inside the repo's gitignored
-cache** (or mount it however you prefer — the requirement is only that the bytes never pass through
-a log). Delete that copy afterwards: `shred -u .cache/server.key`.
+The second command is the one place this key is deliberately shown. The encrypted file is in git and
+Velero captures this namespace to R2, so the password-manager copy is belt-and-braces — but it is
+the only copy that survives losing the cluster *and* the backups, and the key cannot be regenerated
+without invalidating every client profile.

-Confirm it exists **by key name only**:
+**3. Confirm the value is a real key**, without printing it — if `wg pubkey` prints a key, the
+value is valid; if it errors, it is still wrong:

 ```bash
-./infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1'
-# expected: {"privatekey   -- a NAME. If you see base64 payload, you printed the key. Rotate it.
+./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
+  platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null && echo "valid WireGuard key"'
 ```

-**Why out of band and not SOPS.** An agent must not mint a long-lived private key, and a key that
-passed through a session transcript would have to be treated as compromised. The trade-off is that
-this Secret is **not reconstructible from git** — so:
+That is the same check the pod's init container runs, so a pass here predicts a pass there.

-**Back it up.** Keep `server.key` in your password manager. Velero's `daily` schedule excludes only
-`monitoring` and `unifi` (`platform/velero/helmrelease.yaml`), so this namespace *is* captured
-off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work
-again after a cluster rebuild — **losing it means re-enrolling every device.**
+**4. PR it, and merge.** Flux rolls the DaemonSet and the pods leave CrashLoop.
+
+If you would rather use an editor than `sops set`:
+`./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml` (replace
+`REPLACE-ME`, save, exit — it re-encrypts on write).
+
+### What the pod tells you if this went wrong
+
+The init container validates the value with `wg pubkey` before writing `wg0.conf`, and names the
+specific failure in its log — so you never get a server that silently cannot complete a handshake:
+
+| pod log says | meaning |
+|---|---|
+| `still the committed REPLACE-ME PLACEHOLDER` | step 3 not done (the expected fresh-install state) |
+| `It is EMPTY` | the value was blanked |
+| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
+| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
+| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
+| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |
+
+### What a healthy start looks like
+
+Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key:
+
+```
+wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820.
+```
+
+then the `wireguard` container brings the tunnel up:
+
+```
+**** Found WG conf /config/wg_confs/wg0.conf, adding to list ****
+**** Activating tunnel /config/wg_confs/wg0.conf ****
+[#] ip link add dev wg0 type wireguard
+[#] wg addconf wg0 /dev/fd/63
+[#] ip -4 address add 10.90.0.1/24 dev wg0
+[#] ip link set mtu 1420 up dev wg0
+```
+
+**Ignore the `chown: ... Operation not permitted` run before that**, and the image's own
+"Permissions could not be set ... we will not provide support for it". Capabilities are dropped to
+`NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs
+as root, and `wg-quick` proceeds. It is wrong about the cause and it does not matter.
+
+**A pod stuck `0/1 Running` with `Startup probe failed: Unable to access interface: No such device`
+means `wg-quick` tore the tunnel down** — read the `wireguard` container log for the `[#]` line it
+died on. That is the shape of a real failure; the probe is doing its job of refusing to call a
+server without a `wg0` "ready".

 ## Step 3 — forward udp/51820 on the router (once)

@@ -126,17 +205,37 @@ unauthenticated traffic either way. If the pod is Ready, the interface exists.
 ## Step 5 — enrol a device

 ```bash
-./infra/vpn-ingress/new-peer.sh pixel-8 2              # full tunnel  (LAN + WAN egress)
-./infra/vpn-ingress/new-peer.sh thinkpad 3 --split     # split tunnel (LAN only)
+# in the dev container, like everything else
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2'            # full tunnel
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh thinkpad 3 --split'   # LAN only
 ```

-It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout,
-prints a QR code if `qrencode` is installed, and prints the `[Peer]` stanza to add. Then:
+It derives the server's public key from `wg-server-key.sops.yaml` itself, so there is no local key
+copy to keep in step with anything. Profiles land in `.cache/wg-peers/` — gitignored, and persistent
+because the repo is the container's only read-write mount.

-1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
+It generates the device's keypair, writes a `0600` client profile, prints a QR code, and prints the
+`[Peer]` stanza to add. It refuses to write a profile anywhere git does not ignore.
+
+Then:
+
+1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
   `# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
   comment is the only record of which stanza belongs to which device.
-2. PR → merge. Flux rolls the DaemonSet (~1 min).
+2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
+   **content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
+   which changes the pod template, which forces the roll. (It did **not** work this way at first —
+   a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
+   keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:
+
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
+        -o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
+   ```
+
+   The generation must have **increased** since before the merge. If it is still `1`, the roll did
+   not happen and `wg show` will list no peer no matter how long you wait.
 3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
   Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
 4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
@@ -176,7 +275,7 @@ if the router is not delivering the packets, and a `handshake` with no completed

 ## Revoking a device

-Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
+Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
 next reconcile. **Never re-use that address octet**: an old profile would silently start working
 again as a different device.