feat(keycloak): declare passkey (WebAuthn passwordless) login for the homelab realm

Adds the realm's WebAuthn Passwordless policy and restructures the browser-mfa flow so a passkey is an ALTERNATIVE beside the password form. Additive and optional by design: this IdP gates every SSO app plus kubectl, so a mandatory passkey flow that misfires is a total lock-out. Enrolment is not forced. Policy: RP ID is ${BASE_DOMAIN} (not the keycloak host) because a passkey's RP ID is fixed at enrolment and must not be bound to a name that can move; resident key required (a non-discoverable credential is a second factor, not a passkey); user verification required (without it, possession alone is a complete login). Flow: auth-username-password-form is split into auth-username-form + auth-password-form, because the server must know who is logging in before it can decide whether a passkey exists. Conditional OTP for platform-admins is relocated intact onto the password branch only — a UV-required passkey is already two phishing-resistant factors. No requiredActions block is declared, deliberately: Keycloak 26.0.6 treats requiredActions as REPLACE not merge, so declaring one would delete the default set including CONFIGURE_TOTP and lock every platform-admin out of the OTP step this flow still enforces. webauthn-register-passwordless is already registered enabled-but-not-default by Keycloak's own defaults. UNAPPLIED: the realm import is a one-shot resting at Done=True/Started=False, so this reaches the live realm never. See the handoff entry and the README section. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Merge pull request 'fix(oauth2-proxy): gate startup on Keycloak discovery, or a cold boot CrashLoops the proxy' (#70 ) from fix/oauth2-proxy-oidc-startup-race into main
2026-10-17 23:23:36 -05:00 · 2026-10-18 04:14:52 +00:00 · 2026-10-17 22:47:30 -05:00 · 2026-10-10 16:13:46 +00:00 · 2026-10-10 16:09:53 +00:00 · 2026-10-10 13:55:15 +00:00
31 changed files with 2288 additions and 344 deletions
@@ -37,6 +37,14 @@ coverage/
 infra/tofu/backend.hcl
 infra/tofu/.r2.env
 # Operator scratch, writable from inside the dev container (the repo is its only persistent
 # read-write mount). Holds things that must never be committed: a staged copy of the SOPS age or
 # WireGuard key, rendered VPN client profiles (infra/vpn-ingress/new-peer.sh writes here by
 # default). `infra/ansible/.gitignore` already does this for its own .cache/; this is the same
 # convention at the root. Do NOT rely on `*.key` alone to protect a staged key — that only covers
 # one filename shape, which is why this rule exists.
 .cache/
 # Claude Code local scratch: agent worktrees + session state. Nothing here is tracked;
 # ignoring it keeps `git add -A` from staging embedded worktree repos and keeps the lint
 # stage (which scans untracked files) from tripping on .claude/worktrees/.
@@ -77,7 +77,7 @@ spec:
  storageClassName: longhorn
  resources:
    requests:
-      storage: 8Gi
+      storage: 16Gi
 ---
 # PVC plex-media (R-STATE)
 apiVersion: v1
@@ -351,7 +351,7 @@ exposure: public              # edge port-forward + public DNS; clients + Seerr
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
  - name: config
-    size: 8Gi
+    size: 16Gi
    class: longhorn           # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
    mountPath: /config        # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
                              # renderer default /data/config. Without this the PVC sat unused and Plex wrote to the
@@ -359,7 +359,7 @@ storage:
                                # The live /config contents were copied onto each PVC before this landed,
                                # so API keys and credentials other apps reference were preserved.
  # THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
-  # (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
+  # (192.168.2.30:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
  # /downloads, which is precisely where this app is ALREADY configured to write.
  #
  # WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
@@ -434,7 +434,7 @@ dependencies: []                # default-deny: reaches only DNS; arr reach it n
                                # for weeks and neither has any CNP naming the NFS server or port 2049
                                # (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
                                # egress gateway off this traffic (its selector is a podSelector), and
-                                # 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
+                                # 192.168.2.30 is inside the 192.168.0.0/16 excludedCIDR besides — see
                                # platform/vpn-egress/egress-steering.yaml, "BULK NFS".
 ingressRoutes:
  - host: qbittorrent.${BASE_DOMAIN}
@@ -234,26 +234,28 @@ data:
  WG_CLIENT_CIDR: "10.90.0.0/24"             # remote VPN clients; .1 = server. NOT a LAN range
  # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
-  # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
+  # THE address of the NAS whose ZFS datasets back every `class: bulk` volume: the media library
-  # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
+  # (Storage/plex, 8.33 T — plex, sonarr, radarr, qbittorrent) and the legacy Nextcloud archive
  # (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
  # `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
  # declared in the ServiceDefinition, because the path is identical in every environment and only
  # the ADDRESS is not.
  #
-  # WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
+  # IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read "the two clusters are on
-  # The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
+  # OPPOSITE SIDES of pfSense": the datasets lived on a Proxmox host whose only interface was
-  #   dev  nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
+  # vmbr0 192.168.1.9/24 on pfSense's LAN, which prod (192.168.2.0/24, pfSense's WAN) could reach
-  #        NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
+  # only through a DNAT at 192.168.2.102. Both pools were physically moved onto a standalone
-  #        /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
+  # TrueNAS at 192.168.2.30 — prod's OWN /24 — so there is no crossing to arrange and no candidate
-  #        per the convention of the LAN block above.
+  # values to choose between. Measured 2026-10-07: 192.168.2.102:2049 no longer answers;
-  #   prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
+  # 192.168.2.30:2049 and :111 do. There is now exactly ONE storage host and one correct value;
-  #        22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
+  # this stays a knob only because the renderer reads it per-environment.
-  #        the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
+  #
-  #        DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
+  # *** CHANGING THIS VALUE IS NOT A PLAIN RECONCILE. *** A bound PersistentVolume's spec.nfs is
-  # So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
+  # immutable — measured by server dry-run 2026-10-07:
-  # blast radius of shipping it unset, which is NOT the usual "fails to apply".
+  #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
-  BULK_NFS_SERVER: "192.168.1.9"
+  # so Flux's apply FAILS rather than updating, and the pods keep their old mount. The PVs must be
  # deleted and recreated; the operator sequence is docs/BULK-NFS-STORAGE.md §10.
  BULK_NFS_SERVER: "192.168.2.30"
  # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
  # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
@@ -8,8 +8,8 @@
 #     NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves
 #     (DECISIONS-LOG: "Headlamp gating stall").
 #   * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the
-#     operator creates the out-of-band `wg-server-key` Secret (runbooks/vpn-remote-access.md
+#     operator replaces the `REPLACE-ME` placeholder in platform/vpn-ingress/wg-server-key.sops.yaml
-#     step 1). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
+#     with a real key (runbooks/vpn-remote-access.md step 2). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
 #     the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the
 #     cause (a missing manual step) invisible from any Flux condition.
 #
@@ -51,6 +51,9 @@ spec:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
-  # NO `decryption:` stanza is needed in the overlays: this path carries no *.sops.yaml. The server
+  # SOPS decryption is layered per-environment by the overlay (clusters/dev and clusters/prod each
-  # key is an out-of-band Secret (platform/vpn-ingress/kustomization.yaml says what to do if that
+  # patch THIS Kustomization by name), identical to headlamp/crafty/velero — this path carries
-  # ever changes).
+  # platform/vpn-ingress/wg-server-key.sops.yaml, the WireGuard server private key. The `sops-age`
  # Secret is created out of band and referenced by NAME only. Without the overlay patch the Secret
  # applies cleanly full of ENC[...] and only the tunnel is broken, so the patch is as load-bearing
  # as the encrypted file itself.
@@ -122,6 +122,24 @@ patches:
          provider: sops
          secretRef:
            name: sops-age
  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
  # The `sops-age` Secret is out-of-band (name only).
  - patch: |-
      apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      metadata:
        name: vpn-ingress
        namespace: flux-system
      spec:
        decryption:
          provider: sops
          secretRef:
            name: sops-age
  # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
  # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
  # account and the Anthropic subscription token from `claude setup-token`. Without this
@@ -261,15 +279,15 @@ patches:
      - op: replace
        path: /data/CRAFTY_L4_VIP
        value: "192.168.1.55"
-      # Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
+      # Bulk (external NFS) storage host. THIS VALUE IS INERT — dev is decommissioned, its nodes
-      # (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
+      # (192.168.1.40/.41/.42) no longer exist, and the Proxmox host this used to name (192.168.1.9)
-      # Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
+      # went with the pools it held. It is restated per this overlay's "supply every knob"
-      # Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
+      # convention, not because anything consumes it. It carries the real address so that no file
-      # end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
+      # in the tree renders a host that is gone; it has NOT been proven from a dev node, because
-      # change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
+      # there is no dev node to prove it from.
      - op: replace
        path: /data/BULK_NFS_SERVER
-        value: "192.168.1.9"
+        value: "192.168.2.30"
      # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
      # barmanObjectStore and Velero's backupStorageLocation write to.
      #
@@ -113,6 +113,24 @@ patches:
          provider: sops
          secretRef:
            name: sops-age
  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
  # The `sops-age` Secret is out-of-band (name only).
  - patch: |-
      apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      metadata:
        name: vpn-ingress
        namespace: flux-system
      spec:
        decryption:
          provider: sops
          secretRef:
            name: sops-age
  # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
  # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
  # account and the Anthropic subscription token from `claude setup-token`. Without this
@@ -522,60 +540,48 @@ patches:
        path: /data/LAN_CIDR
        value: "192.168.2.0/24"
      # ------------------------------------------------------------------------------------------
-      # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
+      # BULK (EXTERNAL NFS) STORAGE HOST — the TrueNAS at 192.168.2.30.
      # ------------------------------------------------------------------------------------------
      # Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
-      # radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
+      # radarr, qbittorrent) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
      #
-      # THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
+      # THIS WAS THE REPO'S ONE LOAD-BEARING PLACEHOLDER, AND THE REASON IT COULD BE IS NOW GONE.
-      # vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
+      # The datasets lived on a Proxmox host with a single interface on pfSense's LAN
-      # 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
+      # (192.168.1.9), which prod could reach only through a DNAT at 192.168.2.102 — so the correct
-      # 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
+      # value here depended on an undecided operator choice, and guessing it would have authored a
-      # outcomes give DIFFERENT values here:
+      # cluster that mounts nothing. Both pools were physically moved onto a standalone TrueNAS at
-      #     port-forward 192.168.2.102:2049 -> 192.168.1.9:2049   =>  "192.168.2.102"
+      # 192.168.2.30, prod's OWN /24. No crossing to arrange, no candidates to choose between.
-      #     a WAN->LAN route to 192.168.1.0/24                    =>  "192.168.1.9"
+      # Measured 2026-10-07: 192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and :111 do.
      # Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
      # ever became reachable — mounts something unintended. So it stays a placeholder.
      #
-      # WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
+      # ONE PORT IS STILL ALL IT NEEDS: tcp/2049. The PVs mount nfsvers=4.1 and NFSv4 carries mount
-      # NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
+      # and lock in-protocol — no rpcbind/111, no mountd, no statd. The NAS advertises
-      # over v3 was in part this decision's cost: v3 would have needed four ports and three of them
+      # `+3 +4 +4.1 +4.2` (/proc/fs/nfsd/versions, measured), so 4.1 negotiates.
      # on ephemeral numbers.
      #
      # WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
      # Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
      # the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
      # address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
      # 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
      # addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
      #
      # *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
      # A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
      # placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
      # failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
      # TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
      # ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
      # apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
      # a revision containing the bulk volumes; the dev overlay renders a working value, which is
      # where these manifests are meant to be proven first.
      - op: replace
        path: /data/BULK_NFS_SERVER
-        # 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
+        # VERIFIED ON THE NAS 2026-10-07, before any cluster touched it (`exportfs -v`):
-        # host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
+        #   /mnt/Storage/plex       192.168.2.10/.11/.12  rw,sec=sys,secure,root_squash,no_all_squash
-        # reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
+        #   /mnt/Storage/nextcloud  192.168.2.10/.11/.12  ro,sec=sys,secure,root_squash,no_all_squash
        # keeps the base value 192.168.1.9.
        #
-        # PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
+        # no_all_squash is the flag the entire permission model rests on: client uids pass through
-        #   - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
+        # untouched, so the arrs' PUID=1000 arrives as the owner of a 0755 1000:3000 tree. sec=sys
-        #   - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
+        # keeps that numeric rather than principal-mapped. The export deliberately does NOT carry
-        #     carries no_root_squash, so a root write would have passed and proved nothing, while
+        # no_root_squash, which the Proxmox one did: nothing here needs it, because every write to
-        #     LSIO images default to PUID 911 and would have gone Ready then failed every import
+        # the tree happens after the LSIO drop to uid 1000 and the images only chown their Longhorn
-        #     with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
+        # /config. Both subPath directories (plex/downloads, nextcloud/nextcloud) already exist at
-        #   - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
+        # 0755 1000:3000, so a squashed root can still traverse them for subPath resolution.
-        #     real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
+        #
-        # pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
+        # The ro= posture on the archive is now SERVER-enforced, which the old export also managed.
-        # so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
+        # The media tree's read-only half (plex) stays a CLIENT-side guarantee — all four media pods
-        value: "192.168.2.102"
+        # can land on any node, so the server sees the same three addresses for reader and writers
        # alike and cannot tell them apart. Unchanged by the move; see docs/BULK-NFS-STORAGE.md §5.
        #
        # *** CHANGING THIS VALUE REQUIRES DELETING AND RECREATING THE PVs. *** A bound PV's
        # spec.nfs is immutable — measured by server dry-run 2026-10-07:
        #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
        # so Flux's apply FAILS rather than updating and the pods keep their old, dead mount. The
        # operator sequence is docs/BULK-NFS-STORAGE.md §10. Retain + the renderer's pre-set
        # claimRef make it safe: no data is touched and each recreated PV rebinds its own PVC.
        value: "192.168.2.30"
      # ==========================================================================================
      # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
      # ==========================================================================================
@@ -0,0 +1,187 @@
 ---
 date: 2026-10-05
 branch: fix/vpn-server-key-in-sops
 title: VPN server key moves into SOPS-encrypted git — the out-of-band Secret broke R-SEC-01 and the DR story
 ---
 ## What changed
 **Operator correction (2026-10-05): "secrets are kept in source through sops".** They are right, and
 #55 shipped the WireGuard server private key the wrong way: as a Secret the operator was told to
 `kubectl create` out of band, referenced by name only. This branch moves it into the tree as
 `platform/vpn-ingress/wg-server-key.sops.yaml`, where every other secret in this repo lives.
 **Why the original choice was wrong, not merely unconventional.** Two concrete costs, both of which
 #55's own README half-admitted and then shipped anyway:
 1. **R-SEC-01 / spec docs/07 says secrets live in source, SOPS-encrypted, reaching the cluster only
   through Flux.** An out-of-band Secret is outside the GitOps tree by construction.
 2. **It broke DR for the one credential that cannot be re-derived.** A cluster rebuild restores
   everything in git; it does not restore a hand-made Secret. Losing this key means **re-enrolling
   every device**, because each client profile pins its public half. #55 papered over this with
   "keep a copy in your password manager", which is a workaround for a self-inflicted gap.
 Note `platform/kyverno/admission/secret-provenance.yaml` did **not** forbid the original shape —
 it scopes to `app-namespace=true` namespaces and `vpn-ingress` is deliberately a platform namespace,
 so admission would have accepted the hand-applied Secret. The rule it violated was the written
 requirement, not an enforced control. Recording that because "the gate let it through" was never
 evidence it was correct.
 **How the unfillable-key problem is solved.** An agent still must not mint a long-lived private key
 (CLAUDE.md §6), so the file is committed **already encrypted, with `privatekey: REPLACE-ME`**. That
 makes the wiring complete and the gate green from the first commit — the resource line and both
 overlay decryption patches are present — and leaves the operator exactly one act: replace the value.
 The alternative, a commented-out resource line to uncomment later, is precisely the "file in git with
 no generator entry: reviewed, merged, apparently done, still broken" shape this repo keeps getting
 bitten by (`alert-runbooks-resolvable` exists because of it).
 **The init-container guard is now a validation, not an emptiness check.** It was `[ -z "$key" ]`.
 Because the committed placeholder is non-empty, the likely failure is now a value that is present
 and *wrong*, so the guard pipes the value through `wg pubkey` — which accepts only a real 32-byte
 curve25519 private key — and names which of four cases it hit:
 | case | why it matters |
 |---|---|
 | `REPLACE-ME` placeholder | the expected fresh-install state |
 | empty | blanked value |
 | **starts with `ENC[`** | **the Secret reached the cluster UNDECRYPTED.** A missing overlay decryption patch applies cleanly and leaves every Flux condition green — this is the silent failure the guard exists for |
 | rejected, none of the above | mangled value, or a *public* key pasted by mistake |
 Files:
 - `platform/vpn-ingress/wg-server-key.sops.yaml` (new) — SOPS-encrypted to the repo's age recipient,
  `stringData.privatekey` only, placeholder value.
 - `platform/vpn-ingress/wg-server-key.sops.yaml.example` (new) — plaintext template + operator
  hand-off, following the `crafty-oauth2-proxy-credentials.sops.yaml.example` convention.
 - `platform/vpn-ingress/kustomization.yaml` — the resource added; the "DELIBERATELY ABSENT: a
  Secret" block replaced with the overlay-patch warning.
 - `platform/vpn-ingress/daemonset.yaml` — the `wg pubkey` guard; volume comment rewritten.
 - `clusters/dev/kustomization.yaml`, `clusters/prod/kustomization.yaml` — **the `vpn-ingress`
  `decryption:` patch, in both.** This is the load-bearing half: #55's own kustomization comment
  predicted that forgetting it applies ciphertext cleanly.
 - `clusters/base/vpn-ingress.yaml` — its "NO decryption stanza is needed" comment was now false.
 - `platform/vpn-ingress/README.md`, `runbooks/vpn-remote-access.md` — step 2 rewritten.
 - `infra/vpn-ingress/new-peer.sh` — stale "step 1" pointer.
 **A second finding: `wg` and `sops` were not in the same place — and the first fix for that was
 also wrong.** Measured: `wg` existed only on the host (`/usr/bin/wg`), `sops` only in the dev
 container, `qrencode` on neither. So `sops -d … | wg pubkey` could not run in one shell, and #55's
 step 2 — which assumed it could — would have failed.
 My first fix was to **split every step by where it runs**, and to tell the operator to
 `dnf install qrencode` on their workstation. **Operator correction: "sop, wg, and any other tool
 requirement is supposed to be baked into the devcontainer."** That is CLAUDE.md §5 and the
 devcontainer README's "Don't install the toolchain on the host", and the split fix broke the
 invariant rather than honouring it — it also produced a procedure that could never be reproduced in
 CI, and which no one should have to follow.
 **The real fix:** `wireguard-tools` and `qrencode` are now distro packages in
 `infra/devcontainer/Containerfile`, alongside ansible/openssh/diffutils. Both are in Fedora 44's
 repos (`wireguard-tools` 1.0.20260223 — the same upstream version as the container image this
 component runs — and `qrencode` 4.1.1), unpinned like every other dnf package there, because the
 pinned thing is the Fedora base and neither is a gate tool whose version changes what green means.
 Three things got *simpler* rather than merely relocated, which is the tell that the operator's rule
 was the right one:
 1. **Step 2 is now one pipeline in one shell:** `sops set … "\"$(wg genkey)\""` generates the key
   straight into the already-encrypted document. The plaintext never lands on disk, never reaches a
   terminal, and there is no staged copy to shred. The previous version needed a copy in
   `.cache/server.key`, a `git check-ignore` warning about the `*.key` suffix being the only thing
   protecting it, and a sha256 comparison to prove the right bytes arrived. All of that is gone.
 2. **`new-peer.sh` now derives the server's public key from `wg-server-key.sops.yaml` itself**
   (`sops -d --extract … | wg pubkey`), so there is no local key copy to keep in step with the
   encrypted one. It previously required the operator to keep the private key at a magic host path.
 3. **`.cache/` is now gitignored at the repo root** (mirroring `infra/ansible/.gitignore`), so the
   container has a persistent, non-committable scratch directory. `new-peer.sh` writes client
   profiles to `.cache/wg-peers/` by default — necessary because the repo bind-mount is the
   container's ONLY persistent read-write mount, so a profile written to `/tmp` in there is gone
   before it can be moved to a desktop. Its old blanket "refuse to write inside the checkout" guard
   is replaced by `git check-ignore`, which tests the actual property instead of a proxy.
 **SENSITIVE PATHS — this PR touches `clusters/**` (two overlays) AND adds a `**/*.sops.yaml`, so
 `ci/approval-policy.sh` will BLOCK it.** It needs the operator's `/approve-override <reason>` as a
 PR **review** of type "Comment". I have deliberately not posted it (CLAUDE.md §1).
 ## What I verified
 - **The encrypted file is real and correctly targeted.** `sops -e -i` in the pinned toolbox
  produced `stringData.privatekey: ENC[AES256_GCM,...]` with cleartext keys, `encrypted_regex
  ^(data|stringData)$`, and exactly one age recipient:
  `age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg` — the recipient in the root
  `.sops.yaml`, i.e. the key Flux holds as `sops-age`.
 - **Round-trip, without exposing anything**: `sops -d` then a Python read of key names only →
  `stringData keys: ['privatekey']`, and `value is still the placeholder: True`.
 - **Both overlay patches are present and name the right Kustomization** (`name: vpn-ingress`,
  `namespace: flux-system`, `decryption.secretRef.name: sops-age`), inserted after each overlay's
  `velero` stanza, mirroring headlamp/crafty/velero.
 - **`kubectl kustomize platform/vpn-ingress` builds** and now emits five objects — Namespace,
  ConfigMap, **Secret**, DaemonSet, NetworkPolicy — with `${WG_CLIENT_CIDR}` still an unexpanded
  token for Flux.
 - **The tool gap, measured on both host and container before the fix**: `wg` host-only, `sops`
  container-only, `qrencode` on neither.
 - **Both packages exist in the image's repos**, queried from inside the running container:
  `wireguard-tools` 1.0.20260223 (repo `updates`) and `qrencode` 4.1.1 (repo `fedora`).
 - **The image REBUILDS with them and both tools resolve inside it** —
  `./infra/devcontainer/dev.sh build` succeeded, and `wg --version` / `qrencode --version` /
  `sops --version` all answer in the rebuilt container. This is the claim the change rests on, so
  it was executed rather than reasoned.
 - **`new-peer.sh`'s new SOPS path fails correctly against the placeholder.** Run in the rebuilt
  container it refused with "could not derive the server public key … it still holds the REPLACE-ME
  placeholder", which is both the expected pre-key behaviour and a live test of the
  `sops -d | wg pubkey` pipeline the whole design now depends on.
 - **`.cache/` is ignored regardless of filename** now: `git check-ignore -v .cache/anything.conf`
  and `.cache/serverkey.txt` both match `.gitignore:46:.cache/`. Before this change only `*.key`
  matched, which is the fragility the previous runbook had to warn about.
 - **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no cluster).
  **Not green. Reported as rc 3, not as pass.**
 - **The L0 gate caught THREE real defects in this branch before it could merge — including one
  in the fix for another — and I had asserted one of them was fine without running it.** Both are recorded because the gate's value here is the
  whole point of the exercise:
  1. **`no-plaintext-secrets` FAILED**: "`wg-server-key.sops.yaml.example`: an unencrypted .example
     template is referenced by a kustomization — it would be applied as a plaintext Secret". My
     kustomization named the template in a *comment*, and the check matches comments too (correctly:
     naming one is how a plaintext Secret gets applied by accident). `platform/crafty/kustomization.yaml`
     never names its own example either — the convention was there to copy and I missed it. Fixed by
     not naming the file; the hand-off lives inside the example, where it belongs.
     **An earlier draft of this entry claimed this check passed. It did not. The claim was written
     from expectation rather than from a run.**
  2. **`flux-postbuild-substitution` FAILED** on `${key#ENC[}` in my new guard
     (`daemonset.yaml:155`), flagged `[SILENT-EMPTY]`: envsubst consumes every `${...}`, so my
     shell prefix-strip would have rendered to the empty string — the ciphertext branch of the guard
     would never have fired, which is precisely the silent failure the guard was added to catch.
     Rewritten with `case`, which uses no braces and so has nothing to escape.
  3. **And then it failed a THIRD time, on the fix's own comment** — `[FATAL-PARSE]
     daemonset.yaml:156 ${...}`, the prose explaining the rule, written with the literal form. That
     class is worse than it looks: a `FATAL-PARSE` hazard fails the **entire** Kustomization, every
     resource in the path, not just the line. Rewritten as "dollar-brace expansion". This is the
     second time in this component that writing the hazard literally inside a comment about the
     hazard has been the defect (the first was in #55), so the habit to build is: **in any
     substituted path, describe the construct in words, never type it.**
 - **After both fixes, `no-plaintext-secrets` and `flux-postbuild-substitution` pass** (446 and 198
  assertions respectively), and `l0` is green.
 ## What I did NOT verify
 - **That Flux decrypts this file on prod.** The overlay patches have never reconciled. This is the
  whole risk of the change and it has exactly one observable outcome: after merge the pods should
  CrashLoop reporting the **`REPLACE-ME` placeholder** (proving decryption worked), *not* reporting
  **`ENC[`** (which would prove a patch is wrong). I built the guard to tell those apart because I
  cannot test them.
 - **That `wg pubkey` accepts the operator's real key through this path.** The guard is reasoned from
  `wg pubkey`'s behaviour and has never run — `wg` is absent from both this workstation's container
  and the gate, so the first execution will be in the pod.
 - **The `sops set` command in the hand-off.** I verified `sops -e -i` and `sops -d` in the toolbox;
  I did **not** run `sops set` with the quoting shown, and nested quoting inside
  `dev.sh run bash -c` is exactly where a documented command tends to break. If it misbehaves, the
  editor form (`sops <file>`) is the fallback and does not depend on quoting.
 - **The sha256 comparison in step 2 step 4** is untested end to end (it needs the real key).
 - **Whether anything else in #55 assumed a tool is available where it is not.** I checked `wg`,
  `sops` and `qrencode` because step 2 depended on them. I did not audit every command in the
  runbook the same way — and that class of defect has now produced two corrections in a row
  (`site.sh`, then `sops | wg`), so assume more remain until each step has actually been run.
 - **The live VPN still carries no traffic**, and the earlier gaps stand unchanged: the
  iptables-vs-nftables question, `allowPrivilegeEscalation: false` versus the image's s6 init, and
  any off-LAN reachability.
@@ -0,0 +1,73 @@
 ---
 date: 2026-10-06
 branch: fix/new-peer-ignores-template-comments
 title: new-peer.sh refused the first VPN address because its duplicate check matched its own commented-out example
 ---
 ## What changed
 The operator's first real enrolment failed:
 ```
 $ ./infra/vpn-ingress/new-peer.sh keith-phone 2
 new-peer.sh: 10.90.0.2/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet.
 ```
 **Nothing was allocated.** `peers.yaml` had zero peers — the pod's own init container reported
 `wg0.conf assembled: 0 peer(s)`. The duplicate-address check grepped the WHOLE file, and
 `peers.yaml` line 66 is a **commented-out example stanza** I wrote, containing literally
 `#   AllowedIPs = 10.90.0.2/32`. So `.2` — the first address anybody would ask for, and the one my
 own runbook uses in its example — was permanently un-allocatable, with a message asserting the
 opposite of the truth.
 Both halves fixed, because either alone leaves the trap half-set:
 - **`infra/vpn-ingress/new-peer.sh`**: the check strips comment lines before matching
  (`grep -vE '^[[:space:]]*#' | grep -q …`). That is the correct predicate rather than a patch: a
  real stanza's `AllowedIPs` line is never commented, while both the template and every peer's
  `# <name> — <owner>` label always are.
 - **`platform/vpn-ingress/peers.yaml`**: the example now uses placeholders that cannot match
  anything real — `# <device-name> — <owner>, enrolled <YYYY-MM-DD>` and
  `AllowedIPs = 10.90.0.<octet>/32`. A future grep cannot be fooled by it either.
 The bug class is worth naming: **a documentation example that is byte-identical to real data will
 eventually be parsed as real data.** The same shape caused two earlier defects in this component —
 `flux-postbuild-substitution` flagged a `${...}` written inside a comment *explaining* that hazard,
 and `no-plaintext-secrets` flagged the `.sops.yaml.example` because the kustomization named it in a
 comment. Three instances now; the rule is to make examples structurally unmatchable, not merely
 commented out.
 No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `decisions/**`), so this
 merges on a green gate with no override.
 ## What I verified
 Run in the pinned toolbox against the live tree, 2026-10-06.
 - **`.2` now works**, and it went through the REAL key path: `new-peer.sh keith-phone 2` rendered a
  profile with `Address 10.90.0.2/32`, `AllowedIPs = 10.90.0.2/32` in the printed stanza, full
  tunnel, at `.cache/wg-peers/keith-phone.conf` (0600). The throwaway keypair was deleted after.
 - **This incidentally closes a gap left open in #60**: the profile could only have been rendered by
  deriving the server's public key via `sops -d --extract … | wg pubkey` from
  `wg-server-key.sops.yaml` — so that pipeline is now proven **against the operator's real key**,
  not just against a placeholder it was expected to reject.
 - **A real duplicate is still refused.** Appending an uncommented
  `AllowedIPs = 10.90.0.5/32` stanza and asking for octet 5 produced the correct refusal; the tree
  was restored afterwards and `git diff --stat` showed only the intended template edit.
 - **Zero active allocations remain**: `grep -vE '^[[:space:]]*#' peers.yaml | grep -c AllowedIPs`
  → `0`.
 - **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips (`gitea-settings`,
  conformance L2/L3). **Not green. Reported as rc 3.**
 ## What I did NOT verify
 - **No peer is enrolled by this branch.** The keypair I generated was a throwaway and was deleted;
  the operator must run the script themselves, since the device's private key must exist only on
  the device. So the end of the enrolment path — a stanza merged into `peers.yaml`, Flux rolling
  the DaemonSet, and `wg show` listing a peer — is still unexercised.
 - **The duplicate-NAME warning** (`grep -qi "^ *# *${NAME} "`) deliberately still inspects
  comments, because per-peer labels are comments. It is no longer tripped by the template, but I
  did not test it against a real enrolled peer.
 - **Nothing about client traffic.** The WAN port-forward is still not configured and there is still
  no off-LAN vantage point, so the tunnel has carried no packet.
@@ -0,0 +1,103 @@
 ---
 date: 2026-10-06
 branch: fix/peers-configmap-content-hash
 title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
 ---
 ## What changed
 The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
 2026-10-06, with the peer already on `main`:
 | observation | value |
 |---|---|
 | peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
 | GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
 | `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
 | live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
 | DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
 | DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
 | init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
 The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
 fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
 no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
 into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
 would have had **no effect at all** until something unrelated happened to restart the pods. For a
 *revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
 see a green PR, and the device would keep working.
 My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
 never true.
 **This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
 *"The generated name carries kustomize's content hash, so editing any runbook rolls the server
 automatically and a stale projection is impossible."*
 So the register becomes a **content-hashed generator**:
 - `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
  plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
  carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
  documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
  stripped.
 - **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
  ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
  `nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
  the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
  `wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
  by building the output and reading the volumes block.
 - **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
  register is now **fatal** instead of skipped. The old guard was
  `[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
  duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
  as a defect in its own right.
 - Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
  `networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
  operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
 - The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
  `generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
  merge.
 Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
 No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
 `decisions/**`), so this merges on a green gate with no override.
 ## What I verified
 Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
 - **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
  namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
  `namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
  bearing, measured both ways.
 - **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
  append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
  and reversible, so an edit necessarily changes the pod template.
 - **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
  `peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
  `main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
 - **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
  NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
  transformer did not rewrite it).
 - **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
  "append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
  real `.2` now occupies it — the first time the duplicate check has been exercised against real
  data rather than against its own example.
 - **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
  as rc 3.**
 ## What I did NOT verify
 - **That the roll actually happens on the cluster.** The whole point of the change is unproven
  until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
  `wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
 - **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
  port-forward is reportedly configured but has never been exercised by a packet I can see, and I
  have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
 - **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
  should now roll the pods and drop the peer; untested.
 - **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
  `prune: true` on the Kustomization, not observed.
@@ -0,0 +1,113 @@
 ---
 date: 2026-10-06
 branch: fix/vpn-ingress-ip-forward
 title: The VPN tunnel never came up — a hostNetwork pod cannot write net.ipv4.ip_forward, and wg-quick treats a failed PostUp as fatal
 ---
 ## What changed
 The remote-access VPN reconciled for the first time with a real server key (#60 then #62) and
 **the tunnel did not come up**. One line in my own manifest was the whole cause.
 Measured in the `wireguard` container on prod, 2026-10-06:
 ```
 [#] ip link add dev wg0 type wireguard
 [#] wg addconf wg0 /dev/fd/63
 [#] ip -4 address add 10.90.0.1/24 dev wg0
 [#] ip link set mtu 1420 up dev wg0
 [#] sysctl -q -w net.ipv4.ip_forward=1
 sysctl: permission denied on key "net.ipv4.ip_forward"
 [#] ip link delete dev wg0
 **** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! ****
 ```
 Everything up to that point worked. **The cause is not a missing capability.** With
 `hostNetwork: true` the runtime mounts `/proc/sys` **read-only** in the container — confirmed in the
 pod's own `/proc/mounts`: `proc /proc/sys proc ro,nosuid,nodev,noexec,relatime` — precisely because
 those sysctls are the *host's* globals and a container must not rewrite them. `CAP_NET_ADMIN` cannot
 lift a read-only mount, so this can never succeed from here. And `wg-quick` treats any failed
 `PostUp` as fatal, so it deleted the interface it had just finished building.
 **The line was never needed.** `node_baseline` writes `net.ipv4.ip_forward=1` to
 `/etc/sysctl.d/99-k3s-cilium.conf`, and reading the value from inside the pod returns `1`. I had
 written the header comment *correctly* — "ip_forward is ALREADY 1 on these nodes … so no pod sysctl
 is needed" — and then emitted the sysctl anyway, justified in its own comment as "re-asserted so
 this component does not silently depend on another role's sysctl file surviving". That defensive
 re-assertion is the entire outage. **The lesson is narrow and reusable: in a hostNetwork pod,
 verify host state, never write it.**
 Fix:
 - **`platform/vpn-ingress/daemonset.yaml`**: the `PostUp = sysctl …` line is gone, replaced by a
  **read** in the init container — `grep -qx 1 /proc/sys/net/ipv4/ip_forward` — which fails closed
  with a message naming the `node_baseline` role. Reading `/proc/sys` is permitted; only writing is
  not. Fail-closed still matters: a tunnel that comes up but cannot forward would satisfy every
  probe in this component and silently drop every client's traffic.
 - The header bullet that said "no pod sysctl is needed" now also says **do not attempt one**, with
  the mechanism, so the next author does not re-add it for the same reason I did.
 - The `securityContext` comment claimed NET_ADMIN was for "create the wg0 link, set the host sysctl,
  and write the netfilter rules". The middle clause was false; corrected.
 - The image's `chown: … Operation not permitted` startup run — plus its own "Permissions could not
  be set … we will not provide support for it" — is now documented as **expected and harmless**, in
  both the manifest and the runbook, so nobody chases it. Capabilities are dropped to `NET_ADMIN`
  only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs as root,
  and `wg-quick` proceeds past it (as it did here — the failure was 30 lines later).
 - **`runbooks/vpn-remote-access.md`**: a "what a healthy start looks like" block with the real log
  lines; `ip_forward is 0` added to the init-failure table; and the note that `0/1 Running` with
  `Startup probe failed: Unable to access interface: No such device` is the signature of `wg-quick`
  tearing the tunnel down — read the `[#]` line it died on.
 - **`platform/vpn-ingress/README.md`**: a table of what the first reconcile settled (below).
 No sensitive paths: `platform/vpn-ingress/**`, `runbooks/**` and `decisions/**` only. This merges on
 a green gate with no override.
 ## What I verified
 Live prod, read-only, 2026-10-06. **Four questions this component's design rested on are now
 measured rather than assumed — three went the right way, one was the defect above.**
 - **The GitOps path is proven end to end.** `Kustomization vpn-ingress` `lastAppliedRevision:
  main@sha1:cf7d5e4` equals the GitRepository artifact revision. The Secret carries
  `kustomize.toolkit.fluxcd.io/name: vpn-ingress`, so Flux delivered it.
 - **SOPS decryption works, proven by digest rather than by eye.** The value in
  `platform/vpn-ingress/wg-server-key.sops.yaml` on `main`, decrypted, is 44 bytes, is **not** the
  placeholder, and **`wg pubkey` accepts it**. Its SHA-256 equals the SHA-256 of the live Secret's
  decoded `data.privatekey` (`d7c8f8de6f252649…` both sides). No key material was printed at any
  point — lengths, booleans and digests only.
 - **The `wg pubkey` guard fired correctly, and its `ENC[` branch was never needed.** During the
  kubelet window before the updated Secret propagated, all three pods reported *"It is still the
  committed REPLACE-ME PLACEHOLDER"* — the guard distinguishing a stale/unfilled value from
  ciphertext, exactly as designed. It cleared on its own once the mount refreshed; **a transient
  `REPLACE-ME` CrashLoop for a minute or two after the key PR merges is expected, not a fault.**
 - **The kernel module reaches the pod**: `ip link add dev wg0 type wireguard` succeeded and
  `/proc/modules` lists `wireguard`. So the `node_baseline` addition works and `CAP_SYS_MODULE` is
  correctly not required.
 - **`allowPrivilegeEscalation: false` does NOT break the image's s6 init** — the container reaches
  `wg-quick` normally. The flag stays; the fear recorded in #55 is closed.
 - **The iptables backends MATCH, which was my single biggest documented doubt.** The image ships
  `iptables v1.8.13 (nf_tables)` via `/usr/sbin/xtables-nft-multi`, and Debian 13 is nf_tables. The
  nightmare case — rules accepted into a table the kernel never consults, so handshakes succeed and
  every forwarded connection hangs — cannot occur on this pairing.
 - **`net.ipv4.ip_forward` is `1`** in the host netns the pod shares, and `/proc/sys` is `ro` there.
  Both read directly from the pod.
 - **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
  environmental skips (`gitea-settings`, conformance L2/L3). **Not green. Reported as rc 3.**
 ## What I did NOT verify
 - **That the tunnel now comes up.** This fix has not reconciled. The next reconcile is the test, and
  its success criterion is specific: pods `1/1 Running`, and `wg show` reporting `interface: wg0`
  with `listening port: 51820`.
 - **That the MASQUERADE and FORWARD rules actually apply.** They are the next statements after the
  one that failed, so **they have never executed**. The backends matching (above) removes the worst
  failure mode but proves nothing about the rules themselves. First check after the pods go Ready:
  `iptables -t nat -S POSTROUTING | grep 10.90.0.0/24` on a node.
 - **Anything about client traffic.** Still zero peers enrolled, still no WAN port-forward, still no
  off-LAN vantage point. The VPN has carried no packet.
 - **Whether the transient `REPLACE-ME` window has an upper bound.** It cleared within ~2 minutes
  here; I did not determine the kubelet refresh behaviour, so a longer stall after a key change is
  uncharacterised.
 - **The `sops set` hand-off command**, still. The operator filled the key in by their own route
  (#62), so the exact quoting I documented inside `dev.sh run bash -c` remains unexercised.
@@ -0,0 +1,85 @@
 ---
 date: 2026-10-07
 branch: fix/bulk-nfs-truenas
 title: Repoint bulk NFS at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
 ---
 ## What changed
 Both ZFS pools were physically moved off the Proxmox host onto a standalone TrueNAS Community
 Edition at **192.168.2.30**, which sits on prod's own `192.168.2.0/24`. `BULK_NFS_SERVER` moves from
 `192.168.2.102` (the pfSense DNAT prod used to mount through) to `192.168.2.30`.
 - `clusters/prod/kustomization.yaml` — `BULK_NFS_SERVER` → `192.168.2.30`. The comment block that
  justified the DNAT, and the "NOT YET DECIDED / THIS PLACEHOLDER IS LOAD-BEARING" banner, are
  replaced with the measured export configuration and the recreate requirement below.
 - `clusters/base/cluster-vars.yaml` — base default `192.168.1.9` → `192.168.2.30`. That host no
  longer exists.
 - `clusters/dev/kustomization.yaml` — same; marked explicitly **inert**, since dev is decommissioned
  and its nodes are gone. It carries the real address only so nothing in the tree renders a dead one.
 - `docs/BULK-NFS-STORAGE.md` — §1 and §5 rewritten for TrueNAS (exports, service settings, the ZFS
  `sharenfs` properties that had to be cleared, and the ACL hazard); §7 rewritten now that the
  placeholder is gone; §9 retargeted from dev to prod; **new §10** documents the PV
  delete-and-recreate. §2/§3/§4/§6 are unchanged by the move.
 - `catalog/qbittorrent.yaml`, `platform/monitoring/node-pressure-dashboard.yaml` — two comment/prose
  references to `192.168.2.102` as the live export address.
 **The headline operational fact: repointing this knob is NOT a plain reconcile.** A bound PV's NFS
 source is immutable, so Flux's apply fails rather than updating and the pods keep their old, dead
 mount. The five PVs must be deleted and recreated. That sequence is §10 and is an operator action;
 it is **not** performed by this PR.
 ## What I verified
 **On the NAS (192.168.2.30), measured 2026-10-07 by the operator at my request:**
 - `zfs list` — `Storage` → `/mnt/Storage`, `Storage/plex` → `/mnt/Storage/plex`,
  `Storage/nextcloud` → `/mnt/Storage/nextcloud`. The PV paths resolve unchanged.
 - Ownership intact: `/mnt/Storage/plex`, `tv/`, `downloads/`, `movies/`,
  `downloads/complete/`, `nextcloud/nextcloud/` all `drwxr-xr-x 1000:3000`. `movies/` 1171 entries
  and `tv/` 107, against 1165 / 107 measured on Proxmox.
 - `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`), `aclinherit=passthrough`, and no
  non-trivial ACLs — mode bits still govern.
 - Three stale `sharenfs` dataset properties (`plex`, `nextcloud`, `docker-volumes`) carried over from
  Proxmox; cleared with `zfs inherit`, re-checked empty.
 - `exportfs -v`: both shares to `192.168.2.10/.11/.12` with `sec=sys,secure,root_squash,no_all_squash`,
  `rw` on plex and `ro` on nextcloud.
 - `/proc/fs/nfsd/versions` → `+3 +4 +4.1 +4.2`, so the PVs' `nfsvers=4.1` negotiates.
 - **A defect was found and fixed during setup:** the plex share listed `192.168.2.13` — an address no
  node holds — and so omitted `192.168.2.12` (k3s-prod-3). Corrected and re-verified.
 **From the cluster:**
 - Node InternalIPs are `192.168.2.10/.11/.12` (`kubectl get nodes`), confirming `.13` was wrong.
 - All five NFS PVs currently name `192.168.2.102`.
 - `192.168.2.30:2049` and `:111` accept connections; `192.168.2.102:2049` does not.
 - **`spec.nfs` immutability, by server-side dry-run** — this is the finding that shapes the change:
  `spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation`.
 - The pods report `Ready` while their mounts are dead: a `stat /media` answers from attribute cache,
  but a `readdir` blocks indefinitely (the PVs mount `hard`). Hence "scale to zero first" in §10.
 **Gate:** see the PR; run in the pinned toolbox from this worktree.
 ## What I did NOT verify
 - **Nothing has mounted 192.168.2.30.** No pod has touched the new server — the PVs still point at
  the old address and cannot be updated in place. Every §9 probe (numeric-id passthrough, the uid-1000
  write, the hardlink/EXDEV check, Plex's read-only posture) is **outstanding**, not passing. The
  equivalents passed against the old host on 2026-09-12; that is not evidence about this one.
 - **The §10 sequence is reasoned, not executed.** Retain + the renderer's pre-set `claimRef` are read
  from the manifests and from `catalog/render.py`; I have not watched a PVC rebind after a PV delete
  in this cluster.
 - **`root_squash` is a behaviour change from the Proxmox export**, which carried `no_root_squash`. The
  argument that nothing needs root on the tree (LSIO drops to 1000; the images chown only their
  Longhorn `/config`; both subPath directories already exist at 0755) is reasoned from the images and
  the measured permissions. It has not been exercised. If a mount or subPath resolution fails with
  EACCES, Maproot `root:root` reproduces the old export and is the first thing to try.
 - **Whether NFSv3 can be disabled.** Enabled Protocols is `NFSv3, NFSv4`; the PVs only need v4.1, so
  narrowing would close the rpcbind/mountd/statd surface. Not done — it is a change to make after the
  cutover is proven, not during it.
 - **Whether pfSense / the legacy stack is actually retired.** I measured only that
  `192.168.2.102:2049` stopped answering. The other legacy consumers of these datasets
  (`192.168.1.6`, `10.123.1.10`, `192.168.1.18`) were dropped from the new exports; if any is still
  live, it has lost access. Not checked.
 - **L2/L3 conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline subset
  only. Nothing in the gate exercises an NFS mount.
@@ -0,0 +1,311 @@
 ---
 date: 2026-10-17
 branch: feat/keycloak-passkeys
 title: Passkeys declared for the homelab realm — and the realm import will not apply them
 ---
 ## What changed
 `platform/keycloak/keycloak-realm-import.yaml` now declares passkey (WebAuthn Passwordless)
 login for the `homelab` realm, in two parts:
 1. **The WebAuthn Passwordless policy.** Ten `webAuthnPolicyPasswordless*` fields on the realm:
   RP ID `${BASE_DOMAIN}`, RP entity name `Homelab SSO`, signature algorithms `[ES256, RS256]`,
   `requireResidentKey: "Yes"`, `userVerificationRequirement: required`,
   `authenticatorAttachment: "not specified"`, `attestationConveyancePreference: none`,
   `acceptableAaguids: []`, `createTimeout: 300`, `avoidSameAuthenticatorRegister: true`.
   Deliberately the **passwordless** family only; the two-factor `webAuthnPolicy*` family is left
   at Keycloak's defaults because nothing in this realm uses `webauthn-authenticator`.
 2. **The browser flow.** `browser-mfa` keeps its alias and stays bound as `browserFlow`; its
   interior is restructured so a passkey is an **ALTERNATIVE** beside the password form:
   ```
   browser-mfa                            (top level, unchanged)
     browser-mfa-forms                    ALTERNATIVE
       auth-username-form                   REQUIRED      <- was auth-username-password-form
       browser-mfa-passkey-or-password      REQUIRED      <- NEW sub-flow
         webauthn-authenticator-passwordless  ALTERNATIVE <- NEW: the passkey
         browser-mfa-password-and-otp         ALTERNATIVE <- NEW sub-flow
           auth-password-form                   REQUIRED  <- was half of the combined form
           browser-mfa-conditional-otp          CONDITIONAL (unchanged, relocated intact)
   ```
 The design decisions, with their reasoning, are written into the manifest comments in this repo's
 usual density. The ones that matter:
 * **Additive, not replacing. Enrolment optional, not forced.** This IdP gates Grafana, Gitea,
  Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the \*arr stack *and* `kubectl` via the
  `kubernetes` OIDC client. A mandatory passkey flow that misfires is a total lock-out from the
  platform and the cluster API at once, leaving only the Keycloak bootstrap admin and the
  break-glass cert kubeconfig. The gain from forcing it (one human stops typing a password) does
  not price that risk.
 * **No `requiredActions` block was added, on purpose.** Declaring one would have been the obvious
  way to add `webauthn-register-passwordless`, and it would have been actively dangerous — see
  "What I verified".
 * **RP ID is `${BASE_DOMAIN}`, not `keycloak.${BASE_DOMAIN}`.** A passkey's RP ID is fixed at
  enrolment and can never be re-keyed, so it must not be bound to a name that can move. This realm
  already declares an `auth.${BASE_DOMAIN}` redirect URI, and this repo already owns a reconciler
  built because a hostname move silently failed to reach the realm. Residual risk, unavoidable:
  changing `BASE_DOMAIN` itself invalidates every enrolled passkey.
 * **A passkey login is not also asked for TOTP.** `browser-mfa-conditional-otp` hangs off the
  password branch only. The justification is `userVerificationRequirement: required` — the
  ceremony already proves possession plus a verified human, phishing-resistantly and origin-bound.
  Stacking a replayable shared secret on top of that buys friction. The manifest names the exact
  one-line edit to reverse this decision.
 * **Login becomes two screens** (username, then password) for every user, passkey or not, because
  the server must know who is logging in before it can decide whether a passkey exists. Accepted
  cost; it is the enabling change.
 * **Usernameless login deliberately not used**, although Keycloak 26.0.6 supports it — see "What I
  verified" for the measured reason it would have made every login open on a passkey prompt.
 `platform/keycloak/README.md` gains a `## Passkeys` section: the decision table, the fact that
 this does not reach the live realm, the two operator paths that would apply it, and the four-step
 browser verification nobody has run.
 No secret, key, credential or AAGUID was minted or committed. No file under `clusters/**`,
 `conformance/**`, `ci/**`, `platform/kyverno/**`, `**/rbac.yaml` or `**/*.sops.yaml` was touched,
 so this should not trip `ci/approval-policy.sh`'s sensitive-path review requirement.
 ## What I verified
 **The live realm import has already completed and will never re-run.** Measured on prod,
 2026-10-17:
 ```
 $ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
 {"conditions":[{"message":"","status":"True","type":"Done"},
               {"message":"","status":"False","type":"Started"},
               {"message":"","status":"False","type":"HasErrors"}]}
 ```
 `Done=True` / `Started=False`, exactly the one-shot state `platform/keycloak/README.md` already
 documents for client secrets and client URLs. **So the change in this PR has zero effect on how
 anyone logs in today**, and the client-secret/client-URL reconciler does not and cannot close the
 gap: its hard endpoint allow-list makes `/authentication` unreachable by design, and realm-level
 policy fields are not client fields.
 **The CRD accepts every field I used.** Measured against the live CRD (not the upstream docs):
 `kubectl --context k3s-prod get crd keycloakrealmimports.k8s.keycloak.org` lists all twelve
 `webAuthnPolicyPasswordless*` properties with the types I wrote (`RequireResidentKey`,
 `RpEntityName`, `RpId`, `UserVerificationRequirement`, `AttestationConveyancePreference`,
 `AuthenticatorAttachment` as `string`; `SignatureAlgorithms`, `AcceptableAaguids`, `ExtraOrigins`
 as `array/string`; `CreateTimeout` as `integer`; `AvoidSameAuthenticatorRegister` as `boolean`),
 plus `authenticationFlows`, `authenticatorConfig` and `requiredActions`. The `spec.realm` schema
 has **no** `x-kubernetes-preserve-unknown-fields`, so a misspelled field would have been pruned
 silently rather than rejected — which is why this was checked rather than assumed.
 **Running Keycloak is 26.0.6.** `quay.io/keycloak/keycloak:26.0.6` on `keycloak-0`/`keycloak-1`,
 operator `quay.io/keycloak/keycloak-operator:26.0.6`. Every source claim below is read from the
 `26.0.6` tag, not from memory or from docs.
 **Every provider ID is quoted from upstream source, not guessed.**
 | String I used | Where it is defined at tag 26.0.6 |
 |---|---|
 | `webauthn-authenticator-passwordless` | `WebAuthnPasswordlessAuthenticatorFactory.PROVIDER_ID`, display "WebAuthn Passwordless Authenticator" |
 | `auth-username-form` | `UsernameFormFactory.PROVIDER_ID`, display "Username Form" |
 | `auth-password-form` | `PasswordFormFactory.PROVIDER_ID`, display "Password Form" |
 | `webauthn-register-passwordless` | `WebAuthnPasswordlessRegisterFactory.PROVIDER_ID` (referenced in comments only) |
 **`requiredActions` is REPLACE, not merge — measured, and it is why I did not declare one.**
 `model/storage-private/.../DefaultExportImportManager.java` lines 327-338:
 ```java
 if (rep.getRequiredActions() != null) {
    for (RequiredActionProviderRepresentation action : rep.getRequiredActions()) { ... }
    DefaultRequiredActions.addDeleteAccountAction(newRealm);
 } else {
    DefaultRequiredActions.addActions(newRealm);
 }
 ```
 Declaring one required action therefore skips the entire default set. `RealmManager` is no
 backstop: `setupRequiredActions` is `if (realm.getRequiredActionProvidersStream().count() == 0)`,
 and the count is already non-zero by then (`RepresentationToModel.importRealm` runs at
 `RealmManager.java:585`, the backstop at `:612`). A realm built that way would have had the passkey
 action plus `delete_account` and nothing else — **no `CONFIGURE_TOTP`**, which would leave
 `auth-otp-form` demanding a TOTP that no platform-admin could ever enrol: a hard lock-out
 delivered by a change whose purpose was to *add* a login method.
 **And declaring it would have bought nothing.** `DefaultRequiredActions.addWebAuthnPasswordlessRegisterAction`
 registers `webauthn-register-passwordless` with `setEnabled(true)`, `setDefaultAction(false)`,
 priority 80, gated on `Profile.Feature.WEB_AUTHN` — which is `Type.DEFAULT` in
 `common/.../Profile.java:74`, i.e. on unless explicitly disabled. `enabled=true` /
 `defaultAction=false` *is* the additive-and-optional posture this change wants.
 **`authenticationFlows` has the opposite semantics, which is why declaring flows is safe.**
 `importAuthenticationFlows` adds the declared flows, then backfills each unbound binding
 (browser, registration, direct grant, reset credentials, client auth, first broker login, docker
 auth) from the built-in defaults individually (`DefaultExportImportManager.java` ~1320-1410).
 **The graceful degradation to password is measured from the flow engine, not assumed.** For a user
 with no passkey: `WebAuthnPasswordlessAuthenticator.requiresUser()` returns `false`, so
 `DefaultAuthenticationFlow.processSingleFlowExecutionModel` takes the `else` branch (`daf:456-462`),
 whose four conditions all hold — a user is set (from `auth-username-form`), `configuredFor` is
 false, `WebAuthnPasswordlessAuthenticatorFactory.isUserSetupAllowed()` is `false`, and
 `WebAuthnAuthenticator implements Authenticator, CredentialValidator<...>` — so it raises
 `CREDENTIAL_SETUP_REQUIRED`. The ALTERNATIVE loop (`daf:289-302`) catches exactly that, marks the
 execution `ATTEMPTED` and proceeds to the next alternative, with the upstream comment *"the user,
 since he has alternatives, should be able to go to another alternative and continue"*. Hence: no
 passkey, no passkey screen, straight to the password form.
 **Why usernameless was rejected, measured.** The `configuredFor` guard sits inside the
 `requiresUser()` branch. With `requiresUser()` false and no user yet in context, that guard is
 skipped and the engine just calls `authenticate(context)` — so a passkey execution placed before
 the username form is offered to *everyone*, including users who will never own one.
 **The policy sentinels are read from the shipped code, and one of my initial assumptions was
 wrong.** `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED = "not specified"` is the
 realm-representation sentinel — but `WebAuthnConstants.OPTION_NOT_SPECIFIED = ""` (empty string) is
 the WebAuthn-API-level one. I checked both because they disagree. The login theme's
 `webauthnRegister.js` at 26.0.6 settles the behaviour:
 ```js
 if (input.requireResidentKey !== 'not specified') {
    if (input.requireResidentKey === 'Yes') { authenticatorSelection.requireResidentKey = true; }
    else { authenticatorSelection.requireResidentKey = false; }
 }
 ...
 if (input.createTimeout !== 0) { publicKey.timeout = input.createTimeout * 1000; }
 rp: {id: input.rpId, name: input.rpEntityName}
 ```
 So `"Yes"` is a literal string compare (any other non-sentinel value means *false*), `"not
 specified"` is compared literally for four fields, and `createTimeout` is **seconds** with `0`
 meaning "send no timeout". It also shows 26.0.6 emits only the legacy `requireResidentKey` member,
 never `residentKey: "required"` — the client maps that per WebAuthn L2, but the policy cannot
 express `residentKey: "preferred"` at all.
 `[ES256, RS256]` is exactly `WebAuthnPolicy.DEFAULT_POLICY` /
 `Constants.DEFAULT_WEBAUTHN_POLICY_SIGNATURE_ALGORITHMS` — declared explicitly, not changed.
 `getWebAuthnPolicyPasswordless` was read field by field to confirm `[]` for `acceptableAaguids`
 means "no restriction" and that `RpEntityName` must be non-empty or the **update** path
 (`DefaultExportImportManager.java:842`) drops the whole policy silently.
 **The manifest parses and renders.** `yaml.safe_load` round-trips the file and the flow graph reads
 back exactly as designed (five flows, the executions and priorities above, `browserFlow:
 browser-mfa`, `requiredActions` absent). `python3 ci/render-keycloak-client-urls.py --check` →
 `client-urls.yaml is in sync`, rc 0 (no client URL changed, so no regeneration was needed).
 `yamllint -c ci/yamllint.yaml platform/keycloak/keycloak-realm-import.yaml` → rc 0; no line exceeds
 the 160-column ceiling.
 **`browser-mfa` keeps its alias deliberately.** `conformance/tests/l2-behavioral/keycloak-config-as-code`
 asserts `kubectl get keycloakrealmimport homelab -o jsonpath={.spec.realm.browserFlow} | grep -q
 'browser-mfa'`. Renaming the top-level flow would have failed that test.
 **Envsubst safety.** `platform-config`'s Flux Kustomization declares
 `postBuild.substituteFrom: cluster-vars` (`clusters/base/platform-config.yaml:26-29`), so this file
 is substituted. The only brace token I added is `${BASE_DOMAIN}`, which is declared at
 `clusters/base/cluster-vars.yaml:42` (`home.keithjackson.dev`). Every comment I wrote avoids braces
 entirely — the RP-ID discussion says `keycloak.<BASE_DOMAIN>` with angle brackets precisely so an
 unparseable token cannot fail the whole Kustomization.
 **Gate result: rc 3, zero failures, `gitea-settings` the only stage the gate counts as not-run** —
 the documented honest best local outcome. Run in the pinned toolbox from this worktree with all
 three §3 workarounds (`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main
 checkout's `.git` and the sibling spec repo with `SPEC_REPO=/home-network-k3s`):
 ```
 [ ok ] stage spec-immutability: pass      <- NOT skipped; both shim mounts worked, R-TEST-06 covered
 [ ok ] stage decisions-log: pass          <- archive frozen, 48 entries well-formed
 [ ok ] stage lint: pass
 [ ok ] stage l0: pass                     <- incl. flux-postbuild-substitution + keycloak-client-secret-reconciler
 [ ok ] stage render-golden: pass
 [ ok ] stage l1: pass
 [ ok ] stage approval-policy: pass        <- no sensitive path touched, so no non-author review needed
 [skip] stage gitea-settings: SKIPPED      <- needs SETTINGS_API_TOKEN, CI-only; expected
 [skip] stage conformance: SKIPPED         <- cluster-gated L2/L3; expected
 MERGE GATE: UNDETERMINED — Stages that did not run: gitea-settings
 ```
 No lint finding and no check failure names either file I changed. Touched requirements resolved to
 `R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06`; coverage 51/51, 0 gaps.
 **A note on how that number was obtained, because the first attempt was worthless.** My first gate
 run returned the same rc 3 and the same seven passes — against a tree that did not contain this
 change at all. The gate's `tracked_files()` reads **git**, and my edits were still uncommitted, so
 every stage gated `origin/main`. The tell was in the log and I nearly missed it: the
 touched-requirement subset reported UNDETERMINED with *"check out the feature branch you mean to
 gate (not the base branch)"*, and `decisions-log` counted **47** entries rather than 48. **A green
 gate run on an uncommitted worktree is a gate run on the base branch.** The numbers above are from
 the second run, after committing and rebasing.
 **The base moved under me mid-run.** `origin/main` advanced from `9a41a8e` to `1a0b208` (PR #70,
 oauth2-proxy OIDC startup race) while I was working. This branch is rebased onto `1a0b208`, so the
 gate above ran on the base CI will actually merge into. The rebase was clean — PR #70 touched
 `platform/oauth2-proxy/helmrelease.yaml` and its own decisions entry, disjoint from these files.
 ## What I did NOT verify
 **This change is unapplied. It changes nothing about the live realm, and I did not attempt to make
 it.** `Done=True` / `Started=False` was measured, not inferred, and I ran no mutating `kubectl`,
 no `flux reconcile`, no admin-API call and no `kcadm`. Whether the operator's eventual
 admin-console reconstruction of this flow matches what is declared here is, by construction,
 unverifiable from git.
 **I never enrolled a passkey and never pointed a browser at any of this.** Nothing here has been
 exercised by a real WebAuthn ceremony on any device. Specifically unverified:
 * **That the RP ID works.** `home.keithjackson.dev` is a registrable-domain suffix of
  `keycloak.home.keithjackson.dev`, so it *should* satisfy the browser's check — but I verified
  that by reading the WebAuthn rule, not by watching a browser accept it. A wrong RP ID fails
  **silently, in the browser, before Keycloak is involved**, with no server-side log line. If
  enrolment produces no OS prompt and no error, this is the first suspect.
 * **That `"not specified"` is the right literal for `authenticatorAttachment` in a realm import.**
  I traced it to `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED` and to the theme's string
  compare, but I have not seen a realm import accept it. Omitting the field would default to the
  same value; if an import ever rejects the literal, drop the line.
 * **That `createTimeout: 300` behaves.** Seconds-vs-milliseconds is read from the theme JS. The
  claim that 300 is long enough for the cross-device QR/hybrid flow, and that Chrome clamps rather
  than rejects it, is reasoning about browser behaviour, not a measurement.
 * **Whether a passkey user can still reach the password form.** I argued it from
  `createAuthenticationSelectionList` producing a "Try another way" option, and I noted that for a
  user holding both credentials the *first* screen is chosen by the user's own credential ordering
  rather than by my priorities. Neither was observed.
 * **The two-screen login regression for ordinary users.** `auth-username-form` +
  `auth-password-form` replacing `auth-username-password-form` is a visible change for every user
  including non-passkey family accounts, and nobody has logged in through it.
 * **That conditional OTP for platform-admins still fires on the password branch.** The sub-flow and
  its `conditional-otp-platform-admin` config are byte-identical and merely relocated one level
  deeper, but "identical YAML" is not "still enforced at runtime". This is the highest-value
  regression check and it is untested. The previous breakage of this exact step
  (`condUserRole` vs a non-existent property key) 403'd every platform-admin login.
 * **That a platform-admin authenticating by passkey is not challenged for TOTP.** That is the
  intended behaviour of the shape I wrote; it is unobserved, and it is also a deliberate
  relaxation of R-IDP-04 as previously implemented, which the operator may want to reverse.
 **Whether the realm import reconciles into an existing realm: measured as NO for the completed
 one-shot, but I did not test a re-import.** I did not delete and recreate the
 `KeycloakRealmImport`, so I cannot say what a re-run would do to the existing `homelab` realm —
 whether it would merge the new policy and flows, duplicate flow aliases, or fail. The 2026-09-06
 total SSO outage documented in `platform/keycloak/README.md` is the reason I did not experiment.
 **Treat the re-import path in the README as untested advice, not a runbook.**
 **Nothing was proved about the live realm's current state.** I did not read the realm's existing
 required actions, its existing WebAuthn policies, or its live flow executions — doing so needs the
 admin credential and I did not decrypt or use it. So I cannot confirm that the live realm's
 required-action set actually contains `webauthn-register-passwordless` today; I can only show that
 Keycloak 26.0.6's defaults register it and that this repo's import never declared any required
 actions to suppress them. **That inference has not been checked against the realm.**
 **L2/L3 conformance did not run** (no cluster from the gate container; `vars.CLUSTER_AVAILABLE` is
 unset in CI anyway), so the existing `keycloak-config-as-code` behavioural test did not re-execute
 against this change. I reasoned about its assertion by reading it.
 **No new conformance test was added.** Nothing in `conformance/` asserts the passkey policy or the
 new flow shape, so a future edit can silently remove either and no gate stage will notice. That is
 a real gap this PR leaves open; closing it would touch `conformance/**`, which is a sensitive path
 needing non-author review.
 **The quoted gate run predates this file's own gate-result paragraph.** It gated commit `207f5db`;
 the gate numbers and the rebase note above were then written into this entry, so the pushed tree
 differs from the gated one by this handoff's prose only — no manifest, no check, no generated file.
 `decisions-log` is the only stage that reads this file, and it validates structure (frontmatter,
 the three headings, non-stub bodies), which is unchanged. I did not re-run the full gate for a
 prose edit; CI will gate the pushed commit.
 **`gitea-settings` did not run locally** — it needs `SETTINGS_API_TOKEN`, which exists only in CI.
 I did not re-measure branch protection, the runner, or `required_approvals` against the Gitea API
 this session; the claims in `CLAUDE.md` §2 are taken on trust here and are explicitly flagged there
 as needing re-measurement.
@@ -0,0 +1,99 @@
 ---
 date: 2026-10-17
 branch: fix/oauth2-proxy-oidc-startup-race
 title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
 ---
 ## What changed
 Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
 discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
 serving yet, so the shared Gateway answers 503 and the proxy dies:
 ```
 [provider.go:55] Performing OIDC Discovery...
 [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
             failed to discover OIDC configuration: unexpected status "503": no healthy upstream
 ```
 It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
 replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
 retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
 is already dead), and CrashLoopBackOff backoff is not tunable per-pod.
 `platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
 polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
 accrue CrashLoopBackOff backoff and retries promptly, which is the point.
 **Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
 case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
 for a pod wedged in `Init` forever, which is not an improvement.
 ## Why this shape and not the alternatives
 * **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
  example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
  pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
  at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
  "reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
 * **No new base image.** The chart already runs an `alpine` init container for its built-in
  `initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
  `3.22` (verified to exist upstream) rather than the chart's own `latest`.
 * **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
  this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
  the chart already has a hook for.
 * **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
  more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
  oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
  the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
  the operator out of all of them, so the reversible, additive option wins.
 The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
 with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
 and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
 silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
 the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.
 ## What I verified
 * **The failure is measured, not theorised.** The log above is from the live pod
  (`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
  with `--previous` after the cold boot. The surviving replica's log shows the same startup
  succeeding once Keycloak was up, which is what identifies this as a race rather than a config
  fault.
 * **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
  clean and both replicas have been healthy since.
 * **The chart key is real**, at the pinned version range: `helm search repo` resolves
  `>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
  `extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
 * **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
  assumed from memory.
 * The edited HelmRelease parses as YAML.
 ## What I did NOT verify
 * **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
  seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
  replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
  seven apps, so **do it when you can watch it**, not unattended.
 * **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
  oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
  from the measured failure, not demonstrated against it.
 * **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
  `ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
  failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
  degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
  anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
  init container's log on first apply**: it should print "Keycloak discovery reachable" within
  seconds on a warm cluster, not the give-up message.
 * **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
  re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
  good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
  (`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
  Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
  this should hold — unverified.
 * **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
  not a measured bound.
 * **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
  subset only.
 * I did not re-measure branch protection or runner state (CLAUDE.md §2b).
@@ -1,61 +1,43 @@
 # Bulk NFS storage — the media library and the legacy Nextcloud archive
-The operator-facing companion to the `class: bulk` storage entries in `catalog/{plex,sonarr,radarr,nextcloud}.yaml`.
+The operator-facing companion to the `class: bulk` storage entries in
-Everything here was **measured** on the storage host on 2026-09-07, over read-only SSH; nothing is
+`catalog/{plex,sonarr,radarr,qbittorrent,nextcloud}.yaml`.
-assumed. It resolves the long-standing **OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
+
 **Rewritten 2026-10-07 for the move off Proxmox onto a standalone TrueNAS at 192.168.2.30.** The
 layout (§2), the mount design (§3, §4) and the uid model (§6) are unchanged by the move and were
 measured on the old host on 2026-09-07. Everything about the storage host itself — §1, §5, §10 —
 was re-measured on the NAS on 2026-10-07. Nothing here is assumed. It resolves the long-standing
 **OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
 case that actually exists here — attaching datasets that already hold data — and leaves dynamic bulk
 provisioning unimplemented, because nothing needs it.
 ---
-## 1. The storage host, and why prod cannot reach it
+## 1. The storage host
-The bulk data lives on the **Proxmox host's ZFS pool**. That host has exactly one interface:
+The bulk data lives on a **standalone TrueNAS** (Community Edition) at **192.168.2.30** — the same
 `192.168.2.0/24` as prod's nodes (`192.168.2.10/.11/.12`), so it is reached directly, with no NAT
 and no routing decision.
-```
+**This section used to be the hard part of this page.** The datasets were on the Proxmox host, whose
-vmbr0  192.168.1.9/24        default via 192.168.1.1   (pfSense LAN)
+only interface was `vmbr0 192.168.1.9/24` on pfSense's **LAN**, while prod sits on pfSense's **WAN**.
-```
+tcp/2049 did not traverse that, so prod mounted through a DNAT at `192.168.2.102`, and which address
-
+belonged in `BULK_NFS_SERVER` depended on an undecided operator choice. Both pools were physically
-| | nodes | reaches 192.168.1.9? |
+moved on 2026-10-07 (`docs/plans/proxmox-to-n5-air-migration.md`, Phase C/D) and all of that is
-|---|---|---|
+simply gone. Measured 2026-10-07: **192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and
-| **dev** | 192.168.1.40 / .41 / .42 | **Yes** — same L2 segment, and it already holds an NFS export from this host (`/mnt/Storage/k8s-volumes`, in `/etc/exports`) |
+:111 do.**
 | **prod** | 192.168.2.10 / .11 / .12 | **No** — 192.168.2.0/24 is pfSense's **WAN**. Ports 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. 2049 and 111 do not. |
 Opening the storage path for prod is a **pending operator decision**. Its two candidate outcomes give
 different values for the `BULK_NFS_SERVER` cluster-var:
 | outcome | `BULK_NFS_SERVER` for prod |
 |---|---|
 | port-forward `192.168.2.102:2049` → `192.168.1.9:2049` | `192.168.2.102` |
 | a WAN→LAN route to 192.168.1.0/24 | `192.168.1.9` |
 Until it lands, `clusters/prod` holds `TBD-PROD-BULK-NFS-SERVER`. **See §7 for why that placeholder
 behaves differently from the repo's other ones.**
 ### It only needs ONE port
 The PVs mount `nfsvers=4.1`. **NFSv4 carries mount and lock in-protocol**, so the only thing that
 must be reachable is **tcp/2049** — no rpcbind/111, no mountd, no statd. NFSv3 would have needed
-four ports, three of them on ephemeral numbers, which is a materially worse firewall story. The
+four ports, three of them on ephemeral numbers. The NAS advertises `+3 +4 +4.1 +4.2`
-server advertises `+3 +4 +4.1 +4.2` (`/proc/fs/nfsd/versions`), so v4.1 is available and is the
+(`/proc/fs/nfsd/versions`, measured 2026-10-07), so 4.1 is available and is the conservative floor.
 conservative floor.
-### The export's client list is the SAME under both outcomes — measured, not assumed
+**NFSv4 is not on by default and its absence does not look like a storage problem.** TrueNAS ships
-
+NFSv3-only until NFSv4 is added to *Services → NFS → Enabled Protocols*. Without it the mount never
-**pfSense DNATs the existing port-forward WITHOUT source NAT.** Connecting from this workstation
+completes and the pods sit in `ContainerCreating` — no permission error, nothing in the app logs,
-(192.168.2.132) to `192.168.2.102:22`, the storage host logged the peer as `192.168.2.132`:
+just no volume.
 ```
 SSH_CONNECTION=192.168.2.132 39768 192.168.1.9 22
 ss -tn  ->  192.168.1.9:22   192.168.2.132:39768
 ```
 The original WAN-side source address survives the translation (return traffic finds its way back via
 the host's default route, 192.168.1.1). So under **either** outcome the NFS server will see prod's
 **real** node addresses — 192.168.2.10/.11/.12 — and the `sharenfs` change in §5 is the same either
 way. **It does not have to wait for the routing decision.**
 ---
 ## 2. The layout, as found
@@ -107,16 +89,18 @@ why it is mounted read-only; see §6.
 | plex | `/mnt/Storage/plex` | `/media` | — | **ro** | ReadOnlyMany |
 | sonarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
 | radarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
 | qbittorrent | `/mnt/Storage/plex` | `/downloads` | `downloads` | rw | ReadWriteMany |
 | nextcloud | `/mnt/Storage/nextcloud` | `/mnt/legacy-nextcloud` | `nextcloud` | **ro** | ReadOnlyMany |
 In-app paths that follow: Plex libraries at `/media/movies`, `/media/tv`, `/media/photos`; Sonarr
 Root Folder `/media/tv`; Radarr Root Folder `/media/movies`; both importing from
 `/media/downloads/complete/{tv-sonarr,radarr}`.
-**All three media consumers use the identical mountPath on purpose.** When qbittorrent is eventually
+**The three library consumers use the identical mountPath on purpose**, and qbittorrent has since
-given the same export (it is not in this change), the arrs and the download client will already agree
+been given the same export (narrowed by `subPath: downloads`, presented at `/downloads` to match its
-on absolute paths, which is the second precondition for hardlinked imports — otherwise every import
+existing `Session\DefaultSavePath`). The arrs and the download client therefore agree on absolute
-needs a Remote Path Mapping and quietly falls back to copying when one is missing.
+paths, which is the second precondition for hardlinked imports — otherwise every import needs a
 Remote Path Mapping and quietly falls back to copying when one is missing.
 **`/media` and not `/data`, and that is not cosmetic.** The arrs' Longhorn config PVC renders at the
 renderer default `/data/config`. A media mount at `/data` would be mounted **over** it and mask the
@@ -147,62 +131,123 @@ Narrowing that would cost hardlinks, which is the worse trade.
 ---
-## 5. The export change the operator must apply — STRICTLY ADDITIVE
+## 5. The exports, as configured on TrueNAS — and the settings that decide permissions
-**Do not apply this from the repo.** It is an operator step on the storage host, recorded here so the
+**Do not apply this from the repo.** It is operator configuration on the NAS, recorded here so the
-exact value is reviewable. The existing consumers **must keep working**: 192.168.1.6, 10.123.1.10 and
+exact values are reviewable. Everything below was **measured on 192.168.2.30 on 2026-10-07**, after
-192.168.1.18 on the media tree, and 192.168.1.6 and 192.168.1.5 on the Nextcloud dataset, are the
+the move.
 live non-k3s prod stack.
-Current values (`zfs get sharenfs Storage`):
+### The exports
 ```
-Storage/plex       rw=@192.168.1.6:@10.123.1.10:@192.168.1.18,no_subtree_check,no_root_squash
+/mnt/Storage/plex       192.168.2.10 192.168.2.11 192.168.2.12
-Storage/nextcloud  [email protected]:@192.168.1.5,no_subtree_check,no_root_squash
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)
 /mnt/Storage/nextcloud  192.168.2.10 192.168.2.11 192.168.2.12
                        (sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,root_squash,no_all_squash)
 ```
-### `Storage/plex` — append the six k3s node addresses to the existing `rw=` list
+Per-node addresses rather than `@192.168.2.0/24`, following the precedent of the old host. A `/24`
 would work and would survive adding a node; it also grants the whole subnet.
 **All three nodes must appear on BOTH shares.** Any media pod can be scheduled on any node, so a
 missing node is a latent failure, not an immediate one: it surfaces the first time a pod reschedules
 there, as `access denied by server while mounting`, with the cause weeks stale. One was caught
 exactly this way during setup — the plex share listed `192.168.2.13`, an address no node holds, and
 so was missing `192.168.2.12` (k3s-prod-3). Nothing was failing at the time.
 ### The option flags ARE the permission contract
 | flag | why it matters here |
 |---|---|
 | **`no_all_squash`** | The one the whole model rests on. Client uids pass through untouched, so the arrs' `PUID=1000` arrives as the owner of a `0755 1000:3000` tree. In the UI this is **Mapall User/Group left EMPTY** — setting either squashes every client to one identity and erases the reader/writer distinction entirely. |
 | **`sec=sys`** | AUTH_SYS. Numeric uid/gid, not Kerberos principals. §6 is only true because of this. |
 | **`root_squash`** | Maproot left empty. The Proxmox export carried `no_root_squash`; **nothing here needs it** — every write to the tree happens after the LSIO drop to uid 1000, and the images only chown their Longhorn `/config`. This is a deliberate tightening over the old export, not parity. |
 | **`rw` / `ro`** | The archive's read-only posture is **server-enforced**, as it was before. The media tree cannot get the same treatment — see below. |
 | **`secure`** | Reserved source ports required; "Allow non-root mount" off. The kubelet mounts as root from a reserved port and does not need the exception. |
 **Plex's read-only posture is still a CLIENT-side guarantee.** Plex is ro and the arrs are rw, but
 all four can land on any node, so the server sees the same three source addresses for all of them
 and cannot distinguish the reader from the writers. `spec.nfs.readOnly: true` on `plex-media` plus
 the volumeMount flag is the whole enforcement. Unchanged by the move; an honest limitation, not an
 oversight. If it ever needs to be server-enforced, the lever is a dedicated node pool for Plex.
 ### Service settings that are NOT obvious from the export list
 - **Enabled Protocols must include NFSv4** (above).
 - **Require Kerberos for NFSv4: off.** The numeric AUTH_SYS pass-through is the model.
 - **Manage Groups Server-side: off.** This is `rpc.mountd --manage-gids`: the server *discards the
  client's supplied group list* and looks that uid's groups up in TrueNAS's own user database
  instead. With no local uid 1000 defined, the arrs' `PGID=3000` would simply vanish. Writes would
  survive anyway — uid 1000 **owns** the tree, so the owner bits grant access without the group —
  but it is a free way to lose group semantics.
 - **NFSv4 DNS Domain: blank.** It only feeds `idmapd.conf`, which numeric mapping bypasses.
 - **Bind IP 192.168.2.30** means loopback does not answer: check with `showmount -e 192.168.2.30`,
  not `localhost`.
 There is **no** "NFSv3 ownership model for NFSv4" control on current TrueNAS — it was an older
 TrueNAS/CORE option, and no replacement is needed: on a Linux client against a Linux server over
 AUTH_SYS, both the `nfs` and `nfsd` modules default to `nfs4_disable_idmapping=Y`, so NFSv4 carries
 raw numeric ids rather than `user@domain` principals. §9's `ls -n` probe is how you confirm it.
 ### The ZFS `sharenfs` properties must be CLEARED, not edited
 The pools arrived with the old host's exports still set as dataset properties:
 ```
 Storage/plex              [email protected]:@10.123.1.10:@192.168.1.18:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
 Storage/nextcloud         [email protected]:@192.168.1.5,[email protected]:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
 Storage/docker-volumes    [email protected]:@192.168.1.5:@192.168.1.10:@192.168.1.18,no_subtree_check,no_root_squash
 ```
 TrueNAS generates `/etc/exports` from its own middleware database and will fight a property it did
 not set — the failure mode is two differently-configured exports of one path, with no indication
 which is in force. Cleared 2026-10-07:
 ```sh
-zfs set sharenfs='[email protected]:@10.123.1.10:@192.168.1.18:@192.168.1.40:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/plex
+zfs inherit sharenfs Storage/plex
 zfs inherit sharenfs Storage/nextcloud
 zfs inherit sharenfs Storage/docker-volumes
 ```
-Every pre-existing host is preserved verbatim and no option changes; only clients are added.
+`Storage` is `off`, so each resolves to `off`.
-### `Storage/nextcloud` — add the k3s nodes to a **`ro=`** list, not to `rw=`
+### Do NOT touch the dataset's permissions or ACLs
-```sh
+The tree's ownership **is** the access story (§6), and TrueNAS offers two ways to destroy it
-zfs set sharenfs='[email protected]:@192.168.1.5,[email protected]:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/nextcloud
+irreversibly across 8.33 T.
 ```
-This is the one place read-only can be enforced **server-side**, and it is worth taking: only
+`Storage/plex` carries `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`) and
-Nextcloud consumes this dataset from k3s, so the export itself can refuse writes. Even a
+`aclinherit=passthrough` (inherited). Measured 2026-10-07: the directories carry **trivial ACLs
-misconfigured client cannot then damage the archive.
+only** — `nfs4xdr_getfacl` shows just `owner@`/`group@`/`everyone@` matching the mode — so `0755
 1000:3000` governs exactly as it did under Proxmox.
-**The media tree cannot get the same treatment.** Plex is read-only and the arrs are read-write, but
+- **Stay out of the dataset's Edit Permissions / Edit ACL screen.** On an `acltype=nfsv4` dataset
-all three may be scheduled on any node, so the server sees the same source addresses for all of
+  that is a full ACL editor, and its presets write *real* NFSv4 ACLs, recursively if permitted.
-them and cannot distinguish the reader from the writers. **Plex's read-only posture is therefore a
+  There is no undo.
-client-side guarantee** (`spec.nfs.readOnly: true` on `plex-media`, plus the volumeMount flag) — an
+- `aclinherit=passthrough` means a single ACL written at `/mnt/Storage/plex` would be **inherited by
-honest limitation, not an oversight. If it ever needs to be server-enforced, the lever is a
+  everything created under it afterwards** — the blast radius is the whole future library, not one
-dedicated node pool for Plex, which is a much larger change.
+  directory.
 - Note GNU `ls`'s `+` marker is **not** a reliable ACL indicator here: it looks for a POSIX ACL
  xattr, and this dataset stores NFSv4 ACLs elsewhere. Use `nfs4xdr_getfacl`.
-Per-node addresses rather than `@192.168.2.0/24` follows the precedent already on this host
+`xattr=on` rather than `sa` is a performance detail, not a permissions one, and changing it would
-(`/etc/exports` lists the dev nodes individually, as `/32`). A `/24` would work and would survive
+not affect existing files.
 adding a node; it also grants the whole subnet. Per-node is the tighter default.
 ### Verifying, without changing anything
 ```sh
-exportfs -v                  # the new client lines appear, existing ones unchanged
+cat /proc/fs/nfsd/versions          # +4.1 must be present
-zfs get sharenfs Storage/plex Storage/nextcloud
+exportfs -v                         # the two shares, three nodes each, flags as tabulated above
 showmount -e 192.168.2.30           # NOT localhost -- the service is bound to one address
 zfs get -r -o name,property,value,source sharenfs Storage    # every row off or -
 ```
 ---
 ## 6. uid / gid — the failure this design exists to avoid
-The export uses `sec=sys` and `no_root_squash`, so **the client's numeric uid is passed through
+The export uses `sec=sys` and `no_all_squash`, so **the client's numeric uid is passed through
-verbatim** and the server's on-disk ownership is the entire access story. The media tree is
+verbatim** and the server's on-disk ownership is the entire access story. (It no longer carries
 `no_root_squash`, which the Proxmox export did — see §5. Nothing here needs it, and the uid that
 matters is 1000, not 0.) The media tree is
 `0755 1000:3000` throughout (some directories `1000:1000`), so **only uid 1000 can write it**.
 **`fsGroup` cannot fix this, and reaching for it is how this gets misdiagnosed.** The in-tree NFS
@@ -233,24 +278,27 @@ Nextcloud needs no uid work at all, which is one more reason its mount is read-o
 ---
-## 7. The prod placeholder does NOT fail at apply time — read this before merging
+## 7. Why this knob is unvalidated at create AND immutable at update
 **The `TBD-PROD-BULK-NFS-SERVER` placeholder is gone** (2026-10-07) — prod now carries a real
 address, because the NAS is on prod's own subnet and there is no longer a decision to defer. The
 hazard that made it load-bearing is worth keeping on the page, because it is the half that explains
 why §10 exists.
 Every other `TBD-PROD-*` in `clusters/prod/kustomization.yaml` is rejected loudly by CRD validation
-(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one is not.**
+(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one was not.**
-A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepts
+A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepted
-`TBD-PROD-BULK-NFS-SERVER`, the PV is created, and the PVC **binds** to it — binding is pure
+`TBD-PROD-BULK-NFS-SERVER`, the PV was created, and the PVC **bound** to it — binding is pure
-control-plane and never touches the server. The failure appears only when a pod mounts:
+control-plane and never touches the server. The failure appeared only when a pod mounted:
 ```
 MountVolume.SetUp failed ... Failed to resolve server TBD-PROD-BULK-NFS-SERVER
 ```
-so **plex, sonarr, radarr and nextcloud would sit in ContainerCreating on prod.**
+**The same field is immutable on UPDATE** (§10), which is the other half of the same awkwardness: a
-
+wrong value is accepted silently at create, and then cannot be corrected in place. Together they are
-**Consequently: set `BULK_NFS_SERVER` in `clusters/prod/kustomization.yaml` BEFORE prod reconciles a
+why repointing this knob is a delete-and-recreate rather than a reconcile.
 revision that carries these bulk volumes.** Dev renders a working value and is where these manifests
 are meant to be proven first.
 ### The static guard this convention leans on is missing from `main` — but the gate now refuses the value anyway
@@ -287,10 +335,9 @@ Without the arm, HALF A is what holds the line, by design.
   EPHEMERAL layer"). Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it
   means those apps start from an empty config volume, so it belongs in its own change with the
   operator present. **This change does not make it worse** — that is why the media mount is `/media`.
-2. **qbittorrent has no bulk mount.** Today the arrs' import source is written by the **legacy
+2. **RESOLVED — qbittorrent now has the bulk mount** (`subPath: downloads` at `/downloads`), so the
-   non-k3s** download client, which is coherent while the legacy stack stays live. Giving the k3s
+   import source is written by the k3s download client rather than the legacy non-k3s one. See
-   qbittorrent the same export at the same `/media` path is the natural follow-up, and the paths were
+   `catalog/qbittorrent.yaml`.
   chosen so it needs no Remote Path Mapping.
 3. **`/mnt/Storage/plex/downloads/downloads` is a symlink to `/shared/downloads`**, which is outside
   the export. Over NFS it is a dangling symlink on every client. Harmless, but expect Sonarr to log
   it while scanning `downloads/`.
@@ -300,10 +347,10 @@ Without the arm, HALF A is what holds the line, by design.
 5. **`nconnect` was not set.** It would likely help throughput on the media tree, but it is a tuning
   change that should be measured, not guessed, and it changes client behaviour under server
   failure.
-6. **Nothing was verified against a live cluster.** No `kubectl` context is usable from this
+6. **Every claim about the NAS in §5 is measured (2026-10-07); the post-move MOUNT is not.** The PVs
-   workstation (browser-OIDC only), and this change is manifest-authoring under an explicit
+   still have to be recreated (§10) before any pod mounts 192.168.2.30, so the §9 probes are the
-   do-not-apply constraint. Every claim about the *host* is measured; every claim about *binding and
+   outstanding verification, not a formality. The pre-move equivalents did pass against the old host
-   mounting* is reasoned from the manifests.
+   on 2026-09-12.
 7. **The bulk-tier requirement ID is deliberately not typed on any non-`.md` line of this change.**
   `ci/touched-requirements.sh` attributes requirements from R-IDs appearing in added lines, and the
   in-cluster conformance test for the bulk tier is still a gap, so typing it would select a
@@ -312,36 +359,95 @@ Without the arm, HALF A is what holds the line, by design.
 ---
-## 9. Validating on dev, once the export is extended
+## 9. Validating, after the PVs are recreated
-Dev can prove all of this before prod's network path exists. Read-only checks first:
+These run against **prod**; dev is decommissioned. They are the outstanding verification for the
 move — nothing below has been run against 192.168.2.30. Read-only checks first:
 ```sh
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pv
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pv -o custom-columns=\
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pvc -A | grep -E 'media|legacy-archive'
+NAME:.metadata.name,SERVER:.spec.nfs.server,PATH:.spec.nfs.path,STATUS:.status.phase'
 infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr get pod -o wide
 ```
-Expect **four** PVs — `plex-media`, `sonarr-media`, `radarr-media`, `nextcloud-legacy-archive` (three
+Expect **five** NFS PVs — `plex-media`, `sonarr-media`, `radarr-media`, `qbittorrent-downloads`,
-of them naming the *same* export, one per namespace, which is the cross-namespace model) — each
+`nextcloud-legacy-archive` (four of them naming the *same* export, one per namespace, which is the
-`Bound` to the same-named PVC, with `RECLAIM POLICY Retain` and an empty `STORAGECLASS`.
+cross-namespace model) — each `Bound`, `RECLAIM POLICY Retain`, empty `STORAGECLASS`, and
 **`SERVER 192.168.2.30`**. A PV still showing `192.168.2.102` means the recreate in §10 did not
 happen; Flux will NOT have updated it in place.
-Then the two things that actually matter, from inside a pod:
+Then the four things that actually matter, from inside a pod:
 ```sh
-# (a) the writer can write — this is the uid check, and it is the one that fails in practice
+# (a) numeric ids survived the wire -- the idmapping check, new since the TrueNAS move
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- sh -c 'ls -n /media | head -3'
-  sh -c 'id; touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+#     expect uid 1000 / gid 3000 on the entries. 65534 / nobody means ids are being mapped, not
 #     passed through, and every import will fail EACCES while the pod still reports Ready.
-# (b) hardlinks work BETWEEN the library and the download path — the root-mount check
+# (b) the writer can write -- the uid check, and the one that fails in practice
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
  sh -c 'touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
 # (c) hardlinks work BETWEEN the library and the download path -- the root-mount check
 kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
  sh -c 'set -e; d=/media/downloads/complete/tv-sonarr/.probe; t=/media/tv/.probe-link;
         echo x > "$d"; ln "$d" "$t"; ls -i "$d" "$t"; rm -f "$d" "$t"; echo HARDLINK-OK'
-# (c) the reader really is read-only
+# (d) the reader really is read-only
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n plex exec deploy/plex -- \
+kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c 'touch /media/.probe 2>&1 | head -1'
-  sh -c 'touch /media/.probe 2>&1 | head -1'      # expect "Read-only file system"
+#     expect "Read-only file system"
 ```
-`(b)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
+`(c)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
 prints `EXDEV`, something has split the single mount back into two.
 Reading the failures together is what localises the fault: **(b) EACCES with (a) showing `nobody`**
 is idmapping (§5 service settings). **(b) EACCES with (a) showing `1000`** is on-disk ownership —
 stop and check `ls -ldn` on the NAS before changing anything else.
 ---
 ## 10. Repointing BULK_NFS_SERVER means DELETING AND RECREATING THE PVs
 **A bound PersistentVolume's NFS source is immutable.** Measured by server-side dry-run on
 2026-10-07:
 ```
 $ kubectl patch pv sonarr-media --type=json --dry-run=server \
    -p '[{"op":"replace","path":"/spec/nfs/server","value":"192.168.2.30"}]'
 The PersistentVolume "sonarr-media" is invalid: spec.persistentvolumesource: Forbidden:
  spec.persistentvolumesource is immutable after creation
 ```
 Note the pairing with §7: `spec.nfs.server` is **unvalidated at create** (any string is accepted and
 the PVC binds) and **immutable at update**. So editing `BULK_NFS_SERVER` alone does not fail loudly
 and does not work either — Flux's apply errors on the five PVs and the pods keep their previous,
 now-dead mount.
 This is safe to do because `persistentVolumeReclaimPolicy: Retain` means deleting a PV never touches
 the data, and `catalog/render.py` emits a **pre-set `claimRef`** on each one, so a recreated PV
 rebinds its own PVC with no manual step.
 **Operator sequence** (mutating `kubectl` is an operator action, not an agent action):
 ```sh
 # 1. Stop the consumers. This also releases the hung hard-mounts against the old server:
 #    the pods report Ready, but any real I/O on /media blocks indefinitely once 2049 stops
 #    answering, because the PVs mount `hard`.
 for ns in plex sonarr radarr qbittorrent nextcloud; do
  kubectl --context k3s-prod -n $ns scale deploy --all --replicas=0
 done
 # 2. Delete the five PVs. Retain + no live claim means nothing is reclaimed or erased.
 kubectl --context k3s-prod delete pv plex-media sonarr-media radarr-media \
  qbittorrent-downloads nextcloud-legacy-archive
 # 3. Let Flux reconcile the revision carrying BULK_NFS_SERVER=192.168.2.30, which recreates them.
 #    Confirm .status.lastAppliedRevision matches the GitRepository artifact before continuing.
 # 4. Bring the consumers back, then run every probe in §9.
 ```
 The PVCs pass through `Lost` between steps 2 and 3; that is expected and resolves when the PV with
 the matching `claimRef` reappears. If a PVC is still `Lost` after Flux reconciles, the PV name or
 its `claimRef` namespace does not match — do **not** delete the PVC to "fix" it.
@@ -32,9 +32,23 @@ ARG TARGETARCH=amd64
 # Distro packages: libvirt client libs (harmless disk-image utilities), ansible, ssh,
 # disk-image + cloud-init seed tooling, python3-pip (for the pip-installed test tools),
 # diffutils (provides `diff`, used by L1 golden render checks), and common utilities.
 #
 # wireguard-tools + qrencode serve the REMOTE-ACCESS VPN (platform/vpn-ingress):
 #   * wireguard-tools gives `wg genkey` / `wg pubkey`, needed to mint the server key and to derive
 #     every client profile's `[Peer] PublicKey` (infra/vpn-ingress/new-peer.sh).
 #   * qrencode renders a client profile as a QR code, which is how a PHONE gets enrolled.
 # They are here rather than on the operator's host because this image is THE toolchain
 # (CLAUDE.md §5; README "Don't install the toolchain on the host"). That rule was briefly broken
 # by a runbook that told the operator to run `wg` on the host and `sops` in here, and then split
 # every step across the boundary to work around it — a procedure nobody should have to follow, and
 # one that cannot be reproduced in CI. Both tools in one image means `sops -d … | wg pubkey` is
 # just a pipeline again.
 # Unpinned on purpose, like every other dnf package here: the Fedora base is what is pinned
 # (FEDORA_VERSION), and these are not gate tools whose version changes what green means.
 RUN dnf -y install --setopt=install_weak_deps=False \
      libvirt-client qemu-img cloud-utils genisoimage openssh-clients \
      ansible-core python3-libvirt python3-pip diffutils \
      wireguard-tools qrencode \
      git make jq unzip tar gzip xz which ca-certificates curl \
 && dnf clean all
@@ -14,6 +14,7 @@ baseline, and the *same* binaries CI uses.
 | Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
 | GitOps / cluster | `flux`, `kubectl`, `helm` |
 | Secrets | `sops`, `age` + `age-keygen` |
 | VPN (`platform/vpn-ingress`) | `wg` (wireguard-tools) — mint the server key, derive client public keys; `qrencode` — render a client profile as a phone-scannable QR |
 | **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
 | Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |
@@ -2,10 +2,24 @@
 # =================================================================================================
 # new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
 #
-# Generates a WireGuard keypair ON THIS MACHINE, renders the device's client profile, optionally
+# Generates a WireGuard keypair, renders the device's client profile, prints a QR code for a phone,
-# prints a QR code for a phone, and prints the one `[Peer]` stanza to paste into
+# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.conf. The device's
-# platform/vpn-ingress/peers.yaml. The private half NEVER leaves this machine: it is not sent to the
+# private half is written to ONE 0600 file and nowhere else: not sent to the cluster, not committed,
-# cluster, not committed, and not written to any log.
+# never logged.
 #
 # RUN IT IN THE DEV CONTAINER — that is where the toolchain lives (CLAUDE.md §5):
 #
 #     ./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2'
 #
 # It needs `wg` (wireguard-tools), `qrencode` and `sops`, all three baked into the image. It will
 # run on a host that happens to have them too, but the container is the supported way and the only
 # one that is reproducible.
 #
 # THE SERVER'S PUBLIC KEY IS DERIVED FROM THE ENCRYPTED FILE, not from a copy you keep lying around:
 # `sops -d` the private half out of platform/vpn-ingress/wg-server-key.sops.yaml and pipe it
 # straight into `wg pubkey`. The plaintext never touches disk and never reaches a terminal. (This
 # is why both tools being in ONE image matters — see the Containerfile comment.) Override with
 # WG_SERVER_PUBKEY or WG_SERVER_KEY_FILE if you are working off-tree.
 #
 # WHY A SCRIPT AND NOT A RUNBOOK STEP. Three of these fields are easy to get wrong in ways that are
 # either silently insecure or silently broken, and all three are mechanical:
@@ -25,24 +39,30 @@
 #   <last-octet>    2-254, the device's address inside WG_CLIENT_CIDR. Must be unused.
 #   --split         split tunnel: route ONLY the LAN and the tunnel subnet. Default is a FULL
 #                   tunnel (0.0.0.0/0) so the device also egresses the house WAN.
-#   --out DIR       where to write the profile (default: a 0700 dir under $TMPDIR).
+#   --out DIR       where to write the profile. Default: `.cache/wg-peers/` in the repo, which is
 #                   GITIGNORED (`.gitignore` `.cache/`) and, unlike /tmp inside the container,
 #                   survives the container exiting — the repo is its only persistent read-write
 #                   mount, so a profile written anywhere else is gone before you can move it to a
 #                   desktop. The script refuses any destination git does not ignore.
 #
 # ENVIRONMENT
-#   WG_SERVER_KEY_FILE  server PRIVATE key (default ~/.config/wireguard-home/server.key); its public
+#   WG_SERVER_PUBKEY    the server's public key, if you want to skip the SOPS decrypt entirely.
-#                       half is derived from it. Alternatively set WG_SERVER_PUBKEY directly.
+#   WG_SERVER_KEY_FILE  a local copy of the server PRIVATE key to derive the public half from,
 #                       instead of reading the SOPS file.
 #   WG_SERVER_SOPS_FILE default platform/vpn-ingress/wg-server-key.sops.yaml
 #   WG_ENDPOINT_HOST    default wan.home.keithjackson.dev
 #   WG_CLIENT_CIDR      default 10.90.0.0/24   (must match clusters/base/cluster-vars.yaml)
 #   WG_LAN_CIDR         default 192.168.2.0/24 (prod LAN_CIDR)
 #   WG_DNS              default 192.168.2.23   (prod PIHOLE_DNS_VIP)
 #
-# REQUIRES: wireguard-tools (`wg`). `qrencode` is optional and only used for the QR code.
+# REQUIRES: `wg`, `qrencode`, `sops` — all three in the dev container.
 # =================================================================================================
 set -euo pipefail
 die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; }
 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
+PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
 # --- defaults, each mirroring a value that is DECLARED elsewhere in this repo -------------------
 # These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to
@@ -51,7 +71,7 @@ PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
 : "${WG_CLIENT_CIDR:=10.90.0.0/24}"                 # clusters/base/cluster-vars.yaml
 : "${WG_LAN_CIDR:=192.168.2.0/24}"                  # clusters/prod LAN_CIDR
 : "${WG_DNS:=192.168.2.23}"                         # clusters/prod PIHOLE_DNS_VIP
-: "${WG_SERVER_KEY_FILE:=${HOME}/.config/wireguard-home/server.key}"
+: "${WG_SERVER_SOPS_FILE:=${REPO_ROOT}/platform/vpn-ingress/wg-server-key.sops.yaml}"
 # wan.<base-domain> is an UNPROXIED A record that platform/externaldns/ddns.yaml keeps pointed at
 # the current WAN IP (checked every 5 min). Use it, never a literal address: this house has a
 # DYNAMIC WAN IP, and a profile pinned to today's address stops working without warning and with no
@@ -83,14 +103,28 @@ printf '%s' "$NAME" | grep -Eq '^[a-z0-9][a-z0-9-]{0,30}$' \
 printf '%s' "$OCTET" | grep -Eq '^[0-9]+$' || die "octet must be a number: got '$OCTET'"
 [ "$OCTET" -ge 2 ] && [ "$OCTET" -le 254 ] || die "octet must be 2-254 (.1 is the server): got '$OCTET'"
-command -v wg >/dev/null || die "wireguard-tools not installed ('wg' not on PATH). Debian/Ubuntu: apt install wireguard-tools"
+command -v wg >/dev/null || die "'wg' not on PATH. Run this inside the dev container:
  ./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh $NAME $OCTET'"
 # --- the server's public key --------------------------------------------------------------------
 # Order: an explicit pubkey, then a local private-key file, then the SOPS file in the tree. The
 # last is the normal path and needs no local copy of anything.
 if [ -z "${WG_SERVER_PUBKEY:-}" ]; then
-  [ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read server key '$WG_SERVER_KEY_FILE'.
+  if [ -n "${WG_SERVER_KEY_FILE:-}" ]; then
-  Either point WG_SERVER_KEY_FILE at it, or set WG_SERVER_PUBKEY to the public half.
+    [ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read WG_SERVER_KEY_FILE '$WG_SERVER_KEY_FILE'"
-  If the server key does not exist yet, see runbooks/vpn-remote-access.md step 1."
+    WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
-  WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
+  else
    command -v sops >/dev/null || die "'sops' not on PATH. Run this inside the dev container (see --help)."
    [ -r "$WG_SERVER_SOPS_FILE" ] || die "cannot read '$WG_SERVER_SOPS_FILE'"
    # The decrypted private key goes straight into `wg pubkey` — never to a file, never to a tty.
    # `wg pubkey` fails on the committed REPLACE-ME placeholder, which is the useful error here.
    WG_SERVER_PUBKEY="$(sops -d --extract '["stringData"]["privatekey"]' "$WG_SERVER_SOPS_FILE" \
                          | tr -d ' \r\n' | wg pubkey 2>/dev/null)" || WG_SERVER_PUBKEY=""
    [ -n "$WG_SERVER_PUBKEY" ] || die "could not derive the server public key from
  $WG_SERVER_SOPS_FILE
  Either it still holds the REPLACE-ME placeholder (see runbooks/vpn-remote-access.md step 2),
  or this shell cannot decrypt it (the age keyring is mounted at ~/.config/sops in the container)."
  fi
 fi
 CLIENT_BASE="${WG_CLIENT_CIDR%/*}"; CLIENT_BASE="${CLIENT_BASE%.*}"
@@ -99,22 +133,41 @@ CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
 # --- refuse a collision, because the failure is otherwise invisible -----------------------------
 # Two devices sharing an address does not error anywhere: the server routes that address to
 # whichever peer handshook most recently, so BOTH devices appear to work intermittently.
-if [ -r "$PEERS_FILE" ] && grep -q "AllowedIPs *= *${CLIENT_ADDR}/32" "$PEERS_FILE"; then
+# COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the
-  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
+# COMMENTED-OUT EXAMPLE STANZA in the register, which made `.2` — the first address anyone would
 # ask for — permanently un-allocatable with a message claiming it was already taken. Strip
 # comments first: a real stanza's `AllowedIPs` line is never commented, while both the template
 # and every peer's `# <name> — <owner>` label are.
 # FAIL if the register is missing rather than skipping the check. The path was silently wrong once
 # (it still said peers.yaml after the file became peers.conf), and because the old guard was
 # `[ -r "$PEERS_FILE" ] && ...` the only symptom would have been duplicate detection quietly
 # never firing again — the exact silent-skip shape this repo treats as a defect.
 [ -r "$PEERS_FILE" ] || die "cannot read the peer register '$PEERS_FILE'.
  Duplicate-address detection cannot run, so this script refuses to issue a profile."
 if grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.conf. Pick another octet."
 fi
-if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
+if grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
-  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2
+  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.conf.\n' "$NAME" >&2
 fi
 # --- output location: 0700, and never inside the repo -------------------------------------------
-if [ -z "$OUT_DIR" ]; then
+[ -n "$OUT_DIR" ] || OUT_DIR="${REPO_ROOT}/.cache/wg-peers"
-  OUT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/wg-peer-XXXXXX")"
+mkdir -p "$OUT_DIR"
 else
  mkdir -p "$OUT_DIR"
 fi
 chmod 700 "$OUT_DIR"
-case "$(cd "$OUT_DIR" && pwd)" in
+OUT_ABS="$(cd "$OUT_DIR" && pwd)"
-  "$REPO_ROOT"|"$REPO_ROOT"/*) die "refusing to write a private key inside the git checkout ($OUT_DIR)" ;;
+# A profile holds a private key, so the destination must not be committable. Inside the repo that
 # means git must IGNORE it — test the real property rather than a proxy, because the repo is the
 # container's only persistent read-write mount and so the only sane place to write. Outside the
 # repo, anywhere goes.
 case "$OUT_ABS" in
  "$REPO_ROOT"|"$REPO_ROOT"/*)
    if ! git -C "$REPO_ROOT" check-ignore -q "$OUT_ABS/probe.conf" 2>/dev/null; then
      die "refusing to write a private key to '$OUT_ABS': it is inside the git checkout and NOT
  gitignored, so the profile would be committable. Use the default (.cache/wg-peers) or add an
  ignore rule first. Verify with: git check-ignore -v <path>"
    fi
    ;;
 esac
 PROFILE="${OUT_DIR}/${NAME}.conf"
@@ -162,7 +215,7 @@ cat <<EOF
  Profile     : ${PROFILE}   (0600; contains the private key)
  ------------------------------------------------------------------------------------------
-  STEP 1 — add this to platform/vpn-ingress/peers.yaml, under data.peers.conf, then PR it:
+  STEP 1 — append this to platform/vpn-ingress/peers.conf, then PR it:
    # ${NAME} — <owner>, enrolled $(date -u +%Y-%m-%d)
    [Peer]
@@ -181,7 +234,7 @@ cat <<EOF
  STEP 3 — delete the profile from this machine once the device has it:
    shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE}
-  The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet
+  The tunnel will not work until the peers.conf PR is MERGED and Flux has rolled the DaemonSet
  (~1 min). Until then the device handshakes and gets silence, exactly like an unknown key.
  ------------------------------------------------------------------------------------------
 EOF
@@ -190,6 +243,8 @@ if command -v qrencode >/dev/null; then
  printf '  QR code (this IS the private key — your terminal scrollback now holds a credential):\n\n'
  qrencode -t ansiutf8 < "$PROFILE"
 else
-  printf '  qrencode not installed, so no QR code was printed (Debian/Ubuntu: apt install qrencode).\n'
+  # Should not happen in the dev container: qrencode is baked into the image. Kept as a real
-  printf '  Transfer %s to the device by another route instead.\n' "$PROFILE"
+  # fallback for a host run, rather than failing after the profile has already been written.
  printf '  qrencode not on PATH, so no QR code was printed. It is in the dev container —\n'
  printf '  rerun there, or transfer %s to the device by another route.\n' "$PROFILE"
 fi
@@ -154,6 +154,74 @@ NEW client, a new protocol mapper or a changed scope still needs the realm impor
 (it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
 a realm edit landed.
 ## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17)
 `keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
 policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
 **ALTERNATIVE** beside the password form. Design decisions and their reasoning are in the manifest
 comments, not repeated here. The headline choices:
 | Decision | Value | One-line reason |
 |---|---|---|
 | Additive or replacing? | **Additive** — passkey *or* password | A misfiring mandatory flow locks the single operator out of every SSO app *and* kubectl |
 | Enrolment | **Optional**, never a default action | Same reason; `webauthn-register-passwordless` is already enabled-not-forced by default |
 | Relying-party ID | `${BASE_DOMAIN}` (not `keycloak.${BASE_DOMAIN}`) | An unre-keyable credential must not be bound to a hostname that can move |
 | User verification | `required` | Without it, mere possession of an unlocked device is a complete login |
 | Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
 | OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |
 ### This change does NOT reach the live realm. At all.
 Same one-shot failure this file already documents twice above, third field over. Measured on prod
 2026-10-17:
 ```
 $ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
 {"conditions":[{"status":"True","type":"Done"},{"status":"False","type":"Started"},
               {"status":"False","type":"HasErrors"}]}
 ```
 `Done=True` / `Started=False` — the import has completed and **will not re-run**, so a policy block
 and three new sub-flows added to this file change nothing about how anyone logs in. The
 client-secret/client-URL reconciler does **not** close this gap and was never meant to: its hard
 endpoint allow-list makes `/authentication` unreachable **by design** (see the table above), and
 realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
 mapper or a changed scope still needs the realm import to actually re-run" case named above.
 **So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until
 an operator applies it by hand.** Two paths, both operator actions:
 1. **Admin console (expected path).** *Authentication → Policies → WebAuthn Passwordless Policy*
   for the policy block, then *Authentication → Flows → browser-mfa* to rebuild the executions to
   match the manifest. The flow edit is the fiddly half: split the existing Username Password Form
   into Username Form + Password Form and add the two sub-flows exactly as declared.
 2. **A deliberate re-import.** Deleting and recreating the `KeycloakRealmImport` re-runs it against
   the **existing** realm. Do not reach for this casually — it is the operation whose failure mode
   is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.
 Either way the realm is then configured by hand and git is only the record of intent, which is a
 live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it.
 ### Verifying it actually works (nobody has, as of 2026-10-17)
 No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
 it, in order:
 ```sh
 # 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
 #    Authentication -> Policies -> WebAuthn Passwordless Policy, in the admin console.
 # 2. enrol: account console (https://keycloak.${BASE_DOMAIN}/realms/homelab/account) -> Signing in
 #    -> Passwordless -> set up. A browser/OS passkey prompt must appear; if it does not, the RP ID
 #    is wrong for the origin and NOTHING will say so.
 # 3. log in: a fresh private window to any SSO-gated app. Expect username screen, then a passkey
 #    prompt. A platform-admin must NOT be asked for TOTP on this path.
 # 4. regression (the important one): a user with NO passkey must still get the password screen,
 #    and a platform-admin must still be asked for TOTP there.
 ```
 A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak
 is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.
 ## Exceptions that keep native auth (do not force SSO) — ADR-0014
 Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).
@@ -74,6 +74,142 @@ spec:
    bruteForceProtected: true
    permanentLockout: false
    sslRequired: external
    # ===========================================================================================
    # PASSKEYS — the WebAuthn PASSWORDLESS policy (R-IDP-04). 2026-10-17.
    # ===========================================================================================
    # Read this together with the browser-mfa flow at the bottom of the file: THIS block decides
    # what a passkey must BE, the FLOW decides when one is OFFERED. Keycloak keeps TWO independent
    # WebAuthn policies per realm and they are not interchangeable:
    #
    #   webAuthnPolicy*             -> the TWO-FACTOR policy, consumed by the `webauthn-authenticator`
    #                                  authenticator (a security key presented AFTER a password).
    #   webAuthnPolicyPasswordless* -> the PASSWORDLESS policy, consumed by
    #                                  `webauthn-authenticator-passwordless` (a passkey presented
    #                                  INSTEAD of a password). This is the one passkeys use.
    #
    # Setting the wrong family is a silent no-op: the passwordless authenticator reads
    # getWebAuthnPolicyPasswordless() only (WebAuthnPasswordlessAuthenticator.getWebAuthnPolicy),
    # so a policy written into the two-factor fields is simply never consulted by a passkey login.
    # Only the passwordless family is declared here; the two-factor family is deliberately left at
    # Keycloak's defaults because nothing in this realm uses `webauthn-authenticator` — TOTP
    # (auth-otp-form) is the second factor for platform-admins, see the flow section.
    #
    # ADDITIVE AND OPTIONAL — THE DECISION THIS BLOCK EXISTS TO MAKE
    # -------------------------------------------------------------
    # Passkeys are offered ALONGSIDE the password, never instead of it, and enrolment is NOT forced.
    # This is a single-operator homelab whose IdP gates Grafana, Gitea, Nextcloud, Headlamp,
    # Semaphore, PiHole, Crafty, the *arr stack AND kubectl itself (the `kubernetes` client below).
    # A mandatory passkey flow that misfires is therefore not an inconvenience, it is a total
    # lock-out from the whole platform AND from the cluster API in one step — and the only repair
    # paths left would be the Keycloak bootstrap admin and the break-glass cert kubeconfig. The
    # expected gain from forcing enrolment (one human stops typing a password) is small; the
    # expected loss is the platform. So: additive, optional, and password login keeps working for
    # every user who never enrols.
    #
    # Concretely that means: no `requiredActions` block is declared anywhere in this file (see the
    # long note in the flow section for why declaring one would be actively DANGEROUS), and the
    # passkey step sits as an ALTERNATIVE beside the password form rather than ahead of it.
    #
    # RELYING-PARTY ID — THE FIELD THAT SILENTLY BREAKS EVERY PASSKEY IF IT IS WRONG
    # ------------------------------------------------------------------------------
    # The RP ID is baked into every credential at enrolment and can never be changed afterwards:
    # a passkey created under one RP ID is unusable under another, with no error the user can act
    # on — the browser simply reports that no credential is available. It is also browser-enforced,
    # so it cannot be "fixed" server-side: per WebAuthn, the RP ID must be equal to, or a
    # registrable-domain suffix of, the origin's effective domain. Keycloak is served at
    # keycloak.<BASE_DOMAIN> (httproute.yaml, and the Keycloak CR's hostname), so the legal choices
    # are keycloak.<BASE_DOMAIN>, <BASE_DOMAIN>, or the registrable domain one level up. It is NOT
    # the full URL and NOT a scheme — "https://..." here would fail every ceremony.
    #
    # We choose BASE_DOMAIN, the shared platform suffix, NOT the keycloak host, for one concrete
    # reason drawn from this repo's own history: a credential that cannot be re-keyed must not be
    # bound to a name that can move. The oauth2-proxy client below ALREADY declares a redirect URI
    # on auth.<BASE_DOMAIN>, i.e. this platform already contemplates serving auth on a second
    # hostname; and the client-URL reconciler exists precisely because moving a hostname in git
    # silently fails to reach the realm (see README, "Client URLs: the same one-shot failure").
    # Binding passkeys to the keycloak subdomain would mean that relocating Keycloak invalidates
    # every enrolled passkey; binding them to BASE_DOMAIN survives it. The cost is scope: any
    # future WebAuthn relying party under this base domain could assert the same RP ID. In a
    # homelab where every such party is already behind THIS IdP that is not a trust boundary worth
    # buying. Note the residual, unavoidable risk: changing BASE_DOMAIN itself invalidates every
    # passkey, which is a re-enrolment, not a repair.
    #
    # Written as the cluster-var so it is substituted by the same Flux postBuild pass, with the same
    # value, as every hostname in this file — there is no second place the domain is decided.
    webAuthnPolicyPasswordlessRpId: ${BASE_DOMAIN}
    # The human-readable name the browser/OS shows in the passkey prompt and stores beside the
    # credential ("Save a passkey for Homelab SSO?"). Matches the realm displayName above so the
    # OS keychain entry is recognisable. It is also load-bearing on the UPDATE path: Keycloak's
    # updateRealm applies the passwordless policy ONLY when this field is non-empty
    # (DefaultExportImportManager line ~842), so an admin-API realm update that omits it drops the
    # whole policy silently.
    webAuthnPolicyPasswordlessRpEntityName: "Homelab SSO"
    # ES256 then RS256 — exactly Keycloak's own DEFAULT_POLICY, declared explicitly rather than
    # inherited so the realm's crypto is auditable from git. ES256 is what every platform
    # authenticator (Touch ID/Face ID, Windows Hello, Android) and every FIDO2 key produces;
    # RS256 is the fallback for older Windows Hello TPM-backed credentials. EdDSA/Ed25519 is
    # deliberately NOT offered: authenticator and browser support is uneven, and an authenticator
    # that picks an algorithm the verifier handles badly fails at enrolment, which is the worst
    # moment to discover it. Order is a preference list, not an allow-list ordering requirement.
    webAuthnPolicyPasswordlessSignatureAlgorithms:
      - ES256
      - RS256
    # DISCOVERABLE CREDENTIAL REQUIRED. "Yes" is a Keycloak sentinel, not a boolean, and the login
    # theme compares it as the literal string: webauthnRegister.js maps 'Yes' to
    # authenticatorSelection.requireResidentKey=true and ANY other non-sentinel value to false.
    # This is not optional for passkeys — a non-discoverable credential cannot be found by the
    # browser without the server first naming it, so it is a second factor, not a passkey.
    # (Keycloak 26.0.6 emits only the LEGACY requireResidentKey member, not the newer
    # residentKey:"required"; per WebAuthn L2 the client maps requireResidentKey=true onto
    # residentKey="required", so discoverable credentials are genuinely requested — but it also
    # means this policy cannot express residentKey:"preferred" at all.)
    webAuthnPolicyPasswordlessRequireResidentKey: "Yes"
    # USER VERIFICATION REQUIRED. The authenticator must verify the human (biometric or PIN) and
    # say so in the assertion. This is the field that makes a passkey login at least as strong as
    # the password it replaces: with verification merely "preferred", possession of an unlocked
    # phone or an unattended security key would be a COMPLETE login to every SSO-gated app — a
    # single-factor possession check, strictly weaker than the password+TOTP path beside it.
    # Required makes the passkey branch inherently two-factor (something you have + something you
    # are/know), which is the premise the flow section relies on when it does NOT ask a
    # passkey-authenticated platform-admin for TOTP as well.
    webAuthnPolicyPasswordlessUserVerificationRequirement: required
    # "not specified" is Keycloak's sentinel for "either kind of authenticator"
    # (Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED; the theme JS omits the WebAuthn member
    # entirely when it sees this exact string). Deliberately NOT narrowed: "platform" would reject
    # every YubiKey and every phone used as a cross-device passkey, and "cross-platform" would
    # reject Touch ID and Windows Hello — i.e. either choice silently halves the usable hardware.
    # The operator should be able to enrol a laptop AND a phone AND a hardware key.
    webAuthnPolicyPasswordlessAuthenticatorAttachment: "not specified"
    # ATTESTATION: none. Requesting "direct" would hand Keycloak an attestation statement it has
    # no trust anchors to validate here (no metadata service is configured) and nothing to compare
    # it against, since the AAGUID allow-list below is deliberately empty — so the statement would
    # be collected, unverifiable, and would add an extra "share model information?" consent prompt
    # on Safari/iOS for no security gain. Attestation is how you ENFORCE an approved-hardware list;
    # we do not have one, so asking for it is cost without benefit.
    webAuthnPolicyPasswordlessAttestationConveyancePreference: none
    # Empty = accept any authenticator model. An AAGUID allow-list is a hardware whitelist: it
    # would have to be maintained by hand, and every new phone, laptop or key the operator buys
    # would be rejected at enrolment with a message that does not explain itself. It is also moot
    # while attestation is "none" (above) — with no attestation statement there is no trustworthy
    # AAGUID to match against in the first place.
    webAuthnPolicyPasswordlessAcceptableAaguids: []
    # Ceremony timeout in SECONDS (the login theme multiplies by 1000; 0 means "send no timeout and
    # let the browser decide"). 300s is chosen over a tighter value because the slowest LEGITIMATE
    # path is the cross-device hybrid flow — Keycloak renders a QR code, the operator unlocks a
    # phone, scans it, and completes a biometric check — which routinely takes over a minute. A
    # 60-120s timeout looks tidy and converts that flow into an unexplained failure. Browsers clamp
    # this value (Chrome to at most 10 minutes), so 300 is honoured rather than reinterpreted.
    webAuthnPolicyPasswordlessCreateTimeout: 300
    # Send the user's existing passkeys as excludeCredentials at enrolment, so re-enrolling an
    # authenticator the user already registered is refused by the authenticator itself instead of
    # producing a second, indistinguishable credential in the account console. Cheap, and it keeps
    # "which passkey is which" answerable — which matters when the recovery story is "delete the
    # lost one".
    webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister: true
    # webAuthnPolicyPasswordlessExtraOrigins is intentionally NOT set. It exists to admit non-web
    # origins (native app facet IDs, e.g. android:apk-key-hash:...). Every ceremony here happens in
    # a browser on Keycloak's own origin, which is already covered by the RP ID above; adding
    # origins is only ever a widening of who may ask for these credentials.
    # --- Authorization model (R-IDP-06): groups drive access; platform-admins carry a realm role. ---
    roles:
      realm:
@@ -528,14 +664,105 @@ spec:
              "access.token.claim": "true"
              "userinfo.token.claim": "true"
              "claim.name": groups
-    # --- MFA for platform-admins (R-IDP-04): a browser flow whose OTP step is CONDITIONAL on the
+    # ===========================================================================================
-    #     platform-admin role. Non-admins log in without OTP; platform-admins must enroll/present TOTP.
+    # MFA for platform-admins (R-IDP-04) + PASSKEY login (2026-10-17). The realm browserFlow is
-    #     Set as the realm browserFlow below. VERIFY at bring-up (see runbook): a malformed flow fails
+    # bound to browser-mfa at the bottom of this file.
-    #     the whole import — if so, the fallback is to comment this block + browserFlow and configure
+    # ===========================================================================================
-    #     the conditional-OTP-by-role flow via the admin console. ---
+    # Non-admins log in without OTP; platform-admins must enroll/present TOTP. VERIFY at bring-up
    # (see runbook): a malformed flow fails the whole import — if so, the fallback is to comment
    # this block + browserFlow and configure the flow via the admin console.
    #
    # THE SHAPE, AND WHY IT IS THIS SHAPE
    # -----------------------------------
    #   browser-mfa                           (top level, bound as browserFlow)
    #     auth-cookie                     ALTERNATIVE   already-authenticated SSO cookie
    #     identity-provider-redirector    ALTERNATIVE   (no IdPs configured; kept for parity)
    #     browser-mfa-forms               ALTERNATIVE   the interactive path:
    #       auth-username-form              REQUIRED      WHO are you (no password yet)
    #       browser-mfa-passkey-or-password REQUIRED      ...and PROVE it, one of two ways:
    #         webauthn-authenticator-passwordless  ALTERNATIVE   a passkey, or
    #         browser-mfa-password-and-otp         ALTERNATIVE   the password path:
    #           auth-password-form                   REQUIRED      password, then
    #           browser-mfa-conditional-otp          CONDITIONAL   TOTP iff platform-admin
    #
    # WHY THE USERNAME/PASSWORD SCREEN HAD TO BE SPLIT IN TWO. The previous flow used the combined
    # `auth-username-password-form`, which collects both on one page. A passkey branch cannot live
    # behind that: the server must know WHICH user is logging in before it can decide whether a
    # passkey is even available, and the combined form has already taken the password by the time
    # it knows. So it is replaced by the `auth-username-form` / `auth-password-form` pair — the
    # same decomposition Keycloak's own passwordless recipe uses. THIS IS A VISIBLE UX CHANGE FOR
    # EVERY USER, passkey or not: login becomes two screens (username, then password) instead of
    # one. That cost is accepted deliberately — it is the enabling change, it is what Google and
    # Microsoft logins already do, and the alternative (below) is worse.
    #
    # WHY NOT USERNAMELESS ("just tap your passkey", no username screen at all). Keycloak 26.0.6
    # does support it: WebAuthnPasswordlessAuthenticator.requiresUser() returns FALSE, so the
    # authenticator can legally run as a first step with no user in context. We deliberately do not
    # do that, because of how the flow engine then behaves. In DefaultAuthenticationFlow the
    # "does this user actually have this credential?" guard sits INSIDE the requiresUser() branch;
    # with requiresUser() false and no user yet set, the engine skips that guard and simply invokes
    # the authenticator. A passkey execution placed before the username form is therefore offered
    # to EVERYONE, so every login — including every password-only user and every family member who
    # will never own a passkey — would open on a passkey prompt they must dismiss. That is the
    # opposite of additive. Keeping the passkey step AFTER auth-username-form means a user is in
    # context, which is what makes the graceful degradation below actually work.
    #
    # WHY THE PASSKEY BRANCH DEGRADES CLEANLY TO A PASSWORD (the property the whole "additive"
    # claim rests on). With a user in context, the engine takes the non-requiresUser branch and
    # raises CREDENTIAL_SETUP_REQUIRED when the user has no credential of this type and the
    # authenticator's factory does not allow self-setup — and WebAuthnPasswordlessAuthenticatorFactory
    # .isUserSetupAllowed() is false, and the authenticator is a CredentialValidator, so both
    # conditions hold. That exception is raised INSIDE the ALTERNATIVE loop, which catches it,
    # marks the execution ATTEMPTED and proceeds to the next alternative. Net effect: a user with
    # no passkey never sees a passkey screen and lands on the password form; a user WITH one gets
    # the passkey screen and keeps the password as a "Try another way" option. Note that for a
    # user holding BOTH, which screen appears FIRST is decided by Keycloak's authentication
    # selection list (the user's own credential ordering), not by the priorities below.
    #
    # WHY A PASSKEY LOGIN IS NOT ALSO ASKED FOR TOTP. browser-mfa-conditional-otp hangs off the
    # PASSWORD branch only, so a platform-admin who authenticates with a passkey is not challenged
    # for TOTP. That is intentional, and it rests on the policy above: userVerification is
    # `required`, so the authenticator has already proved possession of hardware AND verified the
    # human (biometric or PIN) in one phishing-resistant, origin-bound ceremony. Stacking TOTP on
    # top would add a replayable, phishable shared secret to a credential that is neither, for
    # friction on every login. If a future operator disagrees, the change is small and local: move
    # the browser-mfa-conditional-otp reference out of browser-mfa-password-and-otp and into
    # browser-mfa-passkey-or-password's parent as a REQUIRED step, so it applies to both branches.
    #
    # WHY THERE IS NO `requiredActions` BLOCK IN THIS FILE — DO NOT ADD ONE
    # --------------------------------------------------------------------
    # Enrolment uses Keycloak's `webauthn-register-passwordless` required action, and the obvious
    # move is to declare it here. DO NOT. In Keycloak 26.0.6 the realm import treats
    # requiredActions as REPLACE, NOT MERGE: DefaultExportImportManager.importRealm branches on
    # `if (rep.getRequiredActions() != null)` and, when the representation declares ANY, imports
    # exactly those plus delete_account — the `else` branch that installs the full default set
    # (DefaultRequiredActions.addActions) never runs. RealmManager.setupRequiredActions is no
    # backstop either: it fires only when the realm has ZERO required-action providers.
    #
    # So declaring one action would create this realm with that action and delete_account and
    # NOTHING ELSE — no CONFIGURE_TOTP, no UPDATE_PASSWORD, no VERIFY_EMAIL, no delete_credential,
    # no UPDATE_PROFILE. The first casualty would be the MFA requirement this very flow enforces:
    # with CONFIGURE_TOTP unregistered, a platform-admin who does not yet have TOTP could never
    # enrol one, while auth-otp-form below still demands it — a hard lock-out of every
    # platform-admin from every SSO-gated app, including Headlamp and kubectl, delivered by a
    # change whose stated purpose was to ADD a login method.
    #
    # And the declaration buys nothing, because the default set ALREADY CONTAINS IT.
    # DefaultRequiredActions registers `webauthn-register-passwordless` with enabled=true,
    # defaultAction=false, priority 80, gated on Profile.Feature.WEB_AUTHN — which is Type.DEFAULT
    # in 26.0.6, i.e. on unless explicitly disabled. enabled=true / defaultAction=false IS the
    # additive-and-optional posture this change wants: the action exists and can be triggered for
    # a user on demand, but it is not forced on anyone at next login. Declaring it would restate
    # the default while deleting ten siblings.
    #
    # (The symmetric question for authenticationFlows has the opposite answer, which is why THIS
    # block is safe: importAuthenticationFlows adds the declared flows and then backfills each
    # unbound binding — browser, registration, direct grant, reset credentials, client auth, first
    # broker login, docker auth — from the built-in defaults individually. Declaring flows does not
    # suppress the built-ins; declaring required actions does suppress them.)
    authenticationFlows:
      - alias: browser-mfa
-        description: "Browser flow with conditional OTP required for platform-admins."
+        description: "Browser flow: passkey or password, with conditional OTP for platform-admins."
        providerId: basic-flow
        topLevel: true
        builtIn: false
@@ -555,13 +782,63 @@ spec:
            priority: 30
            flowAlias: browser-mfa-forms
            userSetupAllowed: false
      # Identify the user FIRST (username only), then prove it. Both steps are REQUIRED: the
      # username is never optional, and neither is proving possession of the account.
      - alias: browser-mfa-forms
-        description: "Username/password then conditional OTP."
+        description: "Username, then passkey or password (+ conditional OTP)."
        providerId: basic-flow
        topLevel: false
        builtIn: false
        authenticationExecutions:
-          - authenticator: auth-username-password-form
+          # auth-username-form = Keycloak's "Username Form" (UsernameFormFactory) — resolves the
          # user WITHOUT taking a password, which is the prerequisite for offering a passkey.
          # Replaces the previous auth-username-password-form; see the header for the UX cost.
          - authenticator: auth-username-form
            authenticatorFlow: false
            requirement: REQUIRED
            priority: 10
            userSetupAllowed: false
          - authenticatorFlow: true
            requirement: REQUIRED
            priority: 20
            flowAlias: browser-mfa-passkey-or-password
            userSetupAllowed: false
      # THE FORK. Two ALTERNATIVEs, so satisfying either one completes the flow. Priority puts the
      # passkey first for users who have one; users who do not fall through to the password branch
      # automatically (CREDENTIAL_SETUP_REQUIRED is caught by the ALTERNATIVE loop — see header).
      - alias: browser-mfa-passkey-or-password
        description: "A passkey (WebAuthn passwordless), or password with conditional OTP."
        providerId: basic-flow
        topLevel: false
        builtIn: false
        authenticationExecutions:
          # webauthn-authenticator-passwordless = "WebAuthn Passwordless Authenticator"
          # (WebAuthnPasswordlessAuthenticatorFactory). It reads the realm's PASSWORDLESS policy
          # declared above — NOT the two-factor webAuthnPolicy* family. Keep it ALTERNATIVE: as
          # REQUIRED it would force a passkey on every user, which is the lock-out this change is
          # explicitly designed not to create.
          - authenticator: webauthn-authenticator-passwordless
            authenticatorFlow: false
            requirement: ALTERNATIVE
            priority: 10
            userSetupAllowed: false
          - authenticatorFlow: true
            requirement: ALTERNATIVE
            priority: 20
            flowAlias: browser-mfa-password-and-otp
            userSetupAllowed: false
      # The unchanged pre-passkey behaviour, relocated intact into the password branch: password,
      # then TOTP iff the user holds the platform-admin role. Nothing about the OTP condition
      # changed; it simply no longer sits on the path a passkey login takes.
      - alias: browser-mfa-password-and-otp
        description: "Password, then conditional OTP for platform-admins."
        providerId: basic-flow
        topLevel: false
        builtIn: false
        authenticationExecutions:
          # auth-password-form = Keycloak's "Password Form" (PasswordFormFactory) — the password
          # half of the old combined form, now that the username has already been collected.
          - authenticator: auth-password-form
            authenticatorFlow: false
            requirement: REQUIRED
            priority: 10
@@ -199,7 +199,7 @@ data:
          "id": 15,
          "type": "timeseries",
          "title": "CAUSE 4 — NFS RPC retransmission ratio, per node",
-          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.102, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
+          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.30, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
          "gridPos": {"h": 8, "w": 12, "x": 12, "y": 39},
          "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
            {"color": "green", "value": null}, {"color": "red", "value": 0.01}
@@ -63,6 +63,75 @@ spec:
      # gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
      skip-provider-button: "true"
      scope: "openid email profile"
    # ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ──────────
    # MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at
    # startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so
    # the Gateway answers 503 and the proxy dies:
    #
    #   [provider.go:55] Performing OIDC Discovery...
    #   [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC
    #                configuration: unexpected status "503": no healthy upstream
    #
    # It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was
    # still down minutes later and only a manual pod delete cleared it. There is no retry flag; the
    # process simply does not survive a cold start. A readinessProbe cannot help (the container is
    # already dead), and CrashLoopBackOff backoff is not tunable per-pod.
    #
    # `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in
    # values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init
    # container for its built-in waitForRedis gate, so this adds no new base image to the platform.
    # A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole
    # point.
    #
    # BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst
    # case is then exactly today's behaviour, never something new. An unbounded wait would trade a
    # CrashLoop for a pod wedged in Init forever, which is not an improvement.
    #
    # SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is
    # rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which
    # CONSUMES braced expansions -- so a braced shell variable renders to the empty string and
    # silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars
    # knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud
    # OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.)
    extraInitContainers:
      - name: wait-for-idp
        image: docker.io/library/alpine:3.22
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh", "-c"]
        args:
          - |
            set -u
            url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
            i=0
            while [ "$i" -lt 60 ]; do
              # busybox wget validates against alpine's ca-certificates-bundle, so a real TLS
              # failure is a non-zero exit here rather than a false pass.
              if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then
                echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy"
                exit 0
              fi
              i=$(( i + 1 ))
              sleep 5
            done
            echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2
            echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2
            exit 0
        securityContext:
          allowPrivilegeEscalation: false
          runAsNonRoot: true
          runAsUser: 65534
          runAsGroup: 65534
          readOnlyRootFilesystem: true
          capabilities:
            drop:
              - ALL
        resources:
          requests:
            cpu: 10m
            memory: 16Mi
          limits:
            memory: 32Mi
    # Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
    metrics:
      enabled: true
@@ -13,8 +13,8 @@ egress out the house WAN.
 | Transport | WireGuard, **udp/51820**, kernel implementation |
 | Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
 | Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
-| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
+| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
-| The one secret | the server's private key, Secret `wg-server-key`, **created out of band** |
+| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
 | Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
 | Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
 | Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
@@ -129,12 +129,18 @@ translations win here.
 | half | where | why |
 |---|---|---|
-| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
+| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
-| the server's **private** key | Secret `wg-server-key`, **out of band** | the only real secret; an agent must not mint a long-lived private key (CLAUDE.md §6) |
+| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
 The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
 so the assembled config holding the private key never touches any node's disk and dies with the pod.
 Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with
 everything else. An out-of-band `kubectl create secret` — the first shape this component shipped in,
 corrected after review — would have put the one credential every enrolled device depends on *outside*
 the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this
 key's public half.
 Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
 new line granting a new device access to the whole LAN.
@@ -160,21 +166,40 @@ NetworkPolicy applies.
 ## Operating it
 * **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
-  stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
+  stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
-* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
+* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
  stops working as soon as the pods roll (the generator's content hash guarantees they do).
 * **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
 * **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
  (`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
 * **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
 ## Measured on prod, 2026-10-06 — what is now known rather than assumed
 The first real reconcile settled four things this component's design rested on. Three went the
 right way; one was a defect in my own manifest.
 | question | answer |
 |---|---|
 | Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. |
 | Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. |
 | Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
 | Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
 ## Prerequisites outside this directory
 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
   (`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
   `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
   CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
-2. **The `wg-server-key` Secret.** Runbook step 1. Until it exists the DaemonSet cannot start —
+2. **The server private key.** `wg-server-key.sops.yaml` is committed, encrypted, with a
-   the correct fail-closed state for an unenrolled VPN.
+   `REPLACE-ME` placeholder, and both overlays already carry its decryption patch — so the wiring
   is complete and the only outstanding act is replacing the value (runbook step 2). Until then the
   DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the
   value with `wg pubkey`, which rejects the placeholder, an empty value, a public key pasted by
   mistake, **and** a Secret that arrived as undecrypted `ENC[...]` ciphertext. That last case is
   the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves
   every Flux condition green.
 3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
   and the half CI can never check.
@@ -187,7 +212,7 @@ NetworkPolicy applies.
  covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
  kube-prometheus-stack. **This is the top follow-up.**
 * **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
-  handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
+  handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
  reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
  `wg-server-key` Secret.
 * **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
@@ -37,6 +37,10 @@
 #     /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
 #     is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
 #     required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change).
 #     COROLLARY LEARNED THE HARD WAY (prod, 2026-10-06): this pod must not merely avoid NEEDING
 #     the sysctl, it must not ATTEMPT it. hostNetwork implies /proc/sys is mounted read-only, so
 #     `sysctl -w` fails with EPERM regardless of capabilities, and wg-quick treats a failed PostUp
 #     as fatal and deletes the interface. Verify the value; never write it.
 #   * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost
 #     this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all.
 #   * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a
@@ -104,7 +108,7 @@ spec:
      dnsPolicy: ClusterFirstWithHostNet
      automountServiceAccountToken: false
      # ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
-      # The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
+      # The two halves are deliberately stored apart (peers.conf explains why) and are only ever
      # joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
      # key is therefore never written to any node's disk, never committed, and never logged.
      initContainers:
@@ -136,9 +140,51 @@ spec:
              # The key arrives as a Secret file; strip any trailing newline the operator's shell
              # added, or wg-quick rejects the config with "Key is not the correct length or format".
              key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey)
-              if [ -z "$key" ]; then
+              # VALIDATE, don't just check for empty. The Secret ships committed with a REPLACE-ME
-                echo "FATAL: /run/wg-server-key/privatekey is empty." >&2
+              # placeholder so the GitOps wiring is complete before the key exists, so the common
-                echo "       See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2
+              # failure is a value that is present and WRONG — a placeholder, a public key pasted by
              # mistake, or a SOPS document that reached the cluster undecrypted (ENC[...], which
              # happens if an overlay is missing its `vpn-ingress` decryption patch). `wg pubkey`
              # accepts a value only if it is a real 32-byte curve25519 private key, so it rejects
              # all four. Without this the bad value would be written into wg0.conf and the pod would
              # fail later and less legibly, or come up with a server nobody can handshake with.
              if ! printf '%s' "$key" | wg pubkey >/dev/null 2>&1; then
                echo "FATAL: the server private key in Secret 'wg-server-key' is not a valid" >&2
                echo "       WireGuard private key (wg pubkey rejected it)." >&2
                if [ -z "$key" ]; then
                  echo "       It is EMPTY." >&2
                elif [ "$key" = "REPLACE-ME" ]; then
                  echo "       It is still the committed REPLACE-ME PLACEHOLDER — the key has not" >&2
                  echo "       been generated yet. This is the expected state on a fresh install." >&2
                else
                  # `case`, NOT a dollar-brace prefix-strip. This manifest is rendered through
                  # envsubst, which consumes every dollar-brace expansion it sees, so a shell
                  # parameter expansion here renders to the empty string and silently disables this
                  # very branch. `case` uses no braces, so there is nothing to escape.
                  # (ci: flux-postbuild-substitution caught both the original expansion AND the
                  # first version of this comment, which spelled the hazard out literally.)
                  case "$key" in
                    ENC\[*)
                      echo "       It is still SOPS CIPHERTEXT (starts with ENC[). The Secret" >&2
                      echo "       reached the cluster undecrypted — check that this environment's" >&2
                      echo "       overlay carries the 'vpn-ingress' decryption patch." >&2
                      ;;
                  esac
                fi
                echo "       Fix: runbooks/vpn-remote-access.md step 2." >&2
                exit 1
              fi
              # The pod shares the HOST netns, so this reads the node's real setting. Reading
              # /proc/sys is permitted even though writing is not (see the PostUp note below).
              # Fail closed: a tunnel that comes up but cannot forward would look healthy to every
              # probe here and simply drop every client's traffic.
              if ! grep -qx 1 /proc/sys/net/ipv4/ip_forward; then
                echo "FATAL: net.ipv4.ip_forward is 0 on this node, so forwarded VPN traffic would" >&2
                echo "       be dropped. This pod CANNOT set it: with hostNetwork the runtime mounts" >&2
                echo "       /proc/sys read-only, because it is the host's global setting." >&2
                echo "       Fix on the NODE: run the node_baseline role (it writes" >&2
                echo "       /etc/sysctl.d/99-k3s-cilium.conf) — runbooks/vpn-remote-access.md step 1." >&2
                exit 1
              fi
@@ -155,9 +201,20 @@ spec:
                echo 'MTU = 1420'
                printf 'PrivateKey = %s\n' "$key"
                # ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ──
-                # ip_forward is already 1 from node_baseline; re-asserted so this component does not
+                # NO `sysctl -w net.ipv4.ip_forward=1` HERE. It is IMPOSSIBLE from this pod and it was
-                # silently depend on another role's sysctl file surviving.
+                # the only thing stopping the tunnel coming up — MEASURED on prod 2026-10-06:
-                echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1'
+                #     [#] sysctl -q -w net.ipv4.ip_forward=1
                #     sysctl: permission denied on key "net.ipv4.ip_forward"
                #     [#] ip link delete dev wg0
                # and wg-quick then tore the whole tunnel down. The cause is not a missing
                # capability: with `hostNetwork: true` the runtime mounts /proc/sys READ-ONLY in the
                # container (confirmed in /proc/mounts: `proc /proc/sys proc ro,...`), precisely
                # because those sysctls are the HOST's globals and a container must not rewrite
                # them. CAP_NET_ADMIN does not and cannot lift a read-only mount.
                # It was never needed anyway: node_baseline writes net.ipv4.ip_forward=1 to
                # /etc/sysctl.d/99-k3s-cilium.conf, and reading it from this pod returns 1. The
                # init container now VERIFIES that (a read, which /proc/sys does allow) and fails
                # closed with a pointed message, which is what the write was pretending to achieve.
                # SNAT client traffic to the node's address. REQUIRED: without it a LAN host would
                # reply to 10.90.0.x, find no route, hand the packet to the router, and the router
                # would drop it — so every connection would hang rather than fail. Matching on
@@ -217,8 +274,10 @@ spec:
            # `kubectl get pod -o wide` shows what this listener owns.
            - { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP }
          securityContext:
-            # NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and
+            # NET_ADMIN is the whole privilege budget: create the wg0 link and write the netfilter
-            # write the two netfilter rules above. Everything else is dropped, and the container is
+            # rules above. (It does NOT let the pod set a host sysctl — /proc/sys is read-only under
            # hostNetwork — which is why nothing here tries.) Everything else is dropped, and the
            # container is
            # NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule.
            # SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node
            # (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container
@@ -232,6 +291,16 @@ spec:
                - NET_ADMIN
            # readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and
            # /var during startup. Same deferral as crafty/headlamp.
            #
            # EXPECTED, HARMLESS STARTUP NOISE — do not "fix" it. Because capabilities are dropped
            # to NET_ADMIN only, the image's s6 init cannot chown /config (no CAP_CHOWN) and logs a
            # run of `chown: ... Operation not permitted` plus "Permissions could not be set. This
            # is probably because your volume mounts are remote or read-only" and a scary "we will
            # not provide support for it". It is wrong about the cause and it does not matter:
            # /config is a writable tmpfs emptyDir, everything here runs as root (uid 0) anyway, and
            # wg-quick proceeds normally — verified on prod 2026-10-06, where the tunnel's only
            # failure was the sysctl above, well past these warnings. Granting CAP_CHOWN to silence
            # log noise would widen the privilege set of an internet-facing pod for nothing.
          resources:
            requests:
              cpu: 25m
@@ -272,14 +341,22 @@ spec:
            # during startup rather than at a point anyone would connect to the volume size. Still
            # counted against the container's 128Mi limit, and still far more than wg0.conf needs.
            sizeLimit: 8Mi
-        # The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by
+        # The ONLY secret in this component. SOPS-encrypted IN THIS TREE (R-SEC-01, spec docs/07) as
-        # name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this
+        # wg-server-key.sops.yaml, delivered by Flux, referenced here by name only (R-IDP-04).
-        # tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key
+        #
-        # that passed through a session transcript would have to be treated as compromised.
+        # It ships committed with a REPLACE-ME placeholder: an agent must not mint a long-lived
-        # DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace
+        # private key (CLAUDE.md §6), so the operator generates it and `sops set`s it in
-        # and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy
+        # (runbooks/vpn-remote-access.md step 2). Shipping the encrypted placeholder rather than no
-        # in the operator's password manager as well — restoring it is what makes every already
+        # file at all is what lets the resource line and both overlay decryption patches be present
-        # enrolled client work again without re-enrolling.
+        # and GREEN from the first commit — the alternative, a commented-out resource to uncomment
        # later, is exactly the "file in git with no generator entry: reviewed, merged, apparently
        # done, still broken" shape this repo keeps getting bitten by.
        #
        # DR: in git means a cluster rebuild restores it with everything else — the property an
        # out-of-band `kubectl create secret` would NOT have had, and losing this key means
        # re-enrolling every device because each client profile pins its public half. Velero also
        # captures this namespace (its `daily` schedule excludes only `monitoring` and `unifi`).
        # Keep a password-manager copy too: that is the one that survives losing cluster AND backups.
        - name: wg-server-key
          secret:
            secretName: wg-server-key
@@ -4,27 +4,66 @@
 # same placement as crafty/headlamp/homepage/runbooks, for the same reason: a leaf whose pod can be
 # legitimately unhealthy must never sit in the gating `platform` tier, where one component's
 # unhealth stalls platform-config and every apps wave behind it (DECISIONS-LOG: "Headlamp gating
-# stall"). That rationale is unusually concrete here: until the operator creates the `wg-server-key`
+# stall"). That rationale is unusually concrete here: until the operator replaces the `REPLACE-ME`
-# Secret (runbooks/vpn-remote-access.md step 1) this DaemonSet CANNOT start, so putting it in the
+# placeholder in wg-server-key.sops.yaml (runbooks/vpn-remote-access.md step 2) this DaemonSet
-# gating tier would have taken the whole platform down on merge.
+# CANNOT start, so putting it in the gating tier would have taken the whole platform down on merge.
 #
 # NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 # REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated
 # ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's
 # nameReference transformer will not rewrite a reference across a namespace boundary — so without
 # this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the
 # pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1).
 # Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own
 # content generator.
 namespace: vpn-ingress
 resources:
  - namespace.yaml
  # The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
  - peers.yaml
  # The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
  - daemonset.yaml
  # Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
  # says so at length rather than implying a protection it cannot provide.
-  - networkpolicy.yaml
+  - networkpolicy.yaml  # The server's WireGuard private key — the ONE secret here. SOPS-encrypted (R-SEC-01, spec
  # docs/07), decrypted per-env by the `vpn-ingress` Kustomization, referenced by name only from
  # daemonset.yaml (R-IDP-04). Its plaintext `.example` sibling carries the operator hand-off --
  # deliberately NOT named here: conformance/tests/l0-static/no-plaintext-secrets fails any
  # kustomization that mentions an unencrypted template, because naming one is how a plaintext
  # Secret gets applied by accident. (It matches comments too. That is correct, not overzealous.)
  #
  # SHIPS WITH A `REPLACE-ME` PLACEHOLDER, already encrypted. An agent must not
  # mint a long-lived private key (CLAUDE.md §6), and the init container validates the value with
  # `wg pubkey`, so the unfilled state is a loud CrashLoop naming this file, never a server that
  # quietly cannot handshake.
  - wg-server-key.sops.yaml
 #
-# DELIBERATELY ABSENT: a Secret. The server's private key (`wg-server-key`) is created out of band
+# THIS COMPONENT CARRIES AN ENCRYPTED SECRET, so BOTH overlays must patch the `vpn-ingress`
-# by the operator and referenced BY NAME from daemonset.yaml. There is no `*.sops.yaml` here, so
+# Kustomization with a `decryption:` stanza — clusters/dev and clusters/prod both do, added by the
-# this component needs NO SOPS decryption stanza in clusters/dev or clusters/prod — unlike
+# same change as this file. That patch is as load-bearing as the file itself: without it kustomize
-# crafty/headlamp/velero. If that key is ever migrated into SOPS, BOTH halves must land together:
+# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
-# the file here, and a `decryption:` patch naming the `vpn-ingress` Kustomization in each overlay.
+# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
-# Without the patch, kustomize hands Flux the raw ciphertext and the pod mounts a key full of
+# that into a named error rather than a silently non-working tunnel.
-# ENC[...] — which applies cleanly and fails only at handshake time.
+
 # ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ─────────────────────────────────────────
 # peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a
 # GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic:
 # kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's
 # volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE
 # PODS automatically.
 #
 # A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux
 # updates it in place, the pod template is byte-identical, no rollout is triggered — and because
 # the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps
 # the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the
 # ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having
 # never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this.
 #
 # DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism.
 configMapGenerator:
  - name: wg-peers
    files:
      - peers.conf
    options:
      labels:
        app.kubernetes.io/name: vpn-ingress
@@ -19,7 +19,7 @@
 # protects this port, and it would be enforcing nothing. So there isn't one.
 #
 # WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
-#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
+#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.conf gets
 #      NO REPLY — see README "Why this is safe to expose". This is the real control.
 #   2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
 #   3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement
@@ -1,4 +1,16 @@
---
+# THIS FILE IS THE MOUNTED CONTENT. It is turned into the `wg-peers` ConfigMap by
 # `configMapGenerator` in kustomization.yaml, whose generated name carries a CONTENT HASH — so
 # editing this file changes the ConfigMap name, which changes the DaemonSet's pod template, which
 # ROLLS THE PODS. That is the whole reason it is a generator and not an inline ConfigMap manifest:
 # a fixed-name ConfigMap updates in place, the pod template never changes, no rollout happens, and
 # the init container's wg0.conf (built once at pod start, in tmpfs) keeps the OLD peer list
 # forever. Measured 2026-10-06: a merged peer sat in the ConfigMap while `wg show` listed none and
 # the DaemonSet reported generation=1 — it had never rolled. Same mechanism and same fix as
 # runbooks/kustomization.yaml.
 #
 # Every line here is WireGuard config. `#` is a legal comment in it, so the documentation below
 # travels inside the file that is actually mounted, rather than in a wrapper that gets stripped.
 #
 # ============================================================================================
 # THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
 # ============================================================================================
@@ -41,33 +53,40 @@
 # the Noise handshake and would be a genuine secret — which would force this register into SOPS
 # and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
 # belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
-apiVersion: v1
+#
-kind: ConfigMap
+# name: wg-peers
-metadata:
+# namespace: vpn-ingress
-  name: wg-peers
+# labels:
-  namespace: vpn-ingress
+# app.kubernetes.io/name: vpn-ingress
-  labels:
+# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
-    app.kubernetes.io/name: vpn-ingress
+#
-data:
+# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
-  # Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
+# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
-  #
+# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
-  # *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
+# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
-  # An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
+# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
-  # authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
+# (`wg show` reports an interface with no peers).
-  # DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
+#
-  # The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
+# COPY THIS TEMPLATE, uncomment, one block per device:
-  # (`wg show` reports an interface with no peers).
+#
-  #
+#   # <device-name> — <owner>, enrolled <YYYY-MM-DD>
-  # COPY THIS TEMPLATE, uncomment, one block per device:
+#   [Peer]
-  #
+#   PublicKey = <the public key new-peer.sh printed>
-  #   # pixel-8 — Keith's phone, enrolled 2026-10-05
+#   AllowedIPs = 10.90.0.<octet>/32
-  #   [Peer]
+#
-  #   PublicKey = <the public key new-peer.sh printed>
+# The placeholders above are deliberately NOT valid values. An earlier version of this template
-  #   AllowedIPs = 10.90.0.2/32
+# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
-  #
+# the whole file — matched it, so `.2` could never be allocated and the script insisted it was
-  # Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
+# already taken. The check now ignores comments; keeping the example unmatchable too means a
-  # tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
+# future grep cannot be fooled either.
-  # it an incident responder cannot tell which stanza to delete.
+#
-  peers.conf: |
+# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
-    # No peers enrolled yet. See the template in this file's comments and
+# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
-    # ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
+# it an incident responder cannot tell which stanza to delete.
 #
 # No peers enrolled yet. See the template in this file's comments and
 # ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
 # keith-phone — <owner>, enrolled 2026-10-06
 [Peer]
 PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
 AllowedIPs = 10.90.0.2/32
@@ -0,0 +1,34 @@
 # WireGuard SERVER private key for platform/vpn-ingress — SOPS-ENCRYPTED (R-SEC-01, spec docs/07).
 # Keys stay cleartext for reviewable diffs; only `stringData` values are encrypted (.sops.yaml
 # `encrypted_regex`). Plaintext template + operator hand-off: wg-server-key.sops.yaml.example.
 apiVersion: v1
 kind: Secret
 metadata:
    name: wg-server-key
    namespace: vpn-ingress
 type: Opaque
 stringData:
    #ENC[AES256_GCM,data:fUoPkfWvasisdTtwdhjeOVOlmdQkTl6Bp9YxYyji8g6k+ClATds3BnHJWOj2iplaGpT2XaNakq1+D4YyOi7QbFvNJui+e4qRpqcY8hWnazvZIHJYxjxfAfzW,iv:xtKd6oMZQumAQJAsNLHqJB+o0wdQJ+vL1kI1mc9jL7w=,tag:aImqLgbSmhsRI36/t5kpyg==,type:comment]
    #ENC[AES256_GCM,data:npDsCW6rQleolIpur5gFkX2jSRrIQzBlGaIPs2Schv0YU3q8AanG39z4QqT3ASDmskV5YuH29Bo0ujEAgBf52YIjQCp58PDDETofrVKOJzhHMa7bV+HCxUFE,iv:7E7thyFBleyqZpqJRGApKvIq3ubCFOkshF/mxim3YPc=,tag:joIrhMe2I6kUnx1ouOPdZw==,type:comment]
    #ENC[AES256_GCM,data:wHUFx09FgP/2+LTyw61EFbKA5UaqfjU7CeGk6Xc5av2y4r4AvQJRefDM5EGlahKecDFEFsf7fSQRFRHtUJqrQTtw+6tcj3ubSCPW4iJvhyJJv0xdb/0puRmcgj+2NfgF,iv:xgDOSDLqMQ1wcI9vrkKqfNMgIDFEusFnnO7Rzq/aOa4=,tag:m1m/fgjkBsG91c+3daqOpA==,type:comment]
    privatekey: ENC[AES256_GCM,data:ML8CPNm4JwhQHUTUUxZO/I+9X15BCIGqOb1Uw9EmX5RTAsxQ9N97YvkYlbY=,iv:5ZUsFdMMmeGsy4fNbk6qbJkzweMVWXZ5g6E7NciAGUg=,tag:pAHv3ieLjoEyFbqbWXsVgg==,type:str]
 sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMays2OXY0VkpRc2FQQmhU
            Q0owM3N0VkhYR0lYTUdNZWRqeTFIMjNHOVJ3CnJSMnc5NFEyU0IzWTFtUkw2YVFS
            MU02VFo1TE14YVpZRzNhVjZwRTZwd0UKLS0tIDFaMWF4NUZjZmhRdXRuZFJYNkg3
            SUUzN1FnMlI4bkkzbVNlMWhUVnJZODAKQ+Iy86CIv/Oj9E9j4E/0y5hDbcctRuqB
            OR825KRPRaTdF/G2hudE1R1JbM4wGbPLMzv078R4ZXHDPGvDttSEkA==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2026-10-06T06:10:31Z"
    mac: ENC[AES256_GCM,data:6wAGAwWL7jI/NEeswBnUiWoRty+bb5DEsGEAPXajKNNYCwtm5yuXBrcOI/q4w6H8FG+nqVAECkdpAOI3S9mjDfc2zlWAzfCJJdDic92RKPXOC0tOH/M1BBSGJ1fVdJGAZZT1yAVxmtYy9WohJpawv3kmsVv+JnD+TIqmZ1CJHXo=,iv:srs4o0olSwmraQQgmdPxb+4h/t6fHHeFKJ2z914kGgY=,tag:jjoIKhU8Tu9i5SJkvkfcCg==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.9.4
@@ -0,0 +1,76 @@
 # =============================================================================
 # TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
 # auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
 #
 # WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
 # (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
 # daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
 # register (peers.conf) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
 #
 # WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
 # source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`
 # would put the single credential every enrolled device depends on OUTSIDE the GitOps tree, where a
 # cluster rebuild silently loses it — and losing it means RE-ENROLLING EVERY DEVICE, because each
 # client profile pins this key's public half.
 #
 # *** THE REAL FILE IS COMMITTED WITH `privatekey: REPLACE-ME`, ALREADY ENCRYPTED. ***
 # That is deliberate, so the wiring is complete and green before the key exists: the resource is
 # already listed in kustomization.yaml and both overlays already carry the `vpn-ingress` decryption
 # patch. Your job is ONLY to replace the value (step 3 below). Until you do, the DaemonSet
 # CrashLoops with an explicit FATAL naming this file — the init container validates the value with
 # `wg pubkey` before writing wg0.conf, so a placeholder fails CLOSED and LOUDLY rather than
 # producing a server that silently cannot complete a handshake.
 #
 # ---------------------------------------------------------------------------------------------
 # EVERYTHING RUNS IN THE DEV CONTAINER. `wg`, `sops` and `qrencode` are all baked into the image
 # (infra/devcontainer/Containerfile) because that image IS the toolchain — CLAUDE.md §5, and the
 # devcontainer README's "Don't install the toolchain on the host". One shell, one toolchain, so
 # `wg genkey | sops set` is just a pipeline.
 # ---------------------------------------------------------------------------------------------
 #
 # HAND-OFF STEPS (operator — an agent must not mint a long-lived private key, CLAUDE.md §6).
 # Full procedure with verification: runbooks/vpn-remote-access.md step 2.
 #
 #  1. Generate the key STRAIGHT INTO this file. The plaintext never lands on disk and never
 #     reaches a terminal; `sops set` edits the already-encrypted document in place:
 #         ./infra/devcontainer/dev.sh run bash -c '
 #           cd /work
 #           umask 077
 #           sops set platform/vpn-ingress/wg-server-key.sops.yaml \
 #             "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
 #         '
 #     (Or edit it: `./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml`.)
 #
 #  2. Record BOTH halves in your PASSWORD MANAGER. The encrypted file is in git and Velero captures
 #     this namespace to R2, so this is belt-and-braces — but it is the only copy that survives
 #     losing the cluster AND the backups, and the key cannot be regenerated without invalidating
 #     every client profile.
 #         # public half (safe to display; it is what every client profile carries)
 #         ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d \
 #           --extract "[\"stringData\"][\"privatekey\"]" \
 #           platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
 #
 #  3. Confirm the value is a real key without printing it — `wg pubkey` succeeds only on a valid
 #     32-byte curve25519 private key, which is exactly the check the pod's init container runs:
 #         ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d \
 #           --extract "[\"stringData\"][\"privatekey\"]" \
 #           platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null \
 #           && echo "valid WireGuard key"'
 #
 #  4. PR it. Nothing else to wire: the resource line and both overlay decryption patches already
 #     exist. Flux rolls the DaemonSet and the pods leave CrashLoop.
 #
 # NOTE infra/vpn-ingress/new-peer.sh reads THIS FILE to derive each client profile's
 # `[Peer] PublicKey`, so there is no separate copy of the key to keep in step with anything.
 # =============================================================================
 apiVersion: v1
 kind: Secret
 metadata:
  name: wg-server-key
  namespace: vpn-ingress
 type: Opaque
 stringData:
  # The server's WireGuard private key: `wg genkey` output — 44 chars of base64 ending in '='.
  # NOT a passphrase, and NOT the public key. The init container validates it with `wg pubkey`
  # before writing wg0.conf, so a placeholder or a mangled value fails closed with a clear message.
  privatekey: REPLACE-ME
@@ -21,7 +21,7 @@ behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's
 DNS, and internet egress out the house WAN.
 Authorisation is one thing only: the device's **public key is in
-`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
+`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.
 > **An enrolled device is as privileged as a machine sitting in the house** — including the three
 > k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
@@ -62,53 +62,109 @@ lsmod | grep '^wireguard'
 grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf
 ```
-## Step 2 — create the server key (once)
+## Step 2 — fill in the server key (once)
-This is the **only** secret in the component, and the one thing that must never be generated in an
+The key is **SOPS-encrypted in this tree** (R-SEC-01, spec docs/07), not created by hand:
-agent session or pasted into a transcript. Generate it on your own machine:
+`platform/vpn-ingress/wg-server-key.sops.yaml`. It is already committed — encrypted, with a
 `REPLACE-ME` placeholder — and it is already listed in the component's `resources:` with a
 `decryption:` patch in both overlays. **Nothing needs wiring. You are only replacing one value.**
 Why in git rather than `kubectl create secret`: an out-of-band Secret puts the one credential every
 enrolled device depends on outside the GitOps tree, where a cluster rebuild silently loses it — and
 losing it means re-enrolling every device, because each client profile pins this key's public half.
 Everything below runs **inside the dev container** — `wg`, `sops` and `qrencode` are all baked into
 the image (CLAUDE.md §5: the toolchain lives only in the pinned container; don't install it on the
 host). One shell, one toolchain.
 **1. Generate the key straight into the encrypted file.** One pipeline: `wg genkey` makes it,
 `sops set` writes it into the already-encrypted document in place. The plaintext never lands on
 disk and never reaches a terminal.
 ```bash
-mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
+./infra/devcontainer/dev.sh run bash -c '
-umask 077
+  cd /work
-wg genkey > ~/.config/wireguard-home/server.key          # keep this; see "Back it up" below
+  umask 077
  sops set platform/vpn-ingress/wg-server-key.sops.yaml \
    "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
 '
 ```
-Load it into the cluster. **Do not echo it, do not pass it on an interactive command line that
+**2. Record it in your PASSWORD MANAGER.** Derive both halves from the file you just wrote:
 lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE.md §6):
 ```bash
-./infra/devcontainer/dev.sh run bash -c \
+# the PUBLIC half — safe to display, and what every client profile carries
-  'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \
+./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
-     --from-file=privatekey=/work/.cache/server.key'
+  platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
 # the PRIVATE half — paste into the password manager, then clear your scrollback
 ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
  platform/vpn-ingress/wg-server-key.sops.yaml'
 ```
-…where `/work/.cache/server.key` is a path you copied the key to (`.cache/` is not tracked; create
+The second command is the one place this key is deliberately shown. The encrypted file is in git and
-it if absent). Delete that copy as soon as the Secret exists: `shred -u .cache/server.key`.
+Velero captures this namespace to R2, so the password-manager copy is belt-and-braces — but it is
 the only copy that survives losing the cluster *and* the backups, and the key cannot be regenerated
 without invalidating every client profile.
-> **KEEP THE `.key` SUFFIX.** That filename is what keeps the key out of git: `.gitignore` line 11
+**3. Confirm the value is a real key**, without printing it — if `wg pubkey` prints a key, the
-> is `*.key`, and it is the **only** rule covering this path — `.cache/` itself is NOT ignored
+value is valid; if it errors, it is still wrong:
 > (`git check-ignore -v .cache/serverkey.txt` matches nothing). So a copy named `server.pem`,
 > `wg-server` or `privatekey` sitting in that directory is **committable**, and the gate's
 > `no-plaintext-secrets` stage does not know a WireGuard private key when it sees one. Verify before
 > you trust it: `git check-ignore -v .cache/<your-filename>` must print a rule, and
 > `git status --short` must not list the file.
 Confirm it exists **by key name only**:
 ```bash
-./infra/devcontainer/dev.sh run bash -c \
+./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
-  'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1'
+  platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null && echo "valid WireGuard key"'
 # expected: {"privatekey   -- a NAME. If you see base64 payload, you printed the key. Rotate it.
 ```
-**Why out of band and not SOPS.** An agent must not mint a long-lived private key, and a key that
+That is the same check the pod's init container runs, so a pass here predicts a pass there.
 passed through a session transcript would have to be treated as compromised. The trade-off is that
 this Secret is **not reconstructible from git** — so:
-**Back it up.** Keep `server.key` in your password manager. Velero's `daily` schedule excludes only
+**4. PR it, and merge.** Flux rolls the DaemonSet and the pods leave CrashLoop.
-`monitoring` and `unifi` (`platform/velero/helmrelease.yaml`), so this namespace *is* captured
+
-off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work
+If you would rather use an editor than `sops set`:
-again after a cluster rebuild — **losing it means re-enrolling every device.**
+`./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml` (replace
 `REPLACE-ME`, save, exit — it re-encrypts on write).
 ### What the pod tells you if this went wrong
 The init container validates the value with `wg pubkey` before writing `wg0.conf`, and names the
 specific failure in its log — so you never get a server that silently cannot complete a handshake:
 | pod log says | meaning |
 |---|---|
 | `still the committed REPLACE-ME PLACEHOLDER` | step 3 not done (the expected fresh-install state) |
 | `It is EMPTY` | the value was blanked |
 | `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
 | `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
 | `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
 | `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |
 ### What a healthy start looks like
 Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key:
 ```
 wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820.
 ```
 then the `wireguard` container brings the tunnel up:
 ```
 **** Found WG conf /config/wg_confs/wg0.conf, adding to list ****
 **** Activating tunnel /config/wg_confs/wg0.conf ****
 [#] ip link add dev wg0 type wireguard
 [#] wg addconf wg0 /dev/fd/63
 [#] ip -4 address add 10.90.0.1/24 dev wg0
 [#] ip link set mtu 1420 up dev wg0
 ```
 **Ignore the `chown: ... Operation not permitted` run before that**, and the image's own
 "Permissions could not be set ... we will not provide support for it". Capabilities are dropped to
 `NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs
 as root, and `wg-quick` proceeds. It is wrong about the cause and it does not matter.
 **A pod stuck `0/1 Running` with `Startup probe failed: Unable to access interface: No such device`
 means `wg-quick` tore the tunnel down** — read the `wireguard` container log for the `[#]` line it
 died on. That is the shape of a real failure; the probe is doing its job of refusing to call a
 server without a `wg0` "ready".
 ## Step 3 — forward udp/51820 on the router (once)
@@ -149,17 +205,37 @@ unauthenticated traffic either way. If the pod is Ready, the interface exists.
 ## Step 5 — enrol a device
 ```bash
-./infra/vpn-ingress/new-peer.sh pixel-8 2              # full tunnel  (LAN + WAN egress)
+# in the dev container, like everything else
-./infra/vpn-ingress/new-peer.sh thinkpad 3 --split     # split tunnel (LAN only)
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2'            # full tunnel
 ./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh thinkpad 3 --split'   # LAN only
 ```
-It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout,
+It derives the server's public key from `wg-server-key.sops.yaml` itself, so there is no local key
-prints a QR code if `qrencode` is installed, and prints the `[Peer]` stanza to add. Then:
+copy to keep in step with anything. Profiles land in `.cache/wg-peers/` — gitignored, and persistent
 because the repo is the container's only read-write mount.
-1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
+It generates the device's keypair, writes a `0600` client profile, prints a QR code, and prints the
 `[Peer]` stanza to add. It refuses to write a profile anywhere git does not ignore.
 Then:
 1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
   `# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
   comment is the only record of which stanza belongs to which device.
-2. PR → merge. Flux rolls the DaemonSet (~1 min).
+2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
   **content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
   which changes the pod template, which forces the roll. (It did **not** work this way at first —
   a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
   keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:
   ```bash
   ./infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
        -o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
   ```
   The generation must have **increased** since before the merge. If it is still `1`, the roll did
   not happen and `wg show` will list no peer no matter how long you wait.
 3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
   Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
 4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
@@ -199,7 +275,7 @@ if the router is not delivering the packets, and a `handshake` with no completed
 ## Revoking a device
-Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
+Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
 next reconcile. **Never re-use that address octet**: an old profile would silently start working
 again as a different device.