Merge branch 'main' into fix/vpn-ingress-l2-lease-collision

Merge pull request 'fix(oauth2-proxy): gate startup on Keycloak discovery, or a cold boot CrashLoops the proxy' (#70 ) from fix/oauth2-proxy-oidc-startup-race into main
Reviewed-on: #70
2026-10-18 04:15:58 +00:00 · 2026-10-18 04:14:52 +00:00 · 2026-10-17 22:47:30 -05:00 · 2026-10-17 21:58:12 -05:00 · 2026-10-10 16:13:46 +00:00 · 2026-10-10 16:09:53 +00:00
30 changed files with 1665 additions and 266 deletions
@@ -77,7 +77,7 @@ spec:
  storageClassName: longhorn
  resources:
    requests:
-      storage: 8Gi
+      storage: 16Gi
 ---
 # PVC plex-media (R-STATE)
 apiVersion: v1
@@ -351,7 +351,7 @@ exposure: public              # edge port-forward + public DNS; clients + Seerr
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
  - name: config
-    size: 8Gi
+    size: 16Gi
    class: longhorn           # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
    mountPath: /config        # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
                              # renderer default /data/config. Without this the PVC sat unused and Plex wrote to the
@@ -359,7 +359,7 @@ storage:
                                # The live /config contents were copied onto each PVC before this landed,
                                # so API keys and credentials other apps reference were preserved.
  # THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
-  # (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
+  # (192.168.2.30:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
  # /downloads, which is precisely where this app is ALREADY configured to write.
  #
  # WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
@@ -434,7 +434,7 @@ dependencies: []                # default-deny: reaches only DNS; arr reach it n
                                # for weeks and neither has any CNP naming the NFS server or port 2049
                                # (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
                                # egress gateway off this traffic (its selector is a podSelector), and
-                                # 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
+                                # 192.168.2.30 is inside the 192.168.0.0/16 excludedCIDR besides — see
                                # platform/vpn-egress/egress-steering.yaml, "BULK NFS".
 ingressRoutes:
  - host: qbittorrent.${BASE_DOMAIN}
@@ -234,26 +234,28 @@ data:
  WG_CLIENT_CIDR: "10.90.0.0/24"             # remote VPN clients; .1 = server. NOT a LAN range

  # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
-  # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
-  # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
+  # THE address of the NAS whose ZFS datasets back every `class: bulk` volume: the media library
+  # (Storage/plex, 8.33 T — plex, sonarr, radarr, qbittorrent) and the legacy Nextcloud archive
  # (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
  # `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
  # declared in the ServiceDefinition, because the path is identical in every environment and only
  # the ADDRESS is not.
  #
-  # WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
-  # The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
-  #   dev  nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
-  #        NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
-  #        /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
-  #        per the convention of the LAN block above.
-  #   prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
-  #        22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
-  #        the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
-  #        DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
-  # So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
-  # blast radius of shipping it unset, which is NOT the usual "fails to apply".
-  BULK_NFS_SERVER: "192.168.1.9"
+  # IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read "the two clusters are on
+  # OPPOSITE SIDES of pfSense": the datasets lived on a Proxmox host whose only interface was
+  # vmbr0 192.168.1.9/24 on pfSense's LAN, which prod (192.168.2.0/24, pfSense's WAN) could reach
+  # only through a DNAT at 192.168.2.102. Both pools were physically moved onto a standalone
+  # TrueNAS at 192.168.2.30 — prod's OWN /24 — so there is no crossing to arrange and no candidate
+  # values to choose between. Measured 2026-10-07: 192.168.2.102:2049 no longer answers;
+  # 192.168.2.30:2049 and :111 do. There is now exactly ONE storage host and one correct value;
+  # this stays a knob only because the renderer reads it per-environment.
+  #
+  # *** CHANGING THIS VALUE IS NOT A PLAIN RECONCILE. *** A bound PersistentVolume's spec.nfs is
+  # immutable — measured by server dry-run 2026-10-07:
+  #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+  # so Flux's apply FAILS rather than updating, and the pods keep their old mount. The PVs must be
+  # deleted and recreated; the operator sequence is docs/BULK-NFS-STORAGE.md §10.
+  BULK_NFS_SERVER: "192.168.2.30"

  # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
  # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
@@ -279,15 +279,15 @@ patches:
      - op: replace
        path: /data/CRAFTY_L4_VIP
        value: "192.168.1.55"
-      # Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
-      # (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
-      # Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
-      # Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
-      # end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
-      # change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
+      # Bulk (external NFS) storage host. THIS VALUE IS INERT — dev is decommissioned, its nodes
+      # (192.168.1.40/.41/.42) no longer exist, and the Proxmox host this used to name (192.168.1.9)
+      # went with the pools it held. It is restated per this overlay's "supply every knob"
+      # convention, not because anything consumes it. It carries the real address so that no file
+      # in the tree renders a host that is gone; it has NOT been proven from a dev node, because
+      # there is no dev node to prove it from.
      - op: replace
        path: /data/BULK_NFS_SERVER
-        value: "192.168.1.9"
+        value: "192.168.2.30"
      # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
      # barmanObjectStore and Velero's backupStorageLocation write to.
      #
@@ -540,60 +540,48 @@ patches:
        path: /data/LAN_CIDR
        value: "192.168.2.0/24"
      # ------------------------------------------------------------------------------------------
-      # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
+      # BULK (EXTERNAL NFS) STORAGE HOST — the TrueNAS at 192.168.2.30.
      # ------------------------------------------------------------------------------------------
      # Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
-      # radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
+      # radarr, qbittorrent) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
      #
-      # THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
-      # vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
-      # 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
-      # 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
-      # outcomes give DIFFERENT values here:
-      #     port-forward 192.168.2.102:2049 -> 192.168.1.9:2049   =>  "192.168.2.102"
-      #     a WAN->LAN route to 192.168.1.0/24                    =>  "192.168.1.9"
-      # Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
-      # ever became reachable — mounts something unintended. So it stays a placeholder.
+      # THIS WAS THE REPO'S ONE LOAD-BEARING PLACEHOLDER, AND THE REASON IT COULD BE IS NOW GONE.
+      # The datasets lived on a Proxmox host with a single interface on pfSense's LAN
+      # (192.168.1.9), which prod could reach only through a DNAT at 192.168.2.102 — so the correct
+      # value here depended on an undecided operator choice, and guessing it would have authored a
+      # cluster that mounts nothing. Both pools were physically moved onto a standalone TrueNAS at
+      # 192.168.2.30, prod's OWN /24. No crossing to arrange, no candidates to choose between.
+      # Measured 2026-10-07: 192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and :111 do.
      #
-      # WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
-      # NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
-      # over v3 was in part this decision's cost: v3 would have needed four ports and three of them
-      # on ephemeral numbers.
-      #
-      # WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
-      # Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
-      # the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
-      # address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
-      # 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
-      # addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
-      #
-      # *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
-      # A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
-      # placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
-      # failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
-      # TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
-      # ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
-      # apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
-      # a revision containing the bulk volumes; the dev overlay renders a working value, which is
-      # where these manifests are meant to be proven first.
+      # ONE PORT IS STILL ALL IT NEEDS: tcp/2049. The PVs mount nfsvers=4.1 and NFSv4 carries mount
+      # and lock in-protocol — no rpcbind/111, no mountd, no statd. The NAS advertises
+      # `+3 +4 +4.1 +4.2` (/proc/fs/nfsd/versions, measured), so 4.1 negotiates.
      - op: replace
        path: /data/BULK_NFS_SERVER
-        # 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
-        # host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
-        # reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
-        # keeps the base value 192.168.1.9.
+        # VERIFIED ON THE NAS 2026-10-07, before any cluster touched it (`exportfs -v`):
+        #   /mnt/Storage/plex       192.168.2.10/.11/.12  rw,sec=sys,secure,root_squash,no_all_squash
+        #   /mnt/Storage/nextcloud  192.168.2.10/.11/.12  ro,sec=sys,secure,root_squash,no_all_squash
        #
-        # PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
-        #   - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
-        #   - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
-        #     carries no_root_squash, so a root write would have passed and proved nothing, while
-        #     LSIO images default to PUID 911 and would have gone Ready then failed every import
-        #     with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
-        #   - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
-        #     real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
-        # pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
-        # so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
-        value: "192.168.2.102"
+        # no_all_squash is the flag the entire permission model rests on: client uids pass through
+        # untouched, so the arrs' PUID=1000 arrives as the owner of a 0755 1000:3000 tree. sec=sys
+        # keeps that numeric rather than principal-mapped. The export deliberately does NOT carry
+        # no_root_squash, which the Proxmox one did: nothing here needs it, because every write to
+        # the tree happens after the LSIO drop to uid 1000 and the images only chown their Longhorn
+        # /config. Both subPath directories (plex/downloads, nextcloud/nextcloud) already exist at
+        # 0755 1000:3000, so a squashed root can still traverse them for subPath resolution.
+        #
+        # The ro= posture on the archive is now SERVER-enforced, which the old export also managed.
+        # The media tree's read-only half (plex) stays a CLIENT-side guarantee — all four media pods
+        # can land on any node, so the server sees the same three addresses for reader and writers
+        # alike and cannot tell them apart. Unchanged by the move; see docs/BULK-NFS-STORAGE.md §5.
+        #
+        # *** CHANGING THIS VALUE REQUIRES DELETING AND RECREATING THE PVs. *** A bound PV's
+        # spec.nfs is immutable — measured by server dry-run 2026-10-07:
+        #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+        # so Flux's apply FAILS rather than updating and the pods keep their old, dead mount. The
+        # operator sequence is docs/BULK-NFS-STORAGE.md §10. Retain + the renderer's pre-set
+        # claimRef make it safe: no data is touched and each recreated PV rebinds its own PVC.
+        value: "192.168.2.30"
      # ==========================================================================================
      # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
      # ==========================================================================================
@@ -0,0 +1,103 @@
+---
+date: 2026-10-06
+branch: fix/peers-configmap-content-hash
+title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
+---
+
+## What changed
+
+The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
+2026-10-06, with the peer already on `main`:
+
+| observation | value |
+|---|---|
+| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
+| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
+| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
+| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
+| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
+| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
+| init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
+
+The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
+fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
+no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
+into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
+would have had **no effect at all** until something unrelated happened to restart the pods. For a
+*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
+see a green PR, and the device would keep working.
+
+My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
+never true.
+
+**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
+*"The generated name carries kustomize's content hash, so editing any runbook rolls the server
+automatically and a stale projection is impossible."*
+
+So the register becomes a **content-hashed generator**:
+
+- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
+  plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
+  carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
+  documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
+  stripped.
+- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
+  ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
+  `nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
+  the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
+  `wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
+  by building the output and reading the volumes block.
+- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
+  register is now **fatal** instead of skipped. The old guard was
+  `[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
+  duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
+  as a defect in its own right.
+- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
+  `networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
+  operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
+- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
+  `generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
+  merge.
+
+Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
+
+No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
+`decisions/**`), so this merges on a green gate with no override.
+
+## What I verified
+
+Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
+
+- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
+  namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
+  `namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
+  bearing, measured both ways.
+- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
+  append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
+  and reversible, so an edit necessarily changes the pod template.
+- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
+  `peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
+  `main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
+- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
+  NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
+  transformer did not rewrite it).
+- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
+  "append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
+  real `.2` now occupies it — the first time the duplicate check has been exercised against real
+  data rather than against its own example.
+- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
+  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
+  as rc 3.**
+
+## What I did NOT verify
+
+- **That the roll actually happens on the cluster.** The whole point of the change is unproven
+  until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
+  `wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
+- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
+  port-forward is reportedly configured but has never been exercised by a packet I can see, and I
+  have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
+- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
+  should now roll the pods and drop the peer; untested.
+- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
+  `prune: true` on the Kustomization, not observed.
@@ -0,0 +1,85 @@
+---
+date: 2026-10-07
+branch: fix/bulk-nfs-truenas
+title: Repoint bulk NFS at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
+---
+
+## What changed
+
+Both ZFS pools were physically moved off the Proxmox host onto a standalone TrueNAS Community
+Edition at **192.168.2.30**, which sits on prod's own `192.168.2.0/24`. `BULK_NFS_SERVER` moves from
+`192.168.2.102` (the pfSense DNAT prod used to mount through) to `192.168.2.30`.
+
+- `clusters/prod/kustomization.yaml` — `BULK_NFS_SERVER` → `192.168.2.30`. The comment block that
+  justified the DNAT, and the "NOT YET DECIDED / THIS PLACEHOLDER IS LOAD-BEARING" banner, are
+  replaced with the measured export configuration and the recreate requirement below.
+- `clusters/base/cluster-vars.yaml` — base default `192.168.1.9` → `192.168.2.30`. That host no
+  longer exists.
+- `clusters/dev/kustomization.yaml` — same; marked explicitly **inert**, since dev is decommissioned
+  and its nodes are gone. It carries the real address only so nothing in the tree renders a dead one.
+- `docs/BULK-NFS-STORAGE.md` — §1 and §5 rewritten for TrueNAS (exports, service settings, the ZFS
+  `sharenfs` properties that had to be cleared, and the ACL hazard); §7 rewritten now that the
+  placeholder is gone; §9 retargeted from dev to prod; **new §10** documents the PV
+  delete-and-recreate. §2/§3/§4/§6 are unchanged by the move.
+- `catalog/qbittorrent.yaml`, `platform/monitoring/node-pressure-dashboard.yaml` — two comment/prose
+  references to `192.168.2.102` as the live export address.
+
+**The headline operational fact: repointing this knob is NOT a plain reconcile.** A bound PV's NFS
+source is immutable, so Flux's apply fails rather than updating and the pods keep their old, dead
+mount. The five PVs must be deleted and recreated. That sequence is §10 and is an operator action;
+it is **not** performed by this PR.
+
+## What I verified
+
+**On the NAS (192.168.2.30), measured 2026-10-07 by the operator at my request:**
+
+- `zfs list` — `Storage` → `/mnt/Storage`, `Storage/plex` → `/mnt/Storage/plex`,
+  `Storage/nextcloud` → `/mnt/Storage/nextcloud`. The PV paths resolve unchanged.
+- Ownership intact: `/mnt/Storage/plex`, `tv/`, `downloads/`, `movies/`,
+  `downloads/complete/`, `nextcloud/nextcloud/` all `drwxr-xr-x 1000:3000`. `movies/` 1171 entries
+  and `tv/` 107, against 1165 / 107 measured on Proxmox.
+- `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`), `aclinherit=passthrough`, and no
+  non-trivial ACLs — mode bits still govern.
+- Three stale `sharenfs` dataset properties (`plex`, `nextcloud`, `docker-volumes`) carried over from
+  Proxmox; cleared with `zfs inherit`, re-checked empty.
+- `exportfs -v`: both shares to `192.168.2.10/.11/.12` with `sec=sys,secure,root_squash,no_all_squash`,
+  `rw` on plex and `ro` on nextcloud.
+- `/proc/fs/nfsd/versions` → `+3 +4 +4.1 +4.2`, so the PVs' `nfsvers=4.1` negotiates.
+- **A defect was found and fixed during setup:** the plex share listed `192.168.2.13` — an address no
+  node holds — and so omitted `192.168.2.12` (k3s-prod-3). Corrected and re-verified.
+
+**From the cluster:**
+
+- Node InternalIPs are `192.168.2.10/.11/.12` (`kubectl get nodes`), confirming `.13` was wrong.
+- All five NFS PVs currently name `192.168.2.102`.
+- `192.168.2.30:2049` and `:111` accept connections; `192.168.2.102:2049` does not.
+- **`spec.nfs` immutability, by server-side dry-run** — this is the finding that shapes the change:
+  `spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation`.
+- The pods report `Ready` while their mounts are dead: a `stat /media` answers from attribute cache,
+  but a `readdir` blocks indefinitely (the PVs mount `hard`). Hence "scale to zero first" in §10.
+
+**Gate:** see the PR; run in the pinned toolbox from this worktree.
+
+## What I did NOT verify
+
+- **Nothing has mounted 192.168.2.30.** No pod has touched the new server — the PVs still point at
+  the old address and cannot be updated in place. Every §9 probe (numeric-id passthrough, the uid-1000
+  write, the hardlink/EXDEV check, Plex's read-only posture) is **outstanding**, not passing. The
+  equivalents passed against the old host on 2026-09-12; that is not evidence about this one.
+- **The §10 sequence is reasoned, not executed.** Retain + the renderer's pre-set `claimRef` are read
+  from the manifests and from `catalog/render.py`; I have not watched a PVC rebind after a PV delete
+  in this cluster.
+- **`root_squash` is a behaviour change from the Proxmox export**, which carried `no_root_squash`. The
+  argument that nothing needs root on the tree (LSIO drops to 1000; the images chown only their
+  Longhorn `/config`; both subPath directories already exist at 0755) is reasoned from the images and
+  the measured permissions. It has not been exercised. If a mount or subPath resolution fails with
+  EACCES, Maproot `root:root` reproduces the old export and is the first thing to try.
+- **Whether NFSv3 can be disabled.** Enabled Protocols is `NFSv3, NFSv4`; the PVs only need v4.1, so
+  narrowing would close the rpcbind/mountd/statd surface. Not done — it is a change to make after the
+  cutover is proven, not during it.
+- **Whether pfSense / the legacy stack is actually retired.** I measured only that
+  `192.168.2.102:2049` stopped answering. The other legacy consumers of these datasets
+  (`192.168.1.6`, `10.123.1.10`, `192.168.1.18`) were dropped from the new exports; if any is still
+  live, it has lost access. Not checked.
+- **L2/L3 conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline subset
+  only. Nothing in the gate exercises an NFS mount.
@@ -0,0 +1,99 @@
+---
+date: 2026-10-17
+branch: fix/oauth2-proxy-oidc-startup-race
+title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
+---
+
+## What changed
+
+Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
+discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
+serving yet, so the shared Gateway answers 503 and the proxy dies:
+
+```
+[provider.go:55] Performing OIDC Discovery...
+[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
+             failed to discover OIDC configuration: unexpected status "503": no healthy upstream
+```
+
+It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
+replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
+retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
+is already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+
+`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
+polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
+accrue CrashLoopBackOff backoff and retries promptly, which is the point.
+
+**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
+case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
+for a pod wedged in `Init` forever, which is not an improvement.
+
+## Why this shape and not the alternatives
+
+* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
+  example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
+  pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
+  at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
+  "reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
+* **No new base image.** The chart already runs an `alpine` init container for its built-in
+  `initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
+  `3.22` (verified to exist upstream) rather than the chart's own `latest`.
+* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
+  this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
+  the chart already has a hook for.
+* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
+  more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
+  oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
+  the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
+  the operator out of all of them, so the reversible, additive option wins.
+
+The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
+with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
+and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
+silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
+the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.
+
+## What I verified
+
+* **The failure is measured, not theorised.** The log above is from the live pod
+  (`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
+  with `--previous` after the cold boot. The surviving replica's log shows the same startup
+  succeeding once Keycloak was up, which is what identifies this as a race rather than a config
+  fault.
+* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
+  clean and both replicas have been healthy since.
+* **The chart key is real**, at the pinned version range: `helm search repo` resolves
+  `>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
+  `extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
+* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
+  assumed from memory.
+* The edited HelmRelease parses as YAML.
+
+## What I did NOT verify
+
+* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
+  seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
+  replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
+  seven apps, so **do it when you can watch it**, not unattended.
+* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
+  oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
+  from the measured failure, not demonstrated against it.
+* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
+  `ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
+  failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
+  degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
+  anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
+  init container's log on first apply**: it should print "Keycloak discovery reachable" within
+  seconds on a warm cluster, not the give-up message.
+* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
+  re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
+  good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
+  (`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
+  Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
+  this should hold — unverified.
+* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
+  not a measured bound.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
+  subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
@@ -0,0 +1,137 @@
+---
+date: 2026-10-17
+branch: fix/vpn-ingress-l2-lease-collision
+title: "A node cannot reach a VIP it announces — so the VPN ingress node must announce nothing"
+---
+
+## What changed
+
+The first device enrolled on the remote-access VPN handshook, resolved names, and could not open a
+single connection to any cluster web app. The tunnel was not at fault. The fault is a property of
+Cilium L2 announcement that nothing in this repo knew about:
+
+**A node cannot reach a LoadBalancer VIP whose `CiliumL2AnnouncementPolicy` lease it holds itself.**
+The announcer answers ARP for the VIP on behalf of other LAN hosts but not for its own request, so
+on the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP is dropped
+before it leaves the node.
+
+`platform/vpn-ingress` is uniquely exposed because it runs WireGuard with `hostNetwork: true` and
+MASQUERADEs remote clients to the node's own address — a VPN client's packets take the node's LAN
+path and inherit its ARP failures. Ordinary pods never do. The router's `udp/51820` forward pointed
+at `k3s-prod-1`, which held `cilium-l2announce-gateway-cilium-gateway-shared`, so **all 16 `shared`
+Gateway hostnames were dead over the VPN** — while PiHole DNS kept answering, because
+`192.168.2.23` is announced by *two* Services and `k3s-prod-2` held the other lease. Names resolved
+and every app hung on connect.
+
+Seven files:
+
+* **`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`** — new
+  `node_baseline_vpn_ingress_node`; the matching node gets
+  `homelab.keithjackson.dev/vpn-ingress=true`, and a second task **removes** the label from any node
+  that no longer matches, so moving the value moves the label rather than adding a second one. Same
+  shape as `roles/vpn_egress_gateway`'s `egress-gateway` label. Tagged `vpn-ingress`; skipped
+  entirely when the value is empty.
+* **`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`** — set to `k3s-prod-1`, with a
+  `[parity]` note that it must equal the router's forward.
+* **`platform-config/cilium/l2-announcement-policy.yaml`** — `nodeSelector` becomes a
+  `matchExpressions` list keeping `kubernetes.io/os: linux` and adding
+  `homelab.keithjackson.dev/vpn-ingress NotIn ["true"]`. The VPN node then holds no lease, so it
+  ARP-resolves every VIP. This fixes **all** VIPs for VPN clients (Gateway, crafty, plex, pihole,
+  unifi), not just the one that happened to collide.
+* **`platform/monitoring/kube-prometheus-stack.yaml`** — the label added to kube-state-metrics'
+  `metricLabelsAllowlist`, which is what lets the invariant be watched at all.
+* **`platform/monitoring/alerts.yaml`** — `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) joins
+  `kube_lease_owner` against `kube_node_labels` via `label_replace`, plus
+  `VpnIngressNodeLabelMissing` (warning, 30m) as its watchdog.
+* **`platform/monitoring/tests/vpn-ingress-l2.test.yaml`** — four promtool cases: fires on the real
+  collision, quiet in the fixed state, quiet across a transient 11-minute lease move (pinning the
+  `for: 15m`), and the label-absent case asserting the critical alert is inert *and* the watchdog
+  fires.
+* **`runbooks/vpn-ingress-l2-collision.md`** (new, registered in `runbooks/kustomization.yaml`),
+  **`runbooks/vpn-remote-access.md`**, **`platform/vpn-ingress/{daemonset.yaml,README.md}`** — step 3
+  of the setup runbook said *"pick any node — all three run an identical listener"* and *"node loss =
+  failover is this one line"*. Both were false and the first caused this. Failover is now documented
+  as two ordered changes.
+
+**Design note — why a node label and not a `cluster-vars` knob.** The first attempt added
+`WG_INGRESS_NODE` to `clusters/base/cluster-vars.yaml` and substituted it into both the policy and
+the alert. The alert half is **impossible**: `platform/monitoring/tests/extract_rules.py` rejects any
+alert whose `expr` contains a `${...}` placeholder, because the rule it would unit-test is not the
+rule deployed. Rather than duplicate the node name into the alert as a literal (two sources that can
+skew), the label is readable from both sides — the selector matches on it and `kube_node_labels`
+carries it into Prometheus. One source of truth, and as a side effect the change touches no
+sensitive path, so it merges on a green gate rather than needing a human review.
+
+The DaemonSet deliberately gained **no** probe for this. A readinessProbe on VIP reachability would
+wedge a rollout behind an unrelated Gateway outage (`maxUnavailable: 1`); a liveness probe would tear
+down a tunnel carrying LAN traffic correctly.
+
+## What I verified
+
+Measured live against prod, `--context k3s-prod`, on 2026-10-08:
+
+* **The whole tunnel is healthy**, which is what made this hard: `wg show` on `k3s-prod-1` shows the
+  peer with a real endpoint and bidirectional transfer (346 KiB in / 200 KiB out);
+  `net.ipv4.ip_forward=1`; `wg0` up at `10.90.0.1/24` MTU 1420; `iptables -t nat -S POSTROUTING`
+  carries the `MASQUERADE` rule with 26 packets; `FORWARD -i/-o wg0` ACCEPT with 108 / 171 packets;
+  DaemonSet generation 2, 3/3 ready on the content-hashed `wg-peers-m9kcckcb5t`.
+* **The ARP failure and its symmetry.** On `k3s-prod-1`: `ip neigh show 192.168.2.20` → `FAILED`,
+  while `.21`/`.23` resolve to prod-2's MAC `04:0e:3c:96:b0:94`. On prod-2/prod-3, `.20` resolves to
+  prod-1's `bc:e9:2f:7e:c4:a6`. TCP agrees: `.20:443` and `.20:80` FAIL from prod-1, OK from the
+  other two. Repeated across 3 nodes × 5 VIPs — each node reached every VIP except the ones it alone
+  announced (prod-2 ✗ `.22`/`.21`; prod-3 ✗ `.24`).
+* **Not an artefact of probing from the host netns.** `curl --interface wg0` from prod-1 fails the
+  same way, and a neighbour entry is node-global state, so the forwarded path cannot differ in ARP
+  resolution.
+* **The Gateway itself was never broken:** from the workstation (192.168.2.133)
+  `home.keithjackson.dev` → `200`, `plex.home.keithjackson.dev` → `401`, both at `192.168.2.20`.
+  From prod-1 the router (`.1`) and TrueNAS (`.30`) are `REACHABLE`.
+* **`NotIn` semantics, empirically** — the basis for the fail-back claim in the policy's comment:
+  `kubectl get nodes -l 'kubernetes.io/hostname notin (k3s-prod-1)'` returns prod-2 and prod-3, and
+  `NotIn` on a key no node carries returns **all three**. So an unapplied label excludes nothing and
+  degrades to the old behaviour rather than taking every VIP off the LAN.
+* **The alert metric is really scraped:** `kube_lease_owner` is served by kube-state-metrics (55
+  series, every `cilium-l2announce-*` lease with its holder in `lease_holder`), and `kube_node_labels`
+  already carries the two `egress-*` labels through the same allowlist (`k3s-prod-3` shows
+  `egress-gateway=true`), so the third entry follows a proven path.
+* **The four promtool cases pass** (`platform/monitoring/tests/run.sh` → `SUCCESS`, 55 rules
+  checked), including that `VpnIngressNodeAnnouncesL2VIP` is inert when the label is absent.
+* All edited YAML parses; the rendered `nodeSelector` is the intended two-expression list.
+
+## What I did NOT verify
+
+* **The fix is UNAPPLIED and therefore UNPROVEN end to end.** Applying it needs a mutating
+  `kubectl`/`flux reconcile` plus an Ansible run, which are operator actions (CLAUDE.md §5). I have
+  **not** observed `k3s-prod-1` resolve `192.168.2.20` after the exclusion lands, nor a VPN client
+  reaching an app. The prediction rests on the measured symmetry (non-announcing nodes resolve every
+  VIP, 15/15 data points), not on a test of the changed state. **Verify after merge, in this order:**
+  re-run `node_baseline --tags vpn-ingress`; `kubectl get nodes -L
+  homelab.keithjackson.dev/vpn-ingress` (exactly one node); `kubectl -n kube-system get lease | grep
+  l2announce` (no holder is that node); `ip neigh show 192.168.2.20` from the prod-1 wireguard pod
+  (expect a `lladdr`, not `FAILED`); then a client.
+* **That Cilium actually releases a lease when its holder stops matching the `nodeSelector`.** I
+  assumed re-election and did not test it. If Cilium keeps renewing, the affected Leases must be
+  deleted once by hand to force an election. Check this first if the fix appears not to work.
+* **The brief interruption on apply.** Moving the Gateway and `pihole-dns-udp` leases off prod-1 will
+  blip `192.168.2.20` and `192.168.2.23` for as long as re-election plus gratuitous ARP takes. Not
+  measured, and I did not test whether in-flight LAN sessions survive it.
+* **The Ansible tasks have never been executed.** They parse as YAML and follow
+  `roles/vpn_egress_gateway`'s shape, but neither the label task nor the label-removal task has run,
+  so the `changed_when`/`failed_when` conditions and the `k3s kubectl label ... -` removal path are
+  unproven. The removal task's `failed_when` tolerates `'not found'` in stderr; that string was
+  assumed from kubectl's behaviour, not observed.
+* **The residual off-repo coupling is NOT closed and cannot be, as built.** The router's forward is
+  unreadable from the cluster. If someone re-points it without moving the label, the alert watches
+  the wrong node and stays silent while the VPN breaks identically against a different set of VIPs.
+  Only the runbook guards it. Closing it needs a WireGuard exporter publishing which node is
+  actually terminating peer traffic (`wg show` knows — only the forwarded node has a peer
+  `endpoint`); listed as "Not built" in the component README and **recommended follow-up, not done
+  here**.
+* **Why Cilium behaves this way** — intended, a known upstream issue, or fixed in a later version. I
+  measured the behaviour and did not read Cilium's source or issue tracker, so I cannot say whether
+  an upgrade would remove the need for this exclusion.
+* **`192.168.2.21:8443` (unifi) fails from all three nodes**, including ones where ARP resolves, so
+  it is a different, pre-existing fault. Not investigated, not addressed.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1 offline
+  subset only; none of the above is covered by a test that runs in the gate.
+* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).
@@ -1,61 +1,43 @@
 # Bulk NFS storage — the media library and the legacy Nextcloud archive

-The operator-facing companion to the `class: bulk` storage entries in `catalog/{plex,sonarr,radarr,nextcloud}.yaml`.
-Everything here was **measured** on the storage host on 2026-09-07, over read-only SSH; nothing is
-assumed. It resolves the long-standing **OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
+The operator-facing companion to the `class: bulk` storage entries in
+`catalog/{plex,sonarr,radarr,qbittorrent,nextcloud}.yaml`.
+
+**Rewritten 2026-10-07 for the move off Proxmox onto a standalone TrueNAS at 192.168.2.30.** The
+layout (§2), the mount design (§3, §4) and the uid model (§6) are unchanged by the move and were
+measured on the old host on 2026-09-07. Everything about the storage host itself — §1, §5, §10 —
+was re-measured on the NAS on 2026-10-07. Nothing here is assumed. It resolves the long-standing
+**OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
 case that actually exists here — attaching datasets that already hold data — and leaves dynamic bulk
 provisioning unimplemented, because nothing needs it.

 ---

-## 1. The storage host, and why prod cannot reach it
+## 1. The storage host

-The bulk data lives on the **Proxmox host's ZFS pool**. That host has exactly one interface:
+The bulk data lives on a **standalone TrueNAS** (Community Edition) at **192.168.2.30** — the same
+`192.168.2.0/24` as prod's nodes (`192.168.2.10/.11/.12`), so it is reached directly, with no NAT
+and no routing decision.

-```
-vmbr0  192.168.1.9/24        default via 192.168.1.1   (pfSense LAN)
-```
-
-| | nodes | reaches 192.168.1.9? |
-|---|---|---|
-| **dev** | 192.168.1.40 / .41 / .42 | **Yes** — same L2 segment, and it already holds an NFS export from this host (`/mnt/Storage/k8s-volumes`, in `/etc/exports`) |
-| **prod** | 192.168.2.10 / .11 / .12 | **No** — 192.168.2.0/24 is pfSense's **WAN**. Ports 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. 2049 and 111 do not. |
-
-Opening the storage path for prod is a **pending operator decision**. Its two candidate outcomes give
-different values for the `BULK_NFS_SERVER` cluster-var:
-
-| outcome | `BULK_NFS_SERVER` for prod |
-|---|---|
-| port-forward `192.168.2.102:2049` → `192.168.1.9:2049` | `192.168.2.102` |
-| a WAN→LAN route to 192.168.1.0/24 | `192.168.1.9` |
-
-Until it lands, `clusters/prod` holds `TBD-PROD-BULK-NFS-SERVER`. **See §7 for why that placeholder
-behaves differently from the repo's other ones.**
+**This section used to be the hard part of this page.** The datasets were on the Proxmox host, whose
+only interface was `vmbr0 192.168.1.9/24` on pfSense's **LAN**, while prod sits on pfSense's **WAN**.
+tcp/2049 did not traverse that, so prod mounted through a DNAT at `192.168.2.102`, and which address
+belonged in `BULK_NFS_SERVER` depended on an undecided operator choice. Both pools were physically
+moved on 2026-10-07 (`docs/plans/proxmox-to-n5-air-migration.md`, Phase C/D) and all of that is
+simply gone. Measured 2026-10-07: **192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and
+:111 do.**

 ### It only needs ONE port

 The PVs mount `nfsvers=4.1`. **NFSv4 carries mount and lock in-protocol**, so the only thing that
 must be reachable is **tcp/2049** — no rpcbind/111, no mountd, no statd. NFSv3 would have needed
-four ports, three of them on ephemeral numbers, which is a materially worse firewall story. The
-server advertises `+3 +4 +4.1 +4.2` (`/proc/fs/nfsd/versions`), so v4.1 is available and is the
-conservative floor.
+four ports, three of them on ephemeral numbers. The NAS advertises `+3 +4 +4.1 +4.2`
+(`/proc/fs/nfsd/versions`, measured 2026-10-07), so 4.1 is available and is the conservative floor.

-### The export's client list is the SAME under both outcomes — measured, not assumed
-
-**pfSense DNATs the existing port-forward WITHOUT source NAT.** Connecting from this workstation
-(192.168.2.132) to `192.168.2.102:22`, the storage host logged the peer as `192.168.2.132`:
-
-```
-SSH_CONNECTION=192.168.2.132 39768 192.168.1.9 22
-ss -tn  ->  192.168.1.9:22   192.168.2.132:39768
-```
-
-The original WAN-side source address survives the translation (return traffic finds its way back via
-the host's default route, 192.168.1.1). So under **either** outcome the NFS server will see prod's
-**real** node addresses — 192.168.2.10/.11/.12 — and the `sharenfs` change in §5 is the same either
-way. **It does not have to wait for the routing decision.**
-
---
+**NFSv4 is not on by default and its absence does not look like a storage problem.** TrueNAS ships
+NFSv3-only until NFSv4 is added to *Services → NFS → Enabled Protocols*. Without it the mount never
+completes and the pods sit in `ContainerCreating` — no permission error, nothing in the app logs,
+just no volume.

 ## 2. The layout, as found

@@ -107,16 +89,18 @@ why it is mounted read-only; see §6.
 | plex | `/mnt/Storage/plex` | `/media` | — | **ro** | ReadOnlyMany |
 | sonarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
 | radarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
+| qbittorrent | `/mnt/Storage/plex` | `/downloads` | `downloads` | rw | ReadWriteMany |
 | nextcloud | `/mnt/Storage/nextcloud` | `/mnt/legacy-nextcloud` | `nextcloud` | **ro** | ReadOnlyMany |

 In-app paths that follow: Plex libraries at `/media/movies`, `/media/tv`, `/media/photos`; Sonarr
 Root Folder `/media/tv`; Radarr Root Folder `/media/movies`; both importing from
 `/media/downloads/complete/{tv-sonarr,radarr}`.

-**All three media consumers use the identical mountPath on purpose.** When qbittorrent is eventually
-given the same export (it is not in this change), the arrs and the download client will already agree
-on absolute paths, which is the second precondition for hardlinked imports — otherwise every import
-needs a Remote Path Mapping and quietly falls back to copying when one is missing.
+**The three library consumers use the identical mountPath on purpose**, and qbittorrent has since
+been given the same export (narrowed by `subPath: downloads`, presented at `/downloads` to match its
+existing `Session\DefaultSavePath`). The arrs and the download client therefore agree on absolute
+paths, which is the second precondition for hardlinked imports — otherwise every import needs a
+Remote Path Mapping and quietly falls back to copying when one is missing.

 **`/media` and not `/data`, and that is not cosmetic.** The arrs' Longhorn config PVC renders at the
 renderer default `/data/config`. A media mount at `/data` would be mounted **over** it and mask the
@@ -147,62 +131,123 @@ Narrowing that would cost hardlinks, which is the worse trade.

 ---

-## 5. The export change the operator must apply — STRICTLY ADDITIVE
+## 5. The exports, as configured on TrueNAS — and the settings that decide permissions

-**Do not apply this from the repo.** It is an operator step on the storage host, recorded here so the
-exact value is reviewable. The existing consumers **must keep working**: 192.168.1.6, 10.123.1.10 and
-192.168.1.18 on the media tree, and 192.168.1.6 and 192.168.1.5 on the Nextcloud dataset, are the
-live non-k3s prod stack.
+**Do not apply this from the repo.** It is operator configuration on the NAS, recorded here so the
+exact values are reviewable. Everything below was **measured on 192.168.2.30 on 2026-10-07**, after
+the move.

-Current values (`zfs get sharenfs Storage`):
+### The exports

 ```
-Storage/plex       rw=@192.168.1.6:@10.123.1.10:@192.168.1.18,no_subtree_check,no_root_squash
-Storage/nextcloud  [email protected]:@192.168.1.5,no_subtree_check,no_root_squash
+/mnt/Storage/plex       192.168.2.10 192.168.2.11 192.168.2.12
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)
+/mnt/Storage/nextcloud  192.168.2.10 192.168.2.11 192.168.2.12
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,root_squash,no_all_squash)
 ```

-### `Storage/plex` — append the six k3s node addresses to the existing `rw=` list
+Per-node addresses rather than `@192.168.2.0/24`, following the precedent of the old host. A `/24`
+would work and would survive adding a node; it also grants the whole subnet.
+
+**All three nodes must appear on BOTH shares.** Any media pod can be scheduled on any node, so a
+missing node is a latent failure, not an immediate one: it surfaces the first time a pod reschedules
+there, as `access denied by server while mounting`, with the cause weeks stale. One was caught
+exactly this way during setup — the plex share listed `192.168.2.13`, an address no node holds, and
+so was missing `192.168.2.12` (k3s-prod-3). Nothing was failing at the time.
+
+### The option flags ARE the permission contract
+
+| flag | why it matters here |
+|---|---|
+| **`no_all_squash`** | The one the whole model rests on. Client uids pass through untouched, so the arrs' `PUID=1000` arrives as the owner of a `0755 1000:3000` tree. In the UI this is **Mapall User/Group left EMPTY** — setting either squashes every client to one identity and erases the reader/writer distinction entirely. |
+| **`sec=sys`** | AUTH_SYS. Numeric uid/gid, not Kerberos principals. §6 is only true because of this. |
+| **`root_squash`** | Maproot left empty. The Proxmox export carried `no_root_squash`; **nothing here needs it** — every write to the tree happens after the LSIO drop to uid 1000, and the images only chown their Longhorn `/config`. This is a deliberate tightening over the old export, not parity. |
+| **`rw` / `ro`** | The archive's read-only posture is **server-enforced**, as it was before. The media tree cannot get the same treatment — see below. |
+| **`secure`** | Reserved source ports required; "Allow non-root mount" off. The kubelet mounts as root from a reserved port and does not need the exception. |
+
+**Plex's read-only posture is still a CLIENT-side guarantee.** Plex is ro and the arrs are rw, but
+all four can land on any node, so the server sees the same three source addresses for all of them
+and cannot distinguish the reader from the writers. `spec.nfs.readOnly: true` on `plex-media` plus
+the volumeMount flag is the whole enforcement. Unchanged by the move; an honest limitation, not an
+oversight. If it ever needs to be server-enforced, the lever is a dedicated node pool for Plex.
+
+### Service settings that are NOT obvious from the export list
+
+- **Enabled Protocols must include NFSv4** (above).
+- **Require Kerberos for NFSv4: off.** The numeric AUTH_SYS pass-through is the model.
+- **Manage Groups Server-side: off.** This is `rpc.mountd --manage-gids`: the server *discards the
+  client's supplied group list* and looks that uid's groups up in TrueNAS's own user database
+  instead. With no local uid 1000 defined, the arrs' `PGID=3000` would simply vanish. Writes would
+  survive anyway — uid 1000 **owns** the tree, so the owner bits grant access without the group —
+  but it is a free way to lose group semantics.
+- **NFSv4 DNS Domain: blank.** It only feeds `idmapd.conf`, which numeric mapping bypasses.
+- **Bind IP 192.168.2.30** means loopback does not answer: check with `showmount -e 192.168.2.30`,
+  not `localhost`.
+
+There is **no** "NFSv3 ownership model for NFSv4" control on current TrueNAS — it was an older
+TrueNAS/CORE option, and no replacement is needed: on a Linux client against a Linux server over
+AUTH_SYS, both the `nfs` and `nfsd` modules default to `nfs4_disable_idmapping=Y`, so NFSv4 carries
+raw numeric ids rather than `user@domain` principals. §9's `ls -n` probe is how you confirm it.
+
+### The ZFS `sharenfs` properties must be CLEARED, not edited
+
+The pools arrived with the old host's exports still set as dataset properties:
+
+```
+Storage/plex              [email protected]:@10.123.1.10:@192.168.1.18:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
+Storage/nextcloud         [email protected]:@192.168.1.5,[email protected]:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
+Storage/docker-volumes    [email protected]:@192.168.1.5:@192.168.1.10:@192.168.1.18,no_subtree_check,no_root_squash
+```
+
+TrueNAS generates `/etc/exports` from its own middleware database and will fight a property it did
+not set — the failure mode is two differently-configured exports of one path, with no indication
+which is in force. Cleared 2026-10-07:

 ```sh
-zfs set sharenfs='[email protected]:@10.123.1.10:@192.168.1.18:@192.168.1.40:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/plex
+zfs inherit sharenfs Storage/plex
+zfs inherit sharenfs Storage/nextcloud
+zfs inherit sharenfs Storage/docker-volumes
 ```

-Every pre-existing host is preserved verbatim and no option changes; only clients are added.
+`Storage` is `off`, so each resolves to `off`.

-### `Storage/nextcloud` — add the k3s nodes to a **`ro=`** list, not to `rw=`
+### Do NOT touch the dataset's permissions or ACLs

-```sh
-zfs set sharenfs='[email protected]:@192.168.1.5,[email protected]:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/nextcloud
-```
+The tree's ownership **is** the access story (§6), and TrueNAS offers two ways to destroy it
+irreversibly across 8.33 T.

-This is the one place read-only can be enforced **server-side**, and it is worth taking: only
-Nextcloud consumes this dataset from k3s, so the export itself can refuse writes. Even a
-misconfigured client cannot then damage the archive.
+`Storage/plex` carries `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`) and
+`aclinherit=passthrough` (inherited). Measured 2026-10-07: the directories carry **trivial ACLs
+only** — `nfs4xdr_getfacl` shows just `owner@`/`group@`/`everyone@` matching the mode — so `0755
+1000:3000` governs exactly as it did under Proxmox.

-**The media tree cannot get the same treatment.** Plex is read-only and the arrs are read-write, but
-all three may be scheduled on any node, so the server sees the same source addresses for all of
-them and cannot distinguish the reader from the writers. **Plex's read-only posture is therefore a
-client-side guarantee** (`spec.nfs.readOnly: true` on `plex-media`, plus the volumeMount flag) — an
-honest limitation, not an oversight. If it ever needs to be server-enforced, the lever is a
-dedicated node pool for Plex, which is a much larger change.
+- **Stay out of the dataset's Edit Permissions / Edit ACL screen.** On an `acltype=nfsv4` dataset
+  that is a full ACL editor, and its presets write *real* NFSv4 ACLs, recursively if permitted.
+  There is no undo.
+- `aclinherit=passthrough` means a single ACL written at `/mnt/Storage/plex` would be **inherited by
+  everything created under it afterwards** — the blast radius is the whole future library, not one
+  directory.
+- Note GNU `ls`'s `+` marker is **not** a reliable ACL indicator here: it looks for a POSIX ACL
+  xattr, and this dataset stores NFSv4 ACLs elsewhere. Use `nfs4xdr_getfacl`.

-Per-node addresses rather than `@192.168.2.0/24` follows the precedent already on this host
-(`/etc/exports` lists the dev nodes individually, as `/32`). A `/24` would work and would survive
-adding a node; it also grants the whole subnet. Per-node is the tighter default.
+`xattr=on` rather than `sa` is a performance detail, not a permissions one, and changing it would
+not affect existing files.

 ### Verifying, without changing anything

 ```sh
-exportfs -v                  # the new client lines appear, existing ones unchanged
-zfs get sharenfs Storage/plex Storage/nextcloud
+cat /proc/fs/nfsd/versions          # +4.1 must be present
+exportfs -v                         # the two shares, three nodes each, flags as tabulated above
+showmount -e 192.168.2.30           # NOT localhost -- the service is bound to one address
+zfs get -r -o name,property,value,source sharenfs Storage    # every row off or -
 ```

---
-
 ## 6. uid / gid — the failure this design exists to avoid

-The export uses `sec=sys` and `no_root_squash`, so **the client's numeric uid is passed through
-verbatim** and the server's on-disk ownership is the entire access story. The media tree is
+The export uses `sec=sys` and `no_all_squash`, so **the client's numeric uid is passed through
+verbatim** and the server's on-disk ownership is the entire access story. (It no longer carries
+`no_root_squash`, which the Proxmox export did — see §5. Nothing here needs it, and the uid that
+matters is 1000, not 0.) The media tree is
 `0755 1000:3000` throughout (some directories `1000:1000`), so **only uid 1000 can write it**.

 **`fsGroup` cannot fix this, and reaching for it is how this gets misdiagnosed.** The in-tree NFS
@@ -233,24 +278,27 @@ Nextcloud needs no uid work at all, which is one more reason its mount is read-o

 ---

-## 7. The prod placeholder does NOT fail at apply time — read this before merging
+## 7. Why this knob is unvalidated at create AND immutable at update
+
+**The `TBD-PROD-BULK-NFS-SERVER` placeholder is gone** (2026-10-07) — prod now carries a real
+address, because the NAS is on prod's own subnet and there is no longer a decision to defer. The
+hazard that made it load-bearing is worth keeping on the page, because it is the half that explains
+why §10 exists.

 Every other `TBD-PROD-*` in `clusters/prod/kustomization.yaml` is rejected loudly by CRD validation
-(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one is not.**
+(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one was not.**

-A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepts
-`TBD-PROD-BULK-NFS-SERVER`, the PV is created, and the PVC **binds** to it — binding is pure
-control-plane and never touches the server. The failure appears only when a pod mounts:
+A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepted
+`TBD-PROD-BULK-NFS-SERVER`, the PV was created, and the PVC **bound** to it — binding is pure
+control-plane and never touches the server. The failure appeared only when a pod mounted:

 ```
 MountVolume.SetUp failed ... Failed to resolve server TBD-PROD-BULK-NFS-SERVER
 ```

-so **plex, sonarr, radarr and nextcloud would sit in ContainerCreating on prod.**
-
-**Consequently: set `BULK_NFS_SERVER` in `clusters/prod/kustomization.yaml` BEFORE prod reconciles a
-revision that carries these bulk volumes.** Dev renders a working value and is where these manifests
-are meant to be proven first.
+**The same field is immutable on UPDATE** (§10), which is the other half of the same awkwardness: a
+wrong value is accepted silently at create, and then cannot be corrected in place. Together they are
+why repointing this knob is a delete-and-recreate rather than a reconcile.

 ### The static guard this convention leans on is missing from `main` — but the gate now refuses the value anyway

@@ -287,10 +335,9 @@ Without the arm, HALF A is what holds the line, by design.
   EPHEMERAL layer"). Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it
   means those apps start from an empty config volume, so it belongs in its own change with the
   operator present. **This change does not make it worse** — that is why the media mount is `/media`.
-2. **qbittorrent has no bulk mount.** Today the arrs' import source is written by the **legacy
-   non-k3s** download client, which is coherent while the legacy stack stays live. Giving the k3s
-   qbittorrent the same export at the same `/media` path is the natural follow-up, and the paths were
-   chosen so it needs no Remote Path Mapping.
+2. **RESOLVED — qbittorrent now has the bulk mount** (`subPath: downloads` at `/downloads`), so the
+   import source is written by the k3s download client rather than the legacy non-k3s one. See
+   `catalog/qbittorrent.yaml`.
 3. **`/mnt/Storage/plex/downloads/downloads` is a symlink to `/shared/downloads`**, which is outside
   the export. Over NFS it is a dangling symlink on every client. Harmless, but expect Sonarr to log
   it while scanning `downloads/`.
@@ -300,10 +347,10 @@ Without the arm, HALF A is what holds the line, by design.
 5. **`nconnect` was not set.** It would likely help throughput on the media tree, but it is a tuning
   change that should be measured, not guessed, and it changes client behaviour under server
   failure.
-6. **Nothing was verified against a live cluster.** No `kubectl` context is usable from this
-   workstation (browser-OIDC only), and this change is manifest-authoring under an explicit
-   do-not-apply constraint. Every claim about the *host* is measured; every claim about *binding and
-   mounting* is reasoned from the manifests.
+6. **Every claim about the NAS in §5 is measured (2026-10-07); the post-move MOUNT is not.** The PVs
+   still have to be recreated (§10) before any pod mounts 192.168.2.30, so the §9 probes are the
+   outstanding verification, not a formality. The pre-move equivalents did pass against the old host
+   on 2026-09-12.
 7. **The bulk-tier requirement ID is deliberately not typed on any non-`.md` line of this change.**
   `ci/touched-requirements.sh` attributes requirements from R-IDs appearing in added lines, and the
   in-cluster conformance test for the bulk tier is still a gap, so typing it would select a
@@ -312,36 +359,95 @@ Without the arm, HALF A is what holds the line, by design.

 ---

-## 9. Validating on dev, once the export is extended
+## 9. Validating, after the PVs are recreated

-Dev can prove all of this before prod's network path exists. Read-only checks first:
+These run against **prod**; dev is decommissioned. They are the outstanding verification for the
+move — nothing below has been run against 192.168.2.30. Read-only checks first:

 ```sh
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pv
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pvc -A | grep -E 'media|legacy-archive'
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr get pod -o wide
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pv -o custom-columns=\
+NAME:.metadata.name,SERVER:.spec.nfs.server,PATH:.spec.nfs.path,STATUS:.status.phase'
 ```

-Expect **four** PVs — `plex-media`, `sonarr-media`, `radarr-media`, `nextcloud-legacy-archive` (three
-of them naming the *same* export, one per namespace, which is the cross-namespace model) — each
-`Bound` to the same-named PVC, with `RECLAIM POLICY Retain` and an empty `STORAGECLASS`.
+Expect **five** NFS PVs — `plex-media`, `sonarr-media`, `radarr-media`, `qbittorrent-downloads`,
+`nextcloud-legacy-archive` (four of them naming the *same* export, one per namespace, which is the
+cross-namespace model) — each `Bound`, `RECLAIM POLICY Retain`, empty `STORAGECLASS`, and
+**`SERVER 192.168.2.30`**. A PV still showing `192.168.2.102` means the recreate in §10 did not
+happen; Flux will NOT have updated it in place.

-Then the two things that actually matter, from inside a pod:
+Then the four things that actually matter, from inside a pod:

 ```sh
-# (a) the writer can write — this is the uid check, and it is the one that fails in practice
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
-  sh -c 'id; touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+# (a) numeric ids survived the wire -- the idmapping check, new since the TrueNAS move
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- sh -c 'ls -n /media | head -3'
+#     expect uid 1000 / gid 3000 on the entries. 65534 / nobody means ids are being mapped, not
+#     passed through, and every import will fail EACCES while the pod still reports Ready.

-# (b) hardlinks work BETWEEN the library and the download path — the root-mount check
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
+# (b) the writer can write -- the uid check, and the one that fails in practice
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
+  sh -c 'touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+
+# (c) hardlinks work BETWEEN the library and the download path -- the root-mount check
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
  sh -c 'set -e; d=/media/downloads/complete/tv-sonarr/.probe; t=/media/tv/.probe-link;
         echo x > "$d"; ln "$d" "$t"; ls -i "$d" "$t"; rm -f "$d" "$t"; echo HARDLINK-OK'

-# (c) the reader really is read-only
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n plex exec deploy/plex -- \
-  sh -c 'touch /media/.probe 2>&1 | head -1'      # expect "Read-only file system"
+# (d) the reader really is read-only
+kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c 'touch /media/.probe 2>&1 | head -1'
+#     expect "Read-only file system"
 ```

-`(b)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
+`(c)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
 prints `EXDEV`, something has split the single mount back into two.
+
+Reading the failures together is what localises the fault: **(b) EACCES with (a) showing `nobody`**
+is idmapping (§5 service settings). **(b) EACCES with (a) showing `1000`** is on-disk ownership —
+stop and check `ls -ldn` on the NAS before changing anything else.
+
+---
+
+## 10. Repointing BULK_NFS_SERVER means DELETING AND RECREATING THE PVs
+
+**A bound PersistentVolume's NFS source is immutable.** Measured by server-side dry-run on
+2026-10-07:
+
+```
+$ kubectl patch pv sonarr-media --type=json --dry-run=server \
+    -p '[{"op":"replace","path":"/spec/nfs/server","value":"192.168.2.30"}]'
+The PersistentVolume "sonarr-media" is invalid: spec.persistentvolumesource: Forbidden:
+  spec.persistentvolumesource is immutable after creation
+```
+
+Note the pairing with §7: `spec.nfs.server` is **unvalidated at create** (any string is accepted and
+the PVC binds) and **immutable at update**. So editing `BULK_NFS_SERVER` alone does not fail loudly
+and does not work either — Flux's apply errors on the five PVs and the pods keep their previous,
+now-dead mount.
+
+This is safe to do because `persistentVolumeReclaimPolicy: Retain` means deleting a PV never touches
+the data, and `catalog/render.py` emits a **pre-set `claimRef`** on each one, so a recreated PV
+rebinds its own PVC with no manual step.
+
+**Operator sequence** (mutating `kubectl` is an operator action, not an agent action):
+
+```sh
+# 1. Stop the consumers. This also releases the hung hard-mounts against the old server:
+#    the pods report Ready, but any real I/O on /media blocks indefinitely once 2049 stops
+#    answering, because the PVs mount `hard`.
+for ns in plex sonarr radarr qbittorrent nextcloud; do
+  kubectl --context k3s-prod -n $ns scale deploy --all --replicas=0
+done
+
+# 2. Delete the five PVs. Retain + no live claim means nothing is reclaimed or erased.
+kubectl --context k3s-prod delete pv plex-media sonarr-media radarr-media \
+  qbittorrent-downloads nextcloud-legacy-archive
+
+# 3. Let Flux reconcile the revision carrying BULK_NFS_SERVER=192.168.2.30, which recreates them.
+#    Confirm .status.lastAppliedRevision matches the GitRepository artifact before continuing.
+
+# 4. Bring the consumers back, then run every probe in §9.
+```
+
+The PVCs pass through `Lost` between steps 2 and 3; that is expected and resolves when the PV with
+the matching `claimRef` reappears. If a PVC is still `Lost` after Flux reconciles, the PV name or
+its `claimRef` namespace does not match — do **not** delete the PVC to "fix" it.
+
@@ -420,3 +420,22 @@ flux_bootstrap_env_path: "clusters/prod"
 # rather than leaving an inert drop-in behind, so this group-wide `true` is still correct
 # for it.
 etcd_io_priority_enabled: true
+
+# --- The node terminating remote-access WireGuard — DECIDED (measured 2026-10-08) ---------------
+# k3s-prod-1 (192.168.2.10) is what the SOHO router forwards WAN udp/51820 to, confirmed from the
+# cluster rather than assumed: only prod-1's `wg show` had a peer with an `endpoint` and a recent
+# handshake, the other two listeners were idle.
+#
+# node_baseline labels it `homelab.keithjackson.dev/vpn-ingress=true`, and
+# platform-config/cilium/l2-announcement-policy.yaml excludes that label from announcing any
+# LoadBalancer VIP -- because a node CANNOT reach a VIP whose Cilium L2 announcement lease it holds
+# itself (the announcer does not answer its own ARP request; the neighbour entry goes to `FAILED`).
+# prod-1 held the `shared` Gateway lease, and since vpn-ingress MASQUERADEs clients to the node's
+# own address, all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved.
+#
+# [parity] MUST equal the node named by the router's forward (runbooks/vpn-remote-access.md step 3).
+# Nothing in CI can read that forward, so re-pointing it means changing BOTH: this value plus a
+# node_baseline re-run, and the router line. `VpnIngressNodeAnnouncesL2VIP` catches the case where
+# the label lands on a node that then takes a lease; it CANNOT catch the forward being moved
+# without this value following, which is why the runbook spells the two-step out.
+node_baseline_vpn_ingress_node: k3s-prod-1
@@ -259,3 +259,30 @@ node_baseline_dns_options:
 # Written as ONE marked block, so removing an entry here removes it from the node, and
 # an empty list removes the block entirely.
 node_baseline_hosts_pins: []
+
+# --- The node that terminates remote-access WireGuard (tags: vpn-ingress) ----------------------
+# Hostname of the node the SOHO router forwards WAN udp/51820 to, i.e. the one platform/vpn-ingress
+# actually serves remote clients from. Set it in group_vars for the environment; EMPTY by default,
+# because dev has no such forward and because a wrong value is worse than none.
+#
+# WHAT IT IS FOR, AND WHY IT IS NOT COSMETIC. The matching node gets
+# `homelab.keithjackson.dev/vpn-ingress=true`, and
+# platform-config/cilium/l2-announcement-policy.yaml EXCLUDES that label from announcing any
+# LoadBalancer VIP. MEASURED on prod 2026-10-08: a node cannot reach a VIP whose Cilium L2
+# announcement lease it holds itself (the announcer does not answer its own ARP request -- the
+# neighbour entry goes to `FAILED`). Because vpn-ingress runs on `hostNetwork` and MASQUERADEs
+# clients to the node's own address, a VPN client inherits that: with the forward on the node
+# holding the `shared` Gateway lease, all 16 Gateway hostnames were dead over the VPN while PiHole
+# DNS still resolved. Names worked, every app hung, nothing alerted.
+#
+# [parity] MUST name the node the router's forward actually points at. The forward is off-repo and
+# no check can read it (runbooks/vpn-remote-access.md step 3). Changing the forward is therefore a
+# two-part change: this value (then re-run the role), and the router line.
+#
+# Only ONE node should ever carry the label: it removes that node from VIP announcement candidacy,
+# so labelling all three would leave nothing announcing any VIP at all. The task below removes the
+# label from a node that no longer matches, so moving the value moves the label.
+node_baseline_vpn_ingress_node: ""
+node_baseline_vpn_ingress_label_key: homelab.keithjackson.dev/vpn-ingress
+# k3s ships its kubeconfig here; every prod node is a k3s server, so `k3s kubectl` works locally.
+node_baseline_vpn_ingress_kubeconfig: /etc/rancher/k3s/k3s.yaml
@@ -1109,3 +1109,45 @@
    state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
    backup: true
  tags: [dns, hosts]
+
+# ── vpn-ingress node label — tag `vpn-ingress` ────────────────────────────────────────────────
+# Marks the node terminating remote-access WireGuard so Cilium's L2 announcement policy can EXCLUDE
+# it from announcing LoadBalancer VIPs (a node cannot reach a VIP it announces itself -- see
+# defaults/main.yml). Same shape as roles/vpn_egress_gateway's egress-gateway label, and for the
+# same reason: the label is the one place BOTH a nodeSelector and a PromQL alert can read it from.
+#
+# Skipped entirely when node_baseline_vpn_ingress_node is empty (dev, or before the forward exists),
+# so this adds nothing to an environment that has no WAN forward.
+- name: Label this node as the vpn-ingress node
+  ansible.builtin.command:
+    cmd: >-
+      k3s kubectl label node {{ inventory_hostname }}
+      {{ node_baseline_vpn_ingress_label_key }}=true --overwrite
+  environment:
+    KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}"
+  register: node_baseline_vpn_ingress_label_apply
+  changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_apply.stdout"
+  when:
+    - node_baseline_vpn_ingress_node | length > 0
+    - inventory_hostname == node_baseline_vpn_ingress_node
+  tags: [vpn-ingress]
+
+# The other half: moving the knob must MOVE the label, not add a second one. Two labelled nodes
+# would remove two of three nodes from announcement candidacy; three would leave no announcer at
+# all and take every VIP off the LAN.
+- name: Remove the vpn-ingress label from a node that no longer holds the forward
+  ansible.builtin.command:
+    cmd: >-
+      k3s kubectl label node {{ inventory_hostname }}
+      {{ node_baseline_vpn_ingress_label_key }}-
+  environment:
+    KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}"
+  register: node_baseline_vpn_ingress_label_remove
+  changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_remove.stdout"
+  failed_when:
+    - node_baseline_vpn_ingress_label_remove.rc != 0
+    - "'not found' not in node_baseline_vpn_ingress_label_remove.stderr"
+  when:
+    - node_baseline_vpn_ingress_node | length > 0
+    - inventory_hostname != node_baseline_vpn_ingress_node
+  tags: [vpn-ingress]
@@ -3,7 +3,7 @@
 # new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
 #
 # Generates a WireGuard keypair, renders the device's client profile, prints a QR code for a phone,
-# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.yaml. The device's
+# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.conf. The device's
 # private half is written to ONE 0600 file and nowhere else: not sent to the cluster, not committed,
 # never logged.
 #
@@ -62,7 +62,7 @@ set -euo pipefail
 die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; }

 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
+PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"

 # --- defaults, each mirroring a value that is DECLARED elsewhere in this repo -------------------
 # These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to
@@ -134,16 +134,21 @@ CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
 # Two devices sharing an address does not error anywhere: the server routes that address to
 # whichever peer handshook most recently, so BOTH devices appear to work intermittently.
 # COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the
-# COMMENTED-OUT EXAMPLE STANZA in peers.yaml, which made `.2` — the first address anyone would
+# COMMENTED-OUT EXAMPLE STANZA in the register, which made `.2` — the first address anyone would
 # ask for — permanently un-allocatable with a message claiming it was already taken. Strip
 # comments first: a real stanza's `AllowedIPs` line is never commented, while both the template
 # and every peer's `# <name> — <owner>` label are.
-if [ -r "$PEERS_FILE" ] \
-   && grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
-  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
+# FAIL if the register is missing rather than skipping the check. The path was silently wrong once
+# (it still said peers.yaml after the file became peers.conf), and because the old guard was
+# `[ -r "$PEERS_FILE" ] && ...` the only symptom would have been duplicate detection quietly
+# never firing again — the exact silent-skip shape this repo treats as a defect.
+[ -r "$PEERS_FILE" ] || die "cannot read the peer register '$PEERS_FILE'.
+  Duplicate-address detection cannot run, so this script refuses to issue a profile."
+if grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
+  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.conf. Pick another octet."
 fi
-if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
-  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2
+if grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
+  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.conf.\n' "$NAME" >&2
 fi

 # --- output location: 0700, and never inside the repo -------------------------------------------
@@ -210,7 +215,7 @@ cat <<EOF
  Profile     : ${PROFILE}   (0600; contains the private key)

  ------------------------------------------------------------------------------------------
-  STEP 1 — add this to platform/vpn-ingress/peers.yaml, under data.peers.conf, then PR it:
+  STEP 1 — append this to platform/vpn-ingress/peers.conf, then PR it:

    # ${NAME} — <owner>, enrolled $(date -u +%Y-%m-%d)
    [Peer]
@@ -229,7 +234,7 @@ cat <<EOF
  STEP 3 — delete the profile from this machine once the device has it:
    shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE}

-  The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet
+  The tunnel will not work until the peers.conf PR is MERGED and Flux has rolled the DaemonSet
  (~1 min). Until then the device handshakes and gets silence, exactly like an unknown key.
  ------------------------------------------------------------------------------------------
 EOF
@@ -16,6 +16,71 @@ spec:
  externalIPs: false
  interfaces:
    - ^e(n|th)
+  # ===============================================================================================
+  # WHY THIS IS A matchExpressions LIST AND NOT `matchLabels: {kubernetes.io/os: linux}`
+  # ===============================================================================================
+  # The second expression excludes every node labelled `homelab.keithjackson.dev/vpn-ingress=true`
+  # from announcing ANY LoadBalancer VIP. That is the node the SOHO router forwards WAN udp/51820 to.
+  # It is not tuning; it is the fix for a measured, silent, total failure of remote access.
+  #
+  # MEASURED ON PROD 2026-10-08: A NODE CANNOT REACH A VIP IT ANNOUNCES ITSELF. Cilium's L2
+  # announcer answers ARP for the VIP on behalf of OTHER LAN hosts, but not for the announcing node
+  # itself. On the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP
+  # is dropped before it leaves the node:
+  #
+  #   k3s-prod-1, holding cilium-l2announce-gateway-cilium-gateway-shared:
+  #     ip neigh show 192.168.2.20  ->  192.168.2.20 dev eno1 FAILED       (its own VIP)
+  #     nc -z 192.168.2.20 443      ->  FAIL          nc -z 192.168.2.20 80 -> FAIL
+  #   k3s-prod-2 and k3s-prod-3, holding no lease for it:
+  #     ip neigh show 192.168.2.20  ->  lladdr bc:e9:2f:7e:c4:a6 (prod-1)   nc 443/80 -> OK
+  #
+  # Symmetric across all 3 nodes x 5 VIPs: every node reached every VIP EXCEPT the ones it alone
+  # announced (prod-2 could not reach crafty 192.168.2.22 or unifi 192.168.2.21; prod-3 could not
+  # reach plex 192.168.2.24). So this is a property of L2 announcement, not of one Service.
+  #
+  # WHY IT ONLY HURTS THE VPN NODE. For an ordinary workload this is harmless: pods reach Services
+  # through the cluster datapath and never ARP for a VIP. platform/vpn-ingress is the exception --
+  # it runs WireGuard with `hostNetwork: true` and MASQUERADEs remote clients to the node's own
+  # address, so a VPN client's packets take the node's LAN path and inherit exactly this ARP
+  # failure. On 2026-10-08 the forward pointed at k3s-prod-1, which held the Gateway lease, and all
+  # 16 Gateway hostnames were unreachable over the VPN while PiHole DNS still answered -- so names
+  # resolved and every app hung. Nothing in the cluster reported it; the alerts below now do.
+  #
+  # WHY A NODE LABEL AND NOT A cluster-vars KNOB. A knob would put the node's identity in git, which
+  # is more reviewable -- but it CANNOT BE USED BY THE ALERT that guards this, and an unguarded
+  # invariant here is worth less than a reviewable one. The alert harness
+  # (platform/monitoring/tests/extract_rules.py) REFUSES any alert whose `expr` contains a
+  # dollar-brace postBuild placeholder (spelled without braces here ON PURPOSE: this file is itself
+  # envsubst'd, and a literal dollar-brace-dot-dot-dot is UNPARSEABLE to envsubst, which fails the
+  # WHOLE Kustomization -- ci: flux-postbuild-substitution caught exactly this comment), because the
+  # rule it would unit-test is not the rule that gets
+  # deployed. A label is readable from BOTH sides: this selector matches on it, and
+  # `kube_node_labels` carries it into Prometheus so `VpnIngressNodeAnnouncesL2VIP` can join lease
+  # holders against it. One source of truth, no git-vs-cluster skew, and the policy stays
+  # env-agnostic as its header above claims. The label is set by
+  # infra/ansible/roles/node_baseline (node_baseline_vpn_ingress_node), the same shape
+  # roles/vpn_egress_gateway already uses for `egress-gateway`.
+  #
+  # THE COST, STATED: VIP announcement candidacy drops from 3 nodes to 2. That is still redundant,
+  # and the trade is deliberate -- the excluded node is the one whose *outbound* reach to every VIP
+  # matters most, because it carries every remote client. If BOTH remaining nodes are down the VIPs
+  # go dark where previously one node might have served them, but at that point the cluster has one
+  # node left and far worse problems.
+  #
+  # `NotIn` with an ABSENT key MATCHES (k8s apimachinery: a missing label is "not in the set"), so
+  # the other two nodes need no label and nothing has to be applied to them. If the label is never
+  # applied anywhere, this excludes nothing and the policy degrades to its pre-2026-10-08 behaviour
+  # -- the status quo, not an outage: it fails back, not down. That silent-degradation path is
+  # exactly why `VpnIngressNodeLabelMissing` exists as a watchdog: without it, the alert below
+  # would be structurally INERT (its join would match nothing) and would look identical to healthy.
  nodeSelector:
-    matchLabels:
-      kubernetes.io/os: linux
+    matchExpressions:
+      - key: kubernetes.io/os
+        operator: In
+        values:
+          - linux
+      # The node terminating remote-access WireGuard must announce nothing -- see above.
+      - key: homelab.keithjackson.dev/vpn-ingress
+        operator: NotIn
+        values:
+          - "true"
@@ -1147,6 +1147,97 @@ spec:
              metricLabelsAllowlist.
            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
        # -----------------------------------------------------------------------------------------
+        # REMOTE-ACCESS VPN (platform/vpn-ingress) -- NOT the egress tunnel above. Different
+        # direction, different workload: this is inbound WireGuard for remote devices.
+        # -----------------------------------------------------------------------------------------
+        - alert: VpnIngressNodeAnnouncesL2VIP
+          # THE FAILURE THIS CATCHES. MEASURED on prod 2026-10-08: a node CANNOT reach a
+          # LoadBalancer VIP whose Cilium L2 announcement lease it holds itself -- the announcer
+          # does not answer its own ARP request, so the neighbour entry settles at `FAILED` and
+          # packets die before leaving the node (`ip neigh show 192.168.2.20` -> FAILED on the
+          # lease holder, resolved on both others). platform/vpn-ingress runs WireGuard on
+          # `hostNetwork` and MASQUERADEs remote clients to the node's own address, so if the node
+          # the router forwards udp/51820 to is also a lease holder, VPN clients cannot reach the
+          # VIPs it announces. On 2026-10-08 that was the `shared` Gateway: all 16 app hostnames
+          # dead over the VPN, while PiHole DNS kept answering (its VIP has two Services and the
+          # other lease sat on prod-2) -- so the tunnel handshook, names resolved, and every app
+          # hung. Nothing reported it for ~2 days.
+          #
+          # platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled
+          # `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so this should be
+          # permanently unsatisfiable. The alert exists because the exclusion has ways to stop
+          # working that no static check on that file's TEXT can see: the label being removed or
+          # moved to the wrong node, a Cilium upgrade changing `NotIn` semantics or the
+          # nodeSelector field, and the selector silently failing to match. This watches the
+          # OUTCOME -- who actually holds the lease -- so all of them surface the same way.
+          #
+          # WHY THE LABEL JOIN RATHER THAN A NODE NAME. The node's identity is NOT written into
+          # this expr, because the harness (tests/extract_rules.py) refuses any alert whose expr
+          # contains a dollar-brace placeholder -- spelled in words because a literal one here is
+          # unparseable to envsubst and would fail this whole Kustomization. Rightly so: the rule it
+          # unit-tests would not be the
+          # rule deployed. Joining `kube_lease_owner` against `kube_node_labels` keeps ONE source
+          # of truth (the label itself, which is also what the policy selects on), so this alert
+          # cannot drift out of agreement with the exclusion it guards. `label_replace` is needed
+          # because kube_lease_owner names the holder `lease_holder` while kube_node_labels names
+          # it `node`; the join is then on a shared `node` label. The label is in
+          # kube-state-metrics' metricLabelsAllowlist (kube-prometheus-stack.yaml) -- without that
+          # line this rule is inert, which `VpnIngressNodeLabelMissing` below exists to catch.
+          #
+          # `for: 15m` because leases legitimately move during a node reboot or a Cilium roll and
+          # Cilium re-elects within seconds; a real collision persists, since the lease holder
+          # keeps renewing and it never self-heals.
+          expr: |
+            max by (lease) (
+              label_replace(
+                kube_lease_owner{namespace="kube-system", lease=~"cilium-l2announce-.*"},
+                "node", "$1", "lease_holder", "(.+)"
+              )
+              * on (node) group_left ()
+              kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"}
+            ) == 1
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "VPN ingress node is announcing a LoadBalancer VIP ({{ $labels.lease }})"
+            description: >-
+              The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access
+              WireGuard, and it now holds the Cilium L2 announcement lease `{{ $labels.lease }}`. A
+              node cannot reach a VIP it announces itself, so every remote VPN client has just lost
+              all access to that VIP's Service -- silently, with DNS still resolving. Check that
+              platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that
+              the label is still on the node the router forwards udp/51820 to.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
+        - alert: VpnIngressNodeLabelMissing
+          # THE WATCHDOG for the rule above, and the reason it is not optional. The exclusion in
+          # l2-announcement-policy.yaml is a `NotIn` on this label, and `NotIn` with an ABSENT key
+          # MATCHES -- so if the label is on NO node, the policy excludes nothing, every node is an
+          # announcement candidate again, and the 2026-10-08 failure is fully re-armed. The alert
+          # above would not fire either: its join would match nothing and return no series. Present,
+          # evaluating, and incapable of firing -- the exact "inert critical alert" shape that
+          # VPNEgressUplinkLabelMissing exists for on the egress side, and that this repo has been
+          # burned by before.
+          #
+          # The label is applied by infra/ansible/roles/node_baseline
+          # (node_baseline_vpn_ingress_node); the most likely cause of it going absent is a node
+          # rebuild where that role has not been re-run.
+          expr: absent(kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"})
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF"
+            description: >-
+              No node publishes homelab.keithjackson.dev/vpn-ingress=true, so
+              l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none:
+              any node may now announce a LoadBalancer VIP, including the one terminating
+              remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so
+              the warning matters more than its severity suggests. Re-run the node_baseline role on
+              the forwarded node, and confirm the label is in kube-state-metrics'
+              metricLabelsAllowlist.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
+        # -----------------------------------------------------------------------------------------
        # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
        # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
        # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
@@ -281,8 +281,20 @@ spec:
      # without anyone having to remember to re-enable it later. Scoped to one label on one resource:
      # 3 extra series, no cardinality concern. Label name is sanitized by KSM to
      # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
+      # `vpn-ingress` (added 2026-10-17) marks the node the SOHO router forwards WAN udp/51820 to.
+      # platform-config/cilium/l2-announcement-policy.yaml excludes that node from announcing any
+      # LoadBalancer VIP, because a node cannot reach a VIP whose L2 announcement lease it holds
+      # itself and vpn-ingress MASQUERADEs remote clients to the node's own address (measured
+      # 2026-10-08: all 16 Gateway hostnames dead over the VPN while DNS still resolved). It is
+      # allowlisted HERE because that is the only way the invariant can be WATCHED:
+      # `VpnIngressNodeAnnouncesL2VIP` joins `kube_lease_owner` against this label, and the alert
+      # unit-test harness refuses a dollar-brace placeholder in an alert expr (written out in words:
+      # a literal one here is unparseable to envsubst and fails the whole Kustomization), so a knob
+      # could not have been used. Without this line BOTH that alert and its
+      # `VpnIngressNodeLabelMissing` watchdog are structurally inert. Same reasoning, and the same
+      # 3-extra-series cost, as the two egress labels below.
      metricLabelsAllowlist:
-        - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
+        - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink,homelab.keithjackson.dev/vpn-ingress]"
      # --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
      # Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
      # `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
@@ -199,7 +199,7 @@ data:
          "id": 15,
          "type": "timeseries",
          "title": "CAUSE 4 — NFS RPC retransmission ratio, per node",
-          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.102, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
+          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.30, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
          "gridPos": {"h": 8, "w": 12, "x": 12, "y": 39},
          "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
            {"color": "green", "value": null}, {"color": "red", "value": 0.01}
@@ -0,0 +1,203 @@
+# promtool rule-unit tests for VpnIngressNodeAnnouncesL2VIP + VpnIngressNodeLabelMissing
+# (platform/monitoring/alerts.yaml)
+#
+# Run with: platform/monitoring/tests/run.sh
+#           (the merge gate runs the same harness via
+#            conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS
+# MEASURED on prod 2026-10-08: a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
+# lease it holds ITSELF -- the announcer does not answer its own ARP request, so on the lease holder
+# the neighbour entry settles at `FAILED` and packets to that VIP die before leaving the node.
+# platform/vpn-ingress runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
+# own address, so a VPN client takes that node's LAN path and inherits the failure. With the
+# router's udp/51820 forward on k3s-prod-1, which held the `shared` Gateway lease, all 16 Gateway
+# hostnames were dead over the VPN -- while PiHole DNS kept answering, because its VIP has two
+# Services and k3s-prod-2 held the other lease. The tunnel handshook, names resolved, every app
+# hung, and nothing reported it for ~2 days.
+#
+# platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled
+# `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so VpnIngressNodeAnnouncesL2VIP
+# should be permanently unsatisfiable. THAT IS EXACTLY WHY IT NEEDS A TEST: an alert expected never
+# to fire is indistinguishable, in production, from an alert that CANNOT fire. Nothing would ever
+# reveal a mistyped label, a broken `label_replace`, or a join on the wrong label. Test 1 proves it
+# fires on the real failure; test 2 proves it is quiet in the fixed state, so it is not a rule that
+# fires on anything; test 3 pins the `for: 15m` against routine lease churn.
+#
+# THE JOIN IS THE FRAGILE PART AND IS TESTED DELIBERATELY. `kube_lease_owner` names the holder in
+# `lease_holder`; `kube_node_labels` names it `node`. The rule bridges them with `label_replace`
+# and joins `on (node) group_left ()`. Fixtures therefore carry BOTH series shapes with real label
+# sets, because a join that silently matches nothing returns no series -- which looks exactly like
+# "healthy" and is the failure mode this whole group exists to prevent.
+#
+# Fixture values are prod's real shape (measured 2026-10-08): leases
+# `cilium-l2announce-gateway-cilium-gateway-shared` (k3s-prod-1) and
+# `cilium-l2announce-pihole-pihole-dns-tcp` (k3s-prod-2), in namespace kube-system, with
+# k3s-prod-1 the vpn-ingress node.
+
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. THE BUG, EXACTLY AS IT HAPPENED -- the vpn-ingress node holds the Gateway lease: FIRES.
+  # =============================================================================================
+  - interval: 1m
+    name: vpn-ingress node holds the shared Gateway L2 lease - VpnIngressNodeAnnouncesL2VIP fires
+    input_series:
+      # The collision. A lease holder keeps renewing, so this series does not lapse on its own --
+      # which is why the condition never self-heals and why a 15m `for:` is safe.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
+        values: '1+0x40'
+      # A second lease held by a DIFFERENT node, present throughout. This is the discriminator that
+      # stops the rule from being "any l2announce lease exists anywhere": if the join were dropped,
+      # this series alone would fire the alert, and test 2 would fail.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-pihole-pihole-dns-tcp",lease_holder="k3s-prod-2"}'
+        values: '1+0x40'
+      # k3s-prod-1 is the vpn-ingress node. Only this node carries the label.
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
+        values: '1+0x40'
+      # The other two nodes publish kube_node_labels WITHOUT the vpn-ingress label. Present so the
+      # join is exercised against non-matching nodes rather than against nothing at all.
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
+        values: '1+0x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
+        values: '1+0x40'
+    alert_rule_test:
+      # Held 15m, so nothing before then.
+      - eval_time: 14m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+      - eval_time: 20m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts:
+          # EXACTLY ONE alert: the pihole lease on prod-2 must NOT produce a second one.
+          - exp_labels:
+              severity: critical
+              lease: cilium-l2announce-gateway-cilium-gateway-shared
+            exp_annotations:
+              summary: "VPN ingress node is announcing a LoadBalancer VIP (cilium-l2announce-gateway-cilium-gateway-shared)"
+              description: >-
+                The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access
+                WireGuard, and it now holds the Cilium L2 announcement lease `cilium-l2announce-gateway-cilium-gateway-shared`. A
+                node cannot reach a VIP it announces itself, so every remote VPN client has just lost
+                all access to that VIP's Service -- silently, with DNS still resolving. Check that
+                platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that
+                the label is still on the node the router forwards udp/51820 to.
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
+      # The label IS present, so the watchdog must stay quiet here.
+      - eval_time: 40m
+        alertname: VpnIngressNodeLabelMissing
+        exp_alerts: []
+
+  # =============================================================================================
+  # 2. THE FIXED STATE -- every lease is held by a node that is NOT the vpn-ingress node: QUIET.
+  # This is what l2-announcement-policy.yaml's exclusion produces, and the assertion that keeps
+  # test 1 meaningful rather than an alert that fires on anything.
+  # =============================================================================================
+  - interval: 1m
+    name: all l2 leases held by other nodes - VpnIngressNodeAnnouncesL2VIP stays quiet
+    input_series:
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-2"}'
+        values: '1+0x40'
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-plex-plex-l4-tcp",lease_holder="k3s-prod-3"}'
+        values: '1+0x40'
+      # A lease the vpn-ingress node DOES legitimately hold, in a different family -- proves the
+      # `cilium-l2announce-.*` matcher is doing work. Every node holds operator/controller leases;
+      # only the ANNOUNCEMENT ones matter here, and a matcher broad enough to catch this would page
+      # on a healthy cluster forever.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-operator-resource-lock",lease_holder="k3s-prod-1"}'
+        values: '1+0x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
+        values: '1+0x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
+        values: '1+0x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
+        values: '1+0x40'
+    alert_rule_test:
+      - eval_time: 40m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+      - eval_time: 40m
+        alertname: VpnIngressNodeLabelMissing
+        exp_alerts: []
+
+  # =============================================================================================
+  # 3. A LEGITIMATE, TRANSIENT LEASE MOVE -- the vpn-ingress node holds it for 11m during a node
+  # reboot or a Cilium roll, then Cilium re-elects away: QUIET. This pins the `for: 15m` as
+  # deliberate. Without it, every rolling restart of the cilium DaemonSet would page, and a paging
+  # alert that cries wolf on routine maintenance gets muted -- which is how the real collision
+  # would go unseen again.
+  # =============================================================================================
+  - interval: 1m
+    name: transient lease move shorter than for 15m - VpnIngressNodeAnnouncesL2VIP stays quiet
+    input_series:
+      # Held by the vpn-ingress node for 11 samples, then the series goes absent (the lease moved,
+      # so kube-state-metrics retires this label set and starts a new one).
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
+        values: '1+0x10 _x30'
+      # Re-elected onto prod-3 from t=11m. A label VALUE change retires one series and starts
+      # another, which is why this is a second series and not a changing value.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-3"}'
+        values: '_x11 1+0x29'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
+        values: '1+0x40'
+    alert_rule_test:
+      # Inside the transient window: pending at most, never firing.
+      - eval_time: 10m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+      # Well past it, with the lease settled on another node.
+      - eval_time: 30m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. THE WATCHDOG -- no node carries the label at all. The policy's `NotIn` then excludes
+  # NOTHING (a missing key matches), so the 2026-10-08 failure is fully re-armed, AND
+  # VpnIngressNodeAnnouncesL2VIP is structurally INERT: its join matches nothing and returns no
+  # series, which is indistinguishable from healthy. Both halves are asserted here, because an
+  # inert critical alert is the exact shape this repo has been burned by before.
+  # =============================================================================================
+  - interval: 1m
+    name: vpn-ingress label absent everywhere - watchdog fires and the critical alert is inert
+    input_series:
+      # A lease held by the node that IS (in reality) the forwarded one -- but with no label
+      # anywhere, nothing can tell. This is the dangerous state.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
+        values: '1+0x60'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}'
+        values: '1+0x60'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
+        values: '1+0x60'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
+        values: '1+0x60'
+    alert_rule_test:
+      # The critical alert CANNOT fire -- this is the inertness being pinned, not an aside.
+      - eval_time: 45m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+      # ... which is precisely why the watchdog must.
+      - eval_time: 45m
+        alertname: VpnIngressNodeLabelMissing
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              # absent() propagates the matcher's own labels onto the series it returns, so the
+              # alert inherits this one. Asserted rather than worked around: it is harmless, and
+              # pinning it means a future change to the matcher cannot alter the alert's identity
+              # (and therefore its Alertmanager grouping) unnoticed.
+              label_homelab_keithjackson_dev_vpn_ingress: "true"
+            exp_annotations:
+              summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF"
+              description: >-
+                No node publishes homelab.keithjackson.dev/vpn-ingress=true, so
+                l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none:
+                any node may now announce a LoadBalancer VIP, including the one terminating
+                remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so
+                the warning matters more than its severity suggests. Re-run the node_baseline role on
+                the forwarded node, and confirm the label is in kube-state-metrics'
+                metricLabelsAllowlist.
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
@@ -63,6 +63,75 @@ spec:
      # gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
      skip-provider-button: "true"
      scope: "openid email profile"
+    # ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ──────────
+    # MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at
+    # startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so
+    # the Gateway answers 503 and the proxy dies:
+    #
+    #   [provider.go:55] Performing OIDC Discovery...
+    #   [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC
+    #                configuration: unexpected status "503": no healthy upstream
+    #
+    # It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was
+    # still down minutes later and only a manual pod delete cleared it. There is no retry flag; the
+    # process simply does not survive a cold start. A readinessProbe cannot help (the container is
+    # already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+    #
+    # `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in
+    # values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init
+    # container for its built-in waitForRedis gate, so this adds no new base image to the platform.
+    # A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole
+    # point.
+    #
+    # BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst
+    # case is then exactly today's behaviour, never something new. An unbounded wait would trade a
+    # CrashLoop for a pod wedged in Init forever, which is not an improvement.
+    #
+    # SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is
+    # rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which
+    # CONSUMES braced expansions -- so a braced shell variable renders to the empty string and
+    # silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars
+    # knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud
+    # OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.)
+    extraInitContainers:
+      - name: wait-for-idp
+        image: docker.io/library/alpine:3.22
+        imagePullPolicy: IfNotPresent
+        command: ["/bin/sh", "-c"]
+        args:
+          - |
+            set -u
+            url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
+            i=0
+            while [ "$i" -lt 60 ]; do
+              # busybox wget validates against alpine's ca-certificates-bundle, so a real TLS
+              # failure is a non-zero exit here rather than a false pass.
+              if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then
+                echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy"
+                exit 0
+              fi
+              i=$(( i + 1 ))
+              sleep 5
+            done
+            echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2
+            echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2
+            exit 0
+        securityContext:
+          allowPrivilegeEscalation: false
+          runAsNonRoot: true
+          runAsUser: 65534
+          runAsGroup: 65534
+          readOnlyRootFilesystem: true
+          capabilities:
+            drop:
+              - ALL
+        resources:
+          requests:
+            cpu: 10m
+            memory: 16Mi
+          limits:
+            memory: 32Mi
+
    # Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
    metrics:
      enabled: true
@@ -13,7 +13,7 @@ egress out the house WAN.
 | Transport | WireGuard, **udp/51820**, kernel implementation |
 | Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
 | Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
-| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
+| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
 | The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
 | Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
 | Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
@@ -117,8 +117,19 @@ boundary around this port is WireGuard's peer authentication, full stop.
 The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
 forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
 A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
-keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no
-re-issued client profiles. The two idle listeners cost ~15 MiB each.
+keys — so losing the forwarded node never costs a re-issued client profile. The two idle listeners
+cost ~15 MiB each.
+
+> **This section used to say "losing the forwarded node is a one-line router change, with no cluster
+> work". It was wrong, and it cost two days of silently broken remote access on 2026-10-08.** The
+> nodes are warm, but they are **not interchangeable**: the node the forward points at must be the
+> one labelled `homelab.keithjackson.dev/vpn-ingress=true`, because that is the node
+> `platform-config/cilium/l2-announcement-policy.yaml` excludes from announcing LoadBalancer VIPs.
+> **A node cannot reach a VIP whose Cilium L2 announcement lease it holds itself** (see "Measured on
+> prod, 2026-10-08" below), and since this pod MASQUERADEs clients to the node's own address, a VPN
+> client inherits that. Failover is therefore **two** changes: move
+> `node_baseline_vpn_ingress_node` and re-run that role, then re-point the router —
+> `runbooks/vpn-remote-access.md` step 3.

 A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
 replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
@@ -129,7 +140,7 @@ translations win here.

 | half | where | why |
 |---|---|---|
-| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
+| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
 | the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |

 The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
@@ -166,8 +177,9 @@ NetworkPolicy applies.
 ## Operating it

 * **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
-  stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
-* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
+  stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
+* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
+  stops working as soon as the pods roll (the generator's content hash guarantees they do).
 * **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
 * **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
  (`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
@@ -185,6 +197,45 @@ right way; one was a defect in my own manifest.
 | Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
 | Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |

+## Measured on prod, 2026-10-08 — why the forwarded node cannot be any node
+
+The first device enrolled, handshook, resolved names, and could not open a single connection to any
+cluster web app. The tunnel was not at fault — every part of it measured healthy:
+`net.ipv4.ip_forward=1`, `wg0` up at `10.90.0.1/24` MTU 1420, the peer authorised with real
+bidirectional transfer (346 KiB in / 200 KiB out), `MASQUERADE` present and counting 26 flows,
+`FORWARD -i/-o wg0 ACCEPT` counting 108 / 171 packets, and PiHole DNS answering from the node.
+
+**The fault was that a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
+holds itself.** Cilium's announcer answers ARP for the VIP on behalf of other LAN hosts, but not for
+the announcing node's own request, so the neighbour entry settles at `FAILED`:
+
+| from | `192.168.2.20` (Gateway) | `192.168.2.22` (crafty) | `192.168.2.24` (plex) | announces |
+|---|---|---|---|---|
+| `k3s-prod-1` | **FAILED** | — | — | `.20` |
+| `k3s-prod-2` | ok | **INCOMPLETE** | ok | `.21`, `.22`, `.23-tcp` |
+| `k3s-prod-3` | ok | ok | **INCOMPLETE** | `.24` |
+
+Symmetric across 3 nodes × 5 VIPs: each node reached every VIP **except** the ones it alone
+announced. It is a property of L2 announcement, not of any one Service, and it is harmless for
+ordinary pods — they reach Services through the cluster datapath and never ARP for a VIP. This
+component is the exception, because `hostNetwork` + `MASQUERADE` put remote clients on the node's
+own LAN path.
+
+The forward pointed at `k3s-prod-1`, which held the `shared` Gateway lease, so **all 16 Gateway
+hostnames were dead over the VPN**. Two things hid it:
+
+* **PiHole survived by accident.** `192.168.2.23` is announced by *two* Services
+  (`pihole-dns-udp` on prod-1, `pihole-dns-tcp` on prod-2), so prod-2 answered the ARP prod-1 could
+  not. Names resolved perfectly and every app hung — the hardest shape to diagnose from a client.
+* **Nothing reported it.** No probe in this repo compares the forwarded node against the lease
+  holders, and from outside WireGuard is silent to unauthenticated traffic anyway.
+
+Closed by labelling the forwarded node `homelab.keithjackson.dev/vpn-ingress=true` and excluding
+that label in `platform-config/cilium/l2-announcement-policy.yaml` (the node then ARP-resolves every
+VIP), with `VpnIngressNodeAnnouncesL2VIP` watching the lease holders so a recurrence is loud and
+`VpnIngressNodeLabelMissing` watching for the label going absent, which would silently re-arm the
+whole failure. Cost: VIP announcement candidacy drops from three nodes to two.
+
 ## Prerequisites outside this directory

 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
@@ -211,7 +262,7 @@ right way; one was a defect in my own manifest.
  covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
  kube-prometheus-stack. **This is the top follow-up.**
 * **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
-  handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
+  handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
  reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
  `wg-server-key` Secret.
 * **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
@@ -62,9 +62,39 @@
 # as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to
 # whichever node the scheduler picked and silently invalidate that forward on every reschedule. A
 # DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of
-# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no
-# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own
-# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each.
+# them, and if that node is lost every other node is already warm and answers the same keys, so no
+# client profile ever changes. Each node binds udp/51820 in its own netns, so there is no conflict,
+# and the two idle listeners cost ~15 MiB each.
+#
+# BUT THE NODES ARE NOT INTERCHANGEABLE, AND THIS COMMENT USED TO CLAIM THEY WERE.
+# It said "FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work". That was wrong, and the
+# wording cost two days of silently broken remote access on 2026-10-08. MEASURED that day: a node
+# cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds ITSELF — the announcer
+# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and packets to
+# that VIP die before leaving the node:
+#
+#   k3s-prod-1, holding the shared Gateway lease:  ip neigh show 192.168.2.20 -> FAILED
+#   k3s-prod-2 / k3s-prod-3:                       ip neigh show 192.168.2.20 -> lladdr …(prod-1)
+#
+# Because this pod MASQUERADEs clients to the node's own address (see PostUp below), a remote client
+# inherits exactly that failure. With the forward on the Gateway's announcing node, all 16 Gateway
+# hostnames were unreachable over the VPN while PiHole DNS still resolved — a tunnel that
+# handshakes, resolves names, and then hangs on every single app.
+#
+# THE FIX IS DECLARATIVE AND LIVES OUTSIDE THIS FILE: the forwarded node is labelled
+# homelab.keithjackson.dev/vpn-ingress=true by infra/ansible/roles/node_baseline (from
+# node_baseline_vpn_ingress_node), and platform-config/cilium/l2-announcement-policy.yaml excludes
+# that label from announcing ANY VIP, so the node can ARP-resolve all of them. A label rather than a
+# cluster-vars knob because the alert that guards this has to read it too, and the alert unit-test
+# harness refuses a postBuild placeholder in an expr. Failover is therefore TWO steps — move
+# node_baseline_vpn_ingress_node and re-run the role, THEN re-point the router
+# (runbooks/vpn-remote-access.md step 3).
+# `VpnIngressNodeAnnouncesL2VIP` (platform/monitoring/alerts.yaml) fires if the two collide again.
+#
+# This pod deliberately does NOT probe that itself. A readinessProbe on VIP reachability would wedge
+# a DaemonSet rollout behind an unrelated Gateway outage (maxUnavailable: 1 below), and a liveness
+# probe would tear down a tunnel that is carrying LAN traffic perfectly well. The condition is a
+# property of the CLUSTER's announcement layout, so it is watched there.
 #
 # All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the
 # DaemonSet to cover the cluster; add them if a node is ever tainted.
@@ -108,7 +138,7 @@ spec:
      dnsPolicy: ClusterFirstWithHostNet
      automountServiceAccountToken: false
      # ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
-      # The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
+      # The two halves are deliberately stored apart (peers.conf explains why) and are only ever
      # joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
      # key is therefore never written to any node's disk, never committed, and never logged.
      initContainers:
@@ -11,10 +11,16 @@
 # NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
+# REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated
+# ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's
+# nameReference transformer will not rewrite a reference across a namespace boundary — so without
+# this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the
+# pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1).
+# Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own
+# content generator.
+namespace: vpn-ingress
 resources:
  - namespace.yaml
-  # The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
-  - peers.yaml
  # The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
  - daemonset.yaml
  # Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
@@ -38,3 +44,26 @@ resources:
 # hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
 # Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
 # that into a named error rather than a silently non-working tunnel.
+
+# ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ─────────────────────────────────────────
+# peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a
+# GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic:
+# kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's
+# volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE
+# PODS automatically.
+#
+# A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux
+# updates it in place, the pod template is byte-identical, no rollout is triggered — and because
+# the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps
+# the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the
+# ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having
+# never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this.
+#
+# DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism.
+configMapGenerator:
+  - name: wg-peers
+    files:
+      - peers.conf
+    options:
+      labels:
+        app.kubernetes.io/name: vpn-ingress
@@ -19,7 +19,7 @@
 # protects this port, and it would be enforcing nothing. So there isn't one.
 #
 # WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
-#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
+#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.conf gets
 #      NO REPLY — see README "Why this is safe to expose". This is the real control.
 #   2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
 #   3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement
@@ -1,4 +1,16 @@
---
+# THIS FILE IS THE MOUNTED CONTENT. It is turned into the `wg-peers` ConfigMap by
+# `configMapGenerator` in kustomization.yaml, whose generated name carries a CONTENT HASH — so
+# editing this file changes the ConfigMap name, which changes the DaemonSet's pod template, which
+# ROLLS THE PODS. That is the whole reason it is a generator and not an inline ConfigMap manifest:
+# a fixed-name ConfigMap updates in place, the pod template never changes, no rollout happens, and
+# the init container's wg0.conf (built once at pod start, in tmpfs) keeps the OLD peer list
+# forever. Measured 2026-10-06: a merged peer sat in the ConfigMap while `wg show` listed none and
+# the DaemonSet reported generation=1 — it had never rolled. Same mechanism and same fix as
+# runbooks/kustomization.yaml.
+#
+# Every line here is WireGuard config. `#` is a legal comment in it, so the documentation below
+# travels inside the file that is actually mounted, rather than in a wrapper that gets stripped.
+#
 # ============================================================================================
 # THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
 # ============================================================================================
@@ -41,43 +53,40 @@
 # the Noise handshake and would be a genuine secret — which would force this register into SOPS
 # and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
 # belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: wg-peers
-  namespace: vpn-ingress
-  labels:
-    app.kubernetes.io/name: vpn-ingress
-data:
-  # Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
-  #
-  # *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
-  # An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
-  # authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
-  # DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
-  # The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
-  # (`wg show` reports an interface with no peers).
-  #
-  # COPY THIS TEMPLATE, uncomment, one block per device:
-  #
-  #   # <device-name> — <owner>, enrolled <YYYY-MM-DD>
-  #   [Peer]
-  #   PublicKey = <the public key new-peer.sh printed>
-  #   AllowedIPs = 10.90.0.<octet>/32
-  #
-  # The placeholders above are deliberately NOT valid values. An earlier version of this template
-  # used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
-  # the whole file — matched it, so `.2` could never be allocated and the script insisted it was
-  # already taken. The check now ignores comments; keeping the example unmatchable too means a
-  # future grep cannot be fooled either.
-  #
-  # Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
-  # tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
-  # it an incident responder cannot tell which stanza to delete.
-  peers.conf: |
-    # No peers enrolled yet. See the template in this file's comments and
-    # ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
-    # keith-phone — <owner>, enrolled 2026-10-06
-    [Peer]
-    PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
-    AllowedIPs = 10.90.0.2/32
+#
+# name: wg-peers
+# namespace: vpn-ingress
+# labels:
+# app.kubernetes.io/name: vpn-ingress
+# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
+#
+# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
+# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
+# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
+# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
+# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
+# (`wg show` reports an interface with no peers).
+#
+# COPY THIS TEMPLATE, uncomment, one block per device:
+#
+#   # <device-name> — <owner>, enrolled <YYYY-MM-DD>
+#   [Peer]
+#   PublicKey = <the public key new-peer.sh printed>
+#   AllowedIPs = 10.90.0.<octet>/32
+#
+# The placeholders above are deliberately NOT valid values. An earlier version of this template
+# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
+# the whole file — matched it, so `.2` could never be allocated and the script insisted it was
+# already taken. The check now ignores comments; keeping the example unmatchable too means a
+# future grep cannot be fooled either.
+#
+# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
+# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
+# it an incident responder cannot tell which stanza to delete.
+#
+# No peers enrolled yet. See the template in this file's comments and
+# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
+# keith-phone — <owner>, enrolled 2026-10-06
+[Peer]
+PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
+AllowedIPs = 10.90.0.2/32
@@ -5,7 +5,7 @@
 # WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
 # (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
 # daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
-# register (peers.yaml) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
+# register (peers.conf) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
 #
 # WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
 # source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`
@@ -119,6 +119,11 @@ configMapGenerator:
      - update-rollback=update-rollback.md
      - vpn-down=vpn-down.md
      - vpn-egress-stale=vpn-egress-stale.md
+      # `VpnIngressNodeAnnouncesL2VIP` — the INBOUND remote-access VPN, not the egress tunnel the two
+      # pages above cover. A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease
+      # it holds itself, and vpn-ingress MASQUERADEs clients to the node's own address, so the
+      # forwarded node must announce nothing (measured 2026-10-08).
+      - vpn-ingress-l2-collision=vpn-ingress-l2-collision.md
      # Operator procedure, not an alert page. Served because the component it completes is INERT
      # until three off-repo manual steps are done (kernel module, server key, WAN port-forward) —
      # and because the one check that actually proves it cannot be run from inside the house.
@@ -0,0 +1,165 @@
+# vpn-ingress-l2-collision — the VPN ingress node is announcing a LoadBalancer VIP
+
+**Alerts:** `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) and `VpnIngressNodeLabelMissing`
+(warning, 30m)
+**File:** `platform/monitoring/alerts.yaml`
+**Related:** `/runbooks/vpn-remote-access` is the setup and enrolment runbook for this component.
+`/runbooks/vpn-egress-stale` and `/runbooks/vpn-down` are the **other** VPN — the outbound PIA
+tunnel — and have nothing to do with this.
+
+## What this means here
+
+**A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
+announcer answers ARP for the VIP on behalf of other LAN hosts, but not for the announcing node's
+own request. On the lease holder the neighbour entry settles at `FAILED`, and every packet to that
+VIP is dropped before it leaves the node. Measured on prod 2026-10-08, symmetric across all three
+nodes and five VIPs — each node reached every VIP **except** the ones it alone announced.
+
+For an ordinary workload this is harmless: pods reach Services through the cluster datapath and
+never ARP for a VIP. `platform/vpn-ingress` is the exception. It runs WireGuard with
+`hostNetwork: true` and MASQUERADEs remote clients to the node's own address, so a remote client's
+packets take that node's LAN path and inherit its ARP failures. **If the node the router forwards
+`udp/51820` to is also an L2 lease holder, every remote VPN client silently loses all access to the
+Services behind the VIPs it announces.**
+
+The node terminating WireGuard is marked with `homelab.keithjackson.dev/vpn-ingress=true`, and
+`platform-config/cilium/l2-announcement-policy.yaml` excludes that label from announcing anything,
+so this should be unreachable.
+
+**The symptom your users report will not sound like a network fault.** On 2026-10-08 the forward sat
+on `k3s-prod-1`, which held the `shared` Gateway lease, so all 16 Gateway hostnames were dead over
+the VPN — while PiHole DNS kept answering, because `192.168.2.23` is announced by *two* Services and
+`k3s-prod-2` held the other lease. So the tunnel handshakes, `wg show` looks perfect, names resolve,
+and **every app hangs on connect**. Expect "the VPN connects but nothing loads", not "the VPN is
+down".
+
+Which Services are affected depends on which lease moved — read `{{ $labels.lease }}` from the
+alert. `cilium-l2announce-gateway-cilium-gateway-shared` is the worst case: it is every web app at
+once.
+
+### If it is `VpnIngressNodeLabelMissing` that fired
+
+That is the watchdog, and it means the protection is **switched off**, not that something is merely
+untidy. The exclusion is a `NotIn` on the label, and `NotIn` with an absent key **matches** — so
+with the label on no node, the policy excludes nothing and every node is an announcement candidate
+again. `VpnIngressNodeAnnouncesL2VIP` is simultaneously **inert**: its join matches nothing and
+returns no series, which is indistinguishable from healthy. Treat a quiet critical alert as
+meaningless until this warning is cleared. Usual cause: a node was rebuilt and `node_baseline` has
+not been re-run on it.
+
+## First commands
+
+Run cluster commands through the pinned toolbox, which supplies the kube context:
+
+```bash
+# 1. WHO HOLDS WHAT, and WHO IS THE VPN NODE — the whole question, in two lines.
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n kube-system get lease | grep l2announce'
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/vpn-ingress'
+# Exactly ONE node should carry vpn-ingress=true, and NO l2announce lease should name it.
+
+# 2. IS THE EXCLUSION STILL IN THE LIVE POLICY? Expect a matchExpressions list with a
+#    homelab.keithjackson.dev/vpn-ingress NotIn [true] entry — not a bare matchLabels.
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get ciliuml2announcementpolicy default-l2 -o jsonpath="{.spec.nodeSelector}"; echo'
+
+# 3. IS THE LABEL REACHING PROMETHEUS? If this is empty the alerts are inert even when
+#    the label is on the node (kube-state-metrics metricLabelsAllowlist).
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring exec deploy/kube-prometheus-stack-kube-state-metrics -- \
+     wget -qO- localhost:8080/metrics | grep vpn_ingress | head'
+```
+
+**Confirm the actual damage**, which is the one measurement that is unambiguous. From the WireGuard
+pod on the affected node (it shares the host netns, so this is the path a VPN client takes):
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress get pods -o wide'   # pick the pod on that node
+
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- \
+     sh -c "ping -c1 -W1 192.168.2.20 >/dev/null 2>&1; ip neigh show 192.168.2.20"'
+# BROKEN:  192.168.2.20 dev eno1 FAILED
+# HEALTHY: 192.168.2.20 dev eno1 lladdr bc:e9:2f:7e:c4:a6 STALE|REACHABLE|DELAY
+```
+
+The `ping` is only there to trigger ARP — the VIP does not answer ICMP even when healthy, so
+**ignore its 100% packet loss**; the neighbour state is the answer. Substitute the VIP that belongs
+to `{{ $labels.lease }}` if the Gateway is not the one that moved.
+
+**Which node actually holds the forward** (the router cannot be queried, but the cluster knows):
+only the forwarded node's `wg show` has a peer with an `endpoint` and a recent `latest handshake`.
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'for p in $(kubectl --context k3s-prod -n vpn-ingress get pods -o name); do
+     echo "== $p"; kubectl --context k3s-prod -n vpn-ingress exec ${p#pod/} -c wireguard -- wg show; done'
+```
+
+## Likely causes, most likely first
+
+1. **The label is on the wrong node** — someone re-pointed the router's forward during a node
+   failover and did not move `node_baseline_vpn_ingress_node`. This is the most likely cause by a
+   wide margin, because the router is off-repo and nothing can check it. Confirm with the `wg show`
+   loop above: the node with the live peer endpoint is the real forward target.
+   *Fix:* merge a PR setting `node_baseline_vpn_ingress_node`
+   (`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`) to that node, then re-run the role —
+   `ansible-playbook -i inventory/hosts.ini <playbook> -e k3s_hosts=k3s_prod --tags vpn-ingress`.
+   The role labels the new node and **removes** the label from the old one. Then confirm Cilium
+   re-elected the lease off the newly-labelled node.
+
+2. **The exclusion was edited out of `platform-config/cilium/l2-announcement-policy.yaml`**, or the
+   file was reverted to a bare `matchLabels: {kubernetes.io/os: linux}`. Command 2 above shows this
+   directly. *Fix:* restore the `matchExpressions` form; the file's own header explains why it is
+   shaped that way.
+
+3. **The label is missing entirely** (`VpnIngressNodeLabelMissing` firing) — a node rebuild without
+   a `node_baseline` re-run. *Fix:* re-run the role as above.
+
+4. **The label is on the node but not in Prometheus** — `metricLabelsAllowlist` in
+   `platform/monitoring/kube-prometheus-stack.yaml` lost the `vpn-ingress` entry. Then the critical
+   alert is inert *and* the watchdog fires. Command 3 above distinguishes this from cause 3.
+
+5. **Cilium kept renewing a lease its holder is no longer eligible for.** The exclusion is correct
+   and in place, but the lease never re-elected. This was never verified when the fix was authored
+   (see `decisions/2026-10-17-vpn-ingress-l2-lease-collision.md`). *Fix:* delete that one Lease
+   object to force an election — `kubectl -n kube-system delete lease <name>` — and watch it come
+   back on a different node.
+
+6. **A Cilium upgrade changed `NotIn` semantics or the `nodeSelector` field.** Least likely, but it
+   is the reason this is a runtime alert rather than a static check on the policy text.
+
+## When to escalate or stop
+
+**Act now, don't wait for a window.** Remote access is fully broken for the affected VIPs while this
+is firing, and it does **not** self-heal: a lease holder keeps renewing, so this condition persists
+indefinitely once entered. That is also why the 15m `for:` is safe — a transient move during a node
+reboot or a Cilium roll resolves on its own well inside it.
+
+**The fastest honest mitigation is to move the forward, not the leases.** Re-pointing the router's
+`udp/51820` at a node that holds no lease for the affected VIP restores service in seconds, with no
+cluster change and no new client profiles — all three nodes answer the same keys. Follow it with the
+`node_baseline_vpn_ingress_node` PR and role re-run so the two agree again; until you do, this alert
+is watching the wrong node. Note each node's own announcements when choosing: as of 2026-10-08,
+prod-2 announced crafty `192.168.2.22` and unifi `192.168.2.21`, prod-3 announced plex
+`192.168.2.24`, so neither is a cost-free target.
+
+**Do not** `kubectl edit` the `CiliumL2AnnouncementPolicy` to fix this — Flux owns it and will revert
+you, and the drift is the kind that outlives the memory of making it. The two acceptable hand
+actions are deleting a single Lease to force re-election (cause 5), and `kubectl label node` as a
+*temporary* stopgap while the PR lands — note that a node rebuild or a `node_baseline` run will undo
+a hand-applied label, so it is not a fix.
+
+**Stop and escalate to the operator if** the leases will not leave the excluded node after a forced
+re-election, or if the intended forward target is unclear. Both need a decision about which node
+carries remote access, which is an operator call, not a runbook step.
+
+**Known gap, so you are not misled:** if someone re-points the router without moving the label,
+`VpnIngressNodeAnnouncesL2VIP` watches the wrong node and **stays silent** while the VPN breaks in
+exactly the way described above. There is no in-cluster signal for that today; closing it needs a
+WireGuard exporter publishing which node is actually terminating peer traffic
+(`platform/vpn-ingress/README.md`, "Not built") — the `wg show` loop above is the manual substitute.
+Until then, if remote access is broken and both alerts are quiet, **check the forward against the
+label by hand before believing the alerts.**
@@ -21,7 +21,7 @@ behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's
 DNS, and internet egress out the house WAN.

 Authorisation is one thing only: the device's **public key is in
-`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
+`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.

 > **An enrolled device is as privileged as a machine sitting in the house** — including the three
 > k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
@@ -135,6 +135,7 @@ specific failure in its log — so you never get a server that silently cannot c
 | `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
 | `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
 | `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
+| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |

 ### What a healthy start looks like

@@ -170,14 +171,57 @@ server without a `wg0` "ready".
 On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
 `192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.

-Pick any node — all three run an identical listener. Note which one you chose here:
+> ### You may NOT pick any node. The forward target must be the `vpn-ingress`-labelled node.
+>
+> This step used to say "pick any node — all three run an identical listener". **That sentence
+> caused a two-day silent outage of all remote access on 2026-10-08**, and it is why there is now a
+> cluster-vars knob instead of a free choice here.
+>
+> **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
+> announcer answers ARP for the VIP on behalf of other LAN hosts but not for itself, so on the lease
+> holder the neighbour entry settles at `FAILED` and packets to that VIP are dropped before they
+> leave the node. Measured on prod 2026-10-08:
+>
+> ```text
+> on k3s-prod-1 (held the shared Gateway lease):   ip neigh show 192.168.2.20 -> FAILED
+> on k3s-prod-2 / k3s-prod-3:                      ip neigh show 192.168.2.20 -> lladdr … (prod-1)
+> ```
+>
+> WireGuard runs on `hostNetwork` and MASQUERADEs remote clients to the node's own address, so a VPN
+> client inherits that node's ARP failure. With the forward on `k3s-prod-1`, which held the Gateway
+> lease, **all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved** — so
+> the tunnel handshook, names looked fine, and every single app hung on connect. Nothing reported
+> it.
+>
+`infra/ansible/roles/node_baseline` now labels the forwarded node
+> `homelab.keithjackson.dev/vpn-ingress=true` (from `node_baseline_vpn_ingress_node` in
+> `group_vars/k3s_prod/vars.yml`), and `platform-config/cilium/l2-announcement-policy.yaml` excludes
+> that label from announcing any VIP. **The forward must point at the node carrying that label.** CI
+> cannot read your router, so this parity is yours to hold — `VpnIngressNodeAnnouncesL2VIP` catches a
+> labelled node taking a lease, but it CANNOT catch the forward being moved without the label.
+
+Forward to the node named by `node_baseline_vpn_ingress_node` (`k3s-prod-1` → `192.168.2.10`).
+Note it here:

 ```
 # WAN udp/51820 -> 192.168.2.__  (k3s-prod-_)   set <date>
+# MUST equal node_baseline_vpn_ingress_node in group_vars/k3s_prod/vars.yml
 ```

-**Node loss = failover is this one line.** Re-point the forward at another node address; every
-device keeps working with the same profile, because all three nodes answer the same keys.
+**Node loss = failover is TWO changes, not one.** It used to be advertised as one router line; that
+was wrong, because re-pointing at another node lands the VPN on a node that announces VIPs and
+reintroduces the failure above against a different set of addresses (prod-2 announces crafty and
+unifi; prod-3 announces plex). Do both, in this order:
+
+1. **Merge a PR** setting `node_baseline_vpn_ingress_node` to the new node, then re-run the role
+   (`--tags vpn-ingress`). It labels the new node and removes the label from the old one; Cilium
+   then moves the leases off the newly-labelled node and onto its peers.
+2. **Re-point the router** forward at the new node's address.
+
+Every device keeps working with the same profile either way; all three nodes answer the same keys,
+and the listener stays warm on all three. Doing step 2 without step 1 gives you a tunnel that
+handshakes and routes nothing — and `VpnIngressNodeAnnouncesL2VIP` will **not** catch it, because
+that alert watches the node the knob names, not the node your router actually points at.

 Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
 already silent to unauthenticated probes (README, "Why this is safe to expose").
@@ -218,10 +262,23 @@ It generates the device's keypair, writes a `0600` client profile, prints a QR c

 Then:

-1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
+1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
   `# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
   comment is the only record of which stanza belongs to which device.
-2. PR → merge. Flux rolls the DaemonSet (~1 min).
+2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
+   **content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
+   which changes the pod template, which forces the roll. (It did **not** work this way at first —
+   a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
+   keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:
+
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
+        -o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
+   ```
+
+   The generation must have **increased** since before the merge. If it is still `1`, the roll did
+   not happen and `wg show` will list no peer no matter how long you wait.
 3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
   Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
 4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
@@ -261,7 +318,7 @@ if the router is not delivering the packets, and a `handshake` with no completed

 ## Revoking a device

-Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
+Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
 next reconcile. **Never re-use that address octet**: an old profile would silently start working
 again as a different device.
Author	SHA1	Message	Date
jakman85	0913ea9450	Merge branch 'main' into fix/vpn-ingress-l2-lease-collision gate / gate (pull_request) Successful in 1m55s Details gate / conformance (pull_request) Has been skipped Details	2026-10-18 04:15:58 +00:00
jakman85	1a0b20857a	Merge pull request 'fix(oauth2-proxy): gate startup on Keycloak discovery, or a cold boot CrashLoops the proxy' (#70 ) from fix/oauth2-proxy-oidc-startup-race into main gate / gate (push) Successful in 1m52s Details gate / conformance (push) Has been skipped Details Reviewed-on: #70	2026-10-18 04:14:52 +00:00
jakman85andClaude Opus 5	c150327a30	fix(oauth2-proxy): gate startup on Keycloak discovery, or a cold boot CrashLoops the proxy gate / gate (pull_request) Successful in 1m53s Details gate / conformance (pull_request) Has been skipped Details Exposed by the 2026-10-17 whole-cluster shutdown. oauth2-proxy performs OIDC discovery ONCE at startup and exits 1 if it fails. On a cold boot Keycloak is not serving yet, the shared Gateway answers 503, and the proxy dies: [provider.go:55] Performing OIDC Discovery... [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC configuration: unexpected status "503": no healthy upstream It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was still down minutes later and only a manual pod delete cleared it. There is no retry flag, a readinessProbe cannot help (the container is already dead), and CrashLoopBackOff backoff is not tunable per-pod. Add an extraInitContainers gate (wait-for-idp) that polls the discovery doc until it answers. A pod in Init accrues no CrashLoopBackOff backoff and retries promptly. BOUNDED and FAILS OPEN: after ~5m it starts the proxy anyway, so the worst case is exactly today's behaviour and never something new. extraInitContainers is the chart's OWN hook for this -- its commented example is literally `- name: wait-for-idp` -- and the key was CONFIRMED against the pinned chart (helm show values, 10.7.1) rather than assumed, because a wrong Helm values key is silently ignored. No new base image: the chart already runs an alpine init container for its built-in waitForRedis gate; pinned to 3.22 (verified upstream) rather than the chart's `latest`. Rejected: Flux postRenderers (no precedent in this tree, and the chart has a hook); --skip-oidc-discovery (removes the fetch entirely, but needs four hardcoded endpoints and I could not verify lazy-JWKS offline -- this component gates SSO for seven apps, so the additive option wins). Shell vars are written BARE with no braced form: platform declares postBuild.substituteFrom, and envsubst would blank a braced shell variable and silently break the loop. Same convention as the gitea/nextcloud OIDC hooks. UNAPPLIED; the init container has never run, and the race was not re-created to prove the fix. Check the init container's log on first apply. See the handoff. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-17 22:47:30 -05:00
jakman85andClaude Opus 5	f5353cbd30	fix(vpn-ingress): the VPN ingress node must announce no L2 VIPs, or clients route nothing gate / gate (pull_request) Successful in 2m22s Details gate / conformance (pull_request) Has been skipped Details A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself: the announcer does not answer its own ARP request, so on the lease holder the neighbour entry settles at FAILED and packets to that VIP die before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced. platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and MASQUERADEs remote clients to the node's own address, so a VPN client takes the node's LAN path and inherits its ARP failures. The router's udp/51820 forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16 Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering (its VIP has two Services and k3s-prod-2 held the other lease). The tunnel handshook, names resolved, every app hung, and nothing reported it. on k3s-prod-1: ip neigh show 192.168.2.20 -> FAILED (its own VIP) on prod-2/3: ip neigh show 192.168.2.20 -> lladdr ... (prod-1), 443 OK Mark the node and exclude it from announcing: * node_baseline labels the forwarded node (node_baseline_vpn_ingress_node, k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes the label from any node that no longer matches so moving it MOVES the label. * l2-announcement-policy.yaml excludes that label, so the node ARP-resolves every VIP. Fixes all VIPs for VPN clients, not just the one that collided. Cost: announcement candidacy drops from 3 nodes to 2. * VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels (label added to the kube-state-metrics allowlist), with VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches, so a missing label would silently re-arm the whole failure AND leave the critical alert inert. Four promtool cases, including that inertness. A node LABEL rather than a cluster-vars knob because the alert must read it too, and extract_rules.py rightly refuses a ${...} placeholder in an alert expr -- the rule unit-tested would not be the rule deployed. One source of truth, and it touches no sensitive path. Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and "failover is this one line". Both false; the first caused this. UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the handoff entry's third section. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-17 21:58:12 -05:00
jakman85	9a41a8e1e5	Merge pull request 'Expand plex volume' (#68 ) from plex/volume-expamsion into main gate / gate (push) Successful in 1m53s Details gate / conformance (push) Has been skipped Details Reviewed-on: #68	2026-10-10 16:13:46 +00:00
jakman85	0132c680b3	Update apps/wave-0/plex.yaml gate / gate (pull_request) Successful in 1m53s Details gate / conformance (pull_request) Has been skipped Details	2026-10-10 16:09:53 +00:00
jakman85	439b35a3f0	Expand plex volume gate / gate (pull_request) Failing after 1m51s Details gate / conformance (pull_request) Has been skipped Details	2026-10-10 13:55:15 +00:00
jakman85	e89df78210	Merge pull request 'fix(bulk-nfs): repoint at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled' (#67 ) from fix/bulk-nfs-truenas into main gate / gate (push) Successful in 1m53s Details gate / conformance (push) Has been skipped Details Reviewed-on: #67 Reviewed-by: Keith Jackson <[email protected]>	2026-10-07 06:26:44 +00:00
jakman85andClaude Opus 5	0747849837	fix(bulk-nfs): repoint at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled gate / gate (pull_request) Successful in 1m53s Details gate / conformance (pull_request) Has been skipped Details Both ZFS pools moved off the Proxmox host onto a standalone TrueNAS on 192.168.2.30 -- prod's OWN /24, so the pfSense crossing that made this knob load-bearing is gone, along with the DNAT at 192.168.2.102 it mounted through. Measured 2026-10-07: 192.168.2.102:2049 no longer answers; .30:2049 and :111 do. BULK_NFS_SERVER -> 192.168.2.30 in prod, base and dev. The dev value is marked inert (dev is decommissioned); it carries the real address only so nothing in the tree renders a host that no longer exists. THE OPERATIONAL POINT, and why this is not just a value change: a bound PersistentVolume's NFS source is immutable. Server dry-run, 2026-10-07: spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation so Flux's apply FAILS rather than updating, and the pods keep their old, dead mount. The five PVs must be deleted and recreated -- safe, because Retain never touches data and render.py emits a pre-set claimRef so each PV rebinds its own PVC. The sequence is docs/BULK-NFS-STORAGE.md section 10 and is an OPERATOR action; this commit does not perform it. Pairs with the hazard already on the page: spec.nfs.server is unvalidated at create (any string binds) and immutable at update. A wrong value is accepted silently and then cannot be corrected in place. docs/BULK-NFS-STORAGE.md: sections 1 and 5 rewritten for TrueNAS -- the measured exports, the service settings that decide whether uid 1000 survives the wire, the three stale ZFS sharenfs properties that had to be cleared, and the ACL hazard (acltype=nfsv4 + aclinherit=passthrough means one click in the ACL editor propagates to the whole future library). Section 7 rewritten now that the placeholder is gone, section 9 retargeted from dev to prod, section 10 new. Sections 2/3/4/6 are unchanged by the move. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-07 01:15:04 -05:00
jakman85	ad37acdf53	Merge pull request 'fix(vpn-ingress): content-hash the peer register, or a merged peer never reaches the tunnel' (#66 ) from fix/peers-configmap-content-hash into main gate / gate (push) Successful in 1m51s Details gate / conformance (push) Has been skipped Details Reviewed-on: #66	2026-10-06 07:58:49 +00:00
jakman85andClaude Opus 5	2acfed460a	fix(vpn-ingress): content-hash the peer register, or a merged peer never reaches the tunnel gate / gate (pull_request) Successful in 1m52s Details gate / conformance (pull_request) Has been skipped Details The operator enrolled a phone, merged the peer, and `wg show` listed none. Measured on prod 2026-10-06 with the peer already on main: live wg-peers ConfigMap, active AllowedIPs .. 0 DaemonSet volume reference ................. configMap.name: wg-peers (FIXED) DaemonSet generation/observedGeneration .... 1 / 1 (never rolled) init container, all three pods ............. "0 peer(s)" Part of it was just the 10m reconcile interval. The defect is the fixed name: Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is byte-identical, so no rollout is triggered -- and the init container assembles wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer had no effect until something unrelated restarted the pods. For a REVOCATION that is a security defect, not an inconvenience: delete a stanza, see a green PR, device keeps working. The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet". Never true. And this repo already documents the fix, in a file I had read -- runbooks/kustomization.yaml: "The generated name carries kustomize's content hash, so editing any runbook rolls the server automatically and a stale projection is impossible." So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a configMapGenerator. The operator's enrolled peer is carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the documentation now lives inside the file that is actually mounted. `namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a generated ConfigMap inherits no namespace, everything else here declares vpn-ingress, and kustomize's nameReference will not rewrite across a namespace boundary -- without it the build mounts the unhashed name and the pods wedge on a missing volume. Caught only by building the output and reading the volumes. new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that after the rename the only symptom would have been duplicate detection quietly never firing again. Stale peers.yaml references swept from six files including the script's own user-facing output. Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content (edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's real allocation. No sensitive paths; merges on a green gate. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-06 02:55:59 -05:00
jakman85	e1bebb857e	Merge pull request 'Added my phone to peers' (#65 ) from vpn/add-keith-phone into main gate / gate (push) Successful in 1m53s Details gate / conformance (push) Has been skipped Details Reviewed-on: #65	2026-10-06 07:42:04 +00:00