Merge branch 'main' into fix/vpn-ingress-l2-lease-collision

Reviewed-on: #70

apps/wave-0/plex.yaml

@@ -77,7 +77,7 @@ spec:
   storageClassName: longhorn
   resources:
     requests:
-      storage: 8Gi
+      storage: 16Gi
 ---
 # PVC plex-media (R-STATE)
 apiVersion: v1

catalog/plex.yaml

@@ -351,7 +351,7 @@ exposure: public              # edge port-forward + public DNS; clients + Seerr
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: config
-    size: 8Gi
+    size: 16Gi
     class: longhorn           # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
     mountPath: /config        # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
                               # renderer default /data/config. Without this the PVC sat unused and Plex wrote to the

catalog/qbittorrent.yaml

@@ -359,7 +359,7 @@ storage:
                                 # The live /config contents were copied onto each PVC before this landed,
                                 # so API keys and credentials other apps reference were preserved.
   # THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
-  # (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
+  # (192.168.2.30:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
   # /downloads, which is precisely where this app is ALREADY configured to write.
   #
   # WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
@@ -434,7 +434,7 @@ dependencies: []                # default-deny: reaches only DNS; arr reach it n
                                 # for weeks and neither has any CNP naming the NFS server or port 2049
                                 # (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
                                 # egress gateway off this traffic (its selector is a podSelector), and
-                                # 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
+                                # 192.168.2.30 is inside the 192.168.0.0/16 excludedCIDR besides — see
                                 # platform/vpn-egress/egress-steering.yaml, "BULK NFS".
 ingressRoutes:
   - host: qbittorrent.${BASE_DOMAIN}

clusters/base/cluster-vars.yaml

@@ -234,26 +234,28 @@ data:
   WG_CLIENT_CIDR: "10.90.0.0/24"             # remote VPN clients; .1 = server. NOT a LAN range
 
   # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
-  # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
-  # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
+  # THE address of the NAS whose ZFS datasets back every `class: bulk` volume: the media library
+  # (Storage/plex, 8.33 T — plex, sonarr, radarr, qbittorrent) and the legacy Nextcloud archive
   # (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
   # `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
   # declared in the ServiceDefinition, because the path is identical in every environment and only
   # the ADDRESS is not.
   #
-  # WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
-  # The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
-  #   dev  nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
-  #        NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
-  #        /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
-  #        per the convention of the LAN block above.
-  #   prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
-  #        22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
-  #        the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
-  #        DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
-  # So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
-  # blast radius of shipping it unset, which is NOT the usual "fails to apply".
-  BULK_NFS_SERVER: "192.168.1.9"
+  # IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read "the two clusters are on
+  # OPPOSITE SIDES of pfSense": the datasets lived on a Proxmox host whose only interface was
+  # vmbr0 192.168.1.9/24 on pfSense's LAN, which prod (192.168.2.0/24, pfSense's WAN) could reach
+  # only through a DNAT at 192.168.2.102. Both pools were physically moved onto a standalone
+  # TrueNAS at 192.168.2.30 — prod's OWN /24 — so there is no crossing to arrange and no candidate
+  # values to choose between. Measured 2026-10-07: 192.168.2.102:2049 no longer answers;
+  # 192.168.2.30:2049 and :111 do. There is now exactly ONE storage host and one correct value;
+  # this stays a knob only because the renderer reads it per-environment.
+  #
+  # *** CHANGING THIS VALUE IS NOT A PLAIN RECONCILE. *** A bound PersistentVolume's spec.nfs is
+  # immutable — measured by server dry-run 2026-10-07:
+  #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+  # so Flux's apply FAILS rather than updating, and the pods keep their old mount. The PVs must be
+  # deleted and recreated; the operator sequence is docs/BULK-NFS-STORAGE.md §10.
+  BULK_NFS_SERVER: "192.168.2.30"
 
   # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
   # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,

clusters/dev/kustomization.yaml

@@ -279,15 +279,15 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.1.55"
-      # Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
-      # (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
-      # Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
-      # Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
-      # end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
-      # change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
+      # Bulk (external NFS) storage host. THIS VALUE IS INERT — dev is decommissioned, its nodes
+      # (192.168.1.40/.41/.42) no longer exist, and the Proxmox host this used to name (192.168.1.9)
+      # went with the pools it held. It is restated per this overlay's "supply every knob"
+      # convention, not because anything consumes it. It carries the real address so that no file
+      # in the tree renders a host that is gone; it has NOT been proven from a dev node, because
+      # there is no dev node to prove it from.
       - op: replace
         path: /data/BULK_NFS_SERVER
-        value: "192.168.1.9"
+        value: "192.168.2.30"
       # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
       # barmanObjectStore and Velero's backupStorageLocation write to.
       #

clusters/prod/kustomization.yaml

@@ -540,60 +540,48 @@ patches:
         path: /data/LAN_CIDR
         value: "192.168.2.0/24"
       # ------------------------------------------------------------------------------------------
-      # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
+      # BULK (EXTERNAL NFS) STORAGE HOST — the TrueNAS at 192.168.2.30.
       # ------------------------------------------------------------------------------------------
       # Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
-      # radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
+      # radarr, qbittorrent) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
       #
-      # THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
-      # vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
-      # 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
-      # 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
-      # outcomes give DIFFERENT values here:
-      #     port-forward 192.168.2.102:2049 -> 192.168.1.9:2049   =>  "192.168.2.102"
-      #     a WAN->LAN route to 192.168.1.0/24                    =>  "192.168.1.9"
-      # Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
-      # ever became reachable — mounts something unintended. So it stays a placeholder.
+      # THIS WAS THE REPO'S ONE LOAD-BEARING PLACEHOLDER, AND THE REASON IT COULD BE IS NOW GONE.
+      # The datasets lived on a Proxmox host with a single interface on pfSense's LAN
+      # (192.168.1.9), which prod could reach only through a DNAT at 192.168.2.102 — so the correct
+      # value here depended on an undecided operator choice, and guessing it would have authored a
+      # cluster that mounts nothing. Both pools were physically moved onto a standalone TrueNAS at
+      # 192.168.2.30, prod's OWN /24. No crossing to arrange, no candidates to choose between.
+      # Measured 2026-10-07: 192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and :111 do.
       #
-      # WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
-      # NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
-      # over v3 was in part this decision's cost: v3 would have needed four ports and three of them
-      # on ephemeral numbers.
-      #
-      # WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
-      # Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
-      # the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
-      # address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
-      # 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
-      # addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
-      #
-      # *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
-      # A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
-      # placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
-      # failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
-      # TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
-      # ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
-      # apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
-      # a revision containing the bulk volumes; the dev overlay renders a working value, which is
-      # where these manifests are meant to be proven first.
+      # ONE PORT IS STILL ALL IT NEEDS: tcp/2049. The PVs mount nfsvers=4.1 and NFSv4 carries mount
+      # and lock in-protocol — no rpcbind/111, no mountd, no statd. The NAS advertises
+      # `+3 +4 +4.1 +4.2` (/proc/fs/nfsd/versions, measured), so 4.1 negotiates.
       - op: replace
         path: /data/BULK_NFS_SERVER
-        # 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
-        # host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
-        # reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
-        # keeps the base value 192.168.1.9.
+        # VERIFIED ON THE NAS 2026-10-07, before any cluster touched it (`exportfs -v`):
+        #   /mnt/Storage/plex       192.168.2.10/.11/.12  rw,sec=sys,secure,root_squash,no_all_squash
+        #   /mnt/Storage/nextcloud  192.168.2.10/.11/.12  ro,sec=sys,secure,root_squash,no_all_squash
         #
-        # PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
-        #   - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
-        #   - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
-        #     carries no_root_squash, so a root write would have passed and proved nothing, while
-        #     LSIO images default to PUID 911 and would have gone Ready then failed every import
-        #     with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
-        #   - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
-        #     real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
-        # pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
-        # so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
-        value: "192.168.2.102"
+        # no_all_squash is the flag the entire permission model rests on: client uids pass through
+        # untouched, so the arrs' PUID=1000 arrives as the owner of a 0755 1000:3000 tree. sec=sys
+        # keeps that numeric rather than principal-mapped. The export deliberately does NOT carry
+        # no_root_squash, which the Proxmox one did: nothing here needs it, because every write to
+        # the tree happens after the LSIO drop to uid 1000 and the images only chown their Longhorn
+        # /config. Both subPath directories (plex/downloads, nextcloud/nextcloud) already exist at
+        # 0755 1000:3000, so a squashed root can still traverse them for subPath resolution.
+        #
+        # The ro= posture on the archive is now SERVER-enforced, which the old export also managed.
+        # The media tree's read-only half (plex) stays a CLIENT-side guarantee — all four media pods
+        # can land on any node, so the server sees the same three addresses for reader and writers
+        # alike and cannot tell them apart. Unchanged by the move; see docs/BULK-NFS-STORAGE.md §5.
+        #
+        # *** CHANGING THIS VALUE REQUIRES DELETING AND RECREATING THE PVs. *** A bound PV's
+        # spec.nfs is immutable — measured by server dry-run 2026-10-07:
+        #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+        # so Flux's apply FAILS rather than updating and the pods keep their old, dead mount. The
+        # operator sequence is docs/BULK-NFS-STORAGE.md §10. Retain + the renderer's pre-set
+        # claimRef make it safe: no data is touched and each recreated PV rebinds its own PVC.
+        value: "192.168.2.30"
       # ==========================================================================================
       # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
       # ==========================================================================================

decisions/2026-10-07-bulk-nfs-truenas.md

@@ -0,0 +1,85 @@
+---
+date: 2026-10-07
+branch: fix/bulk-nfs-truenas
+title: Repoint bulk NFS at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
+---
+
+## What changed
+
+Both ZFS pools were physically moved off the Proxmox host onto a standalone TrueNAS Community
+Edition at **192.168.2.30**, which sits on prod's own `192.168.2.0/24`. `BULK_NFS_SERVER` moves from
+`192.168.2.102` (the pfSense DNAT prod used to mount through) to `192.168.2.30`.
+
+- `clusters/prod/kustomization.yaml` — `BULK_NFS_SERVER` → `192.168.2.30`. The comment block that
+  justified the DNAT, and the "NOT YET DECIDED / THIS PLACEHOLDER IS LOAD-BEARING" banner, are
+  replaced with the measured export configuration and the recreate requirement below.
+- `clusters/base/cluster-vars.yaml` — base default `192.168.1.9` → `192.168.2.30`. That host no
+  longer exists.
+- `clusters/dev/kustomization.yaml` — same; marked explicitly **inert**, since dev is decommissioned
+  and its nodes are gone. It carries the real address only so nothing in the tree renders a dead one.
+- `docs/BULK-NFS-STORAGE.md` — §1 and §5 rewritten for TrueNAS (exports, service settings, the ZFS
+  `sharenfs` properties that had to be cleared, and the ACL hazard); §7 rewritten now that the
+  placeholder is gone; §9 retargeted from dev to prod; **new §10** documents the PV
+  delete-and-recreate. §2/§3/§4/§6 are unchanged by the move.
+- `catalog/qbittorrent.yaml`, `platform/monitoring/node-pressure-dashboard.yaml` — two comment/prose
+  references to `192.168.2.102` as the live export address.
+
+**The headline operational fact: repointing this knob is NOT a plain reconcile.** A bound PV's NFS
+source is immutable, so Flux's apply fails rather than updating and the pods keep their old, dead
+mount. The five PVs must be deleted and recreated. That sequence is §10 and is an operator action;
+it is **not** performed by this PR.
+
+## What I verified
+
+**On the NAS (192.168.2.30), measured 2026-10-07 by the operator at my request:**
+
+- `zfs list` — `Storage` → `/mnt/Storage`, `Storage/plex` → `/mnt/Storage/plex`,
+  `Storage/nextcloud` → `/mnt/Storage/nextcloud`. The PV paths resolve unchanged.
+- Ownership intact: `/mnt/Storage/plex`, `tv/`, `downloads/`, `movies/`,
+  `downloads/complete/`, `nextcloud/nextcloud/` all `drwxr-xr-x 1000:3000`. `movies/` 1171 entries
+  and `tv/` 107, against 1165 / 107 measured on Proxmox.
+- `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`), `aclinherit=passthrough`, and no
+  non-trivial ACLs — mode bits still govern.
+- Three stale `sharenfs` dataset properties (`plex`, `nextcloud`, `docker-volumes`) carried over from
+  Proxmox; cleared with `zfs inherit`, re-checked empty.
+- `exportfs -v`: both shares to `192.168.2.10/.11/.12` with `sec=sys,secure,root_squash,no_all_squash`,
+  `rw` on plex and `ro` on nextcloud.
+- `/proc/fs/nfsd/versions` → `+3 +4 +4.1 +4.2`, so the PVs' `nfsvers=4.1` negotiates.
+- **A defect was found and fixed during setup:** the plex share listed `192.168.2.13` — an address no
+  node holds — and so omitted `192.168.2.12` (k3s-prod-3). Corrected and re-verified.
+
+**From the cluster:**
+
+- Node InternalIPs are `192.168.2.10/.11/.12` (`kubectl get nodes`), confirming `.13` was wrong.
+- All five NFS PVs currently name `192.168.2.102`.
+- `192.168.2.30:2049` and `:111` accept connections; `192.168.2.102:2049` does not.
+- **`spec.nfs` immutability, by server-side dry-run** — this is the finding that shapes the change:
+  `spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation`.
+- The pods report `Ready` while their mounts are dead: a `stat /media` answers from attribute cache,
+  but a `readdir` blocks indefinitely (the PVs mount `hard`). Hence "scale to zero first" in §10.
+
+**Gate:** see the PR; run in the pinned toolbox from this worktree.
+
+## What I did NOT verify
+
+- **Nothing has mounted 192.168.2.30.** No pod has touched the new server — the PVs still point at
+  the old address and cannot be updated in place. Every §9 probe (numeric-id passthrough, the uid-1000
+  write, the hardlink/EXDEV check, Plex's read-only posture) is **outstanding**, not passing. The
+  equivalents passed against the old host on 2026-09-12; that is not evidence about this one.
+- **The §10 sequence is reasoned, not executed.** Retain + the renderer's pre-set `claimRef` are read
+  from the manifests and from `catalog/render.py`; I have not watched a PVC rebind after a PV delete
+  in this cluster.
+- **`root_squash` is a behaviour change from the Proxmox export**, which carried `no_root_squash`. The
+  argument that nothing needs root on the tree (LSIO drops to 1000; the images chown only their
+  Longhorn `/config`; both subPath directories already exist at 0755) is reasoned from the images and
+  the measured permissions. It has not been exercised. If a mount or subPath resolution fails with
+  EACCES, Maproot `root:root` reproduces the old export and is the first thing to try.
+- **Whether NFSv3 can be disabled.** Enabled Protocols is `NFSv3, NFSv4`; the PVs only need v4.1, so
+  narrowing would close the rpcbind/mountd/statd surface. Not done — it is a change to make after the
+  cutover is proven, not during it.
+- **Whether pfSense / the legacy stack is actually retired.** I measured only that
+  `192.168.2.102:2049` stopped answering. The other legacy consumers of these datasets
+  (`192.168.1.6`, `10.123.1.10`, `192.168.1.18`) were dropped from the new exports; if any is still
+  live, it has lost access. Not checked.
+- **L2/L3 conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline subset
+  only. Nothing in the gate exercises an NFS mount.

decisions/2026-10-17-oauth2-proxy-oidc-startup-race.md

@@ -0,0 +1,99 @@
+---
+date: 2026-10-17
+branch: fix/oauth2-proxy-oidc-startup-race
+title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
+---
+
+## What changed
+
+Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
+discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
+serving yet, so the shared Gateway answers 503 and the proxy dies:
+
+```
+[provider.go:55] Performing OIDC Discovery...
+[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
+             failed to discover OIDC configuration: unexpected status "503": no healthy upstream
+```
+
+It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
+replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
+retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
+is already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+
+`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
+polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
+accrue CrashLoopBackOff backoff and retries promptly, which is the point.
+
+**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
+case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
+for a pod wedged in `Init` forever, which is not an improvement.
+
+## Why this shape and not the alternatives
+
+* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
+  example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
+  pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
+  at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
+  "reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
+* **No new base image.** The chart already runs an `alpine` init container for its built-in
+  `initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
+  `3.22` (verified to exist upstream) rather than the chart's own `latest`.
+* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
+  this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
+  the chart already has a hook for.
+* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
+  more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
+  oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
+  the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
+  the operator out of all of them, so the reversible, additive option wins.
+
+The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
+with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
+and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
+silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
+the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.
+
+## What I verified
+
+* **The failure is measured, not theorised.** The log above is from the live pod
+  (`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
+  with `--previous` after the cold boot. The surviving replica's log shows the same startup
+  succeeding once Keycloak was up, which is what identifies this as a race rather than a config
+  fault.
+* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
+  clean and both replicas have been healthy since.
+* **The chart key is real**, at the pinned version range: `helm search repo` resolves
+  `>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
+  `extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
+* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
+  assumed from memory.
+* The edited HelmRelease parses as YAML.
+
+## What I did NOT verify
+
+* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
+  seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
+  replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
+  seven apps, so **do it when you can watch it**, not unattended.
+* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
+  oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
+  from the measured failure, not demonstrated against it.
+* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
+  `ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
+  failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
+  degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
+  anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
+  init container's log on first apply**: it should print "Keycloak discovery reachable" within
+  seconds on a warm cluster, not the give-up message.
+* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
+  re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
+  good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
+  (`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
+  Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
+  this should hold — unverified.
+* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
+  not a measured bound.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
+  subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

decisions/2026-10-17-vpn-ingress-l2-lease-collision.md

@@ -0,0 +1,137 @@
+---
+date: 2026-10-17
+branch: fix/vpn-ingress-l2-lease-collision
+title: "A node cannot reach a VIP it announces — so the VPN ingress node must announce nothing"
+---
+
+## What changed
+
+The first device enrolled on the remote-access VPN handshook, resolved names, and could not open a
+single connection to any cluster web app. The tunnel was not at fault. The fault is a property of
+Cilium L2 announcement that nothing in this repo knew about:
+
+**A node cannot reach a LoadBalancer VIP whose `CiliumL2AnnouncementPolicy` lease it holds itself.**
+The announcer answers ARP for the VIP on behalf of other LAN hosts but not for its own request, so
+on the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP is dropped
+before it leaves the node.
+
+`platform/vpn-ingress` is uniquely exposed because it runs WireGuard with `hostNetwork: true` and
+MASQUERADEs remote clients to the node's own address — a VPN client's packets take the node's LAN
+path and inherit its ARP failures. Ordinary pods never do. The router's `udp/51820` forward pointed
+at `k3s-prod-1`, which held `cilium-l2announce-gateway-cilium-gateway-shared`, so **all 16 `shared`
+Gateway hostnames were dead over the VPN** — while PiHole DNS kept answering, because
+`192.168.2.23` is announced by *two* Services and `k3s-prod-2` held the other lease. Names resolved
+and every app hung on connect.
+
+Seven files:
+
+* **`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`** — new
+  `node_baseline_vpn_ingress_node`; the matching node gets
+  `homelab.keithjackson.dev/vpn-ingress=true`, and a second task **removes** the label from any node
+  that no longer matches, so moving the value moves the label rather than adding a second one. Same
+  shape as `roles/vpn_egress_gateway`'s `egress-gateway` label. Tagged `vpn-ingress`; skipped
+  entirely when the value is empty.
+* **`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`** — set to `k3s-prod-1`, with a
+  `[parity]` note that it must equal the router's forward.
+* **`platform-config/cilium/l2-announcement-policy.yaml`** — `nodeSelector` becomes a
+  `matchExpressions` list keeping `kubernetes.io/os: linux` and adding
+  `homelab.keithjackson.dev/vpn-ingress NotIn ["true"]`. The VPN node then holds no lease, so it
+  ARP-resolves every VIP. This fixes **all** VIPs for VPN clients (Gateway, crafty, plex, pihole,
+  unifi), not just the one that happened to collide.
+* **`platform/monitoring/kube-prometheus-stack.yaml`** — the label added to kube-state-metrics'
+  `metricLabelsAllowlist`, which is what lets the invariant be watched at all.
+* **`platform/monitoring/alerts.yaml`** — `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) joins
+  `kube_lease_owner` against `kube_node_labels` via `label_replace`, plus
+  `VpnIngressNodeLabelMissing` (warning, 30m) as its watchdog.
+* **`platform/monitoring/tests/vpn-ingress-l2.test.yaml`** — four promtool cases: fires on the real
+  collision, quiet in the fixed state, quiet across a transient 11-minute lease move (pinning the
+  `for: 15m`), and the label-absent case asserting the critical alert is inert *and* the watchdog
+  fires.
+* **`runbooks/vpn-ingress-l2-collision.md`** (new, registered in `runbooks/kustomization.yaml`),
+  **`runbooks/vpn-remote-access.md`**, **`platform/vpn-ingress/{daemonset.yaml,README.md}`** — step 3
+  of the setup runbook said *"pick any node — all three run an identical listener"* and *"node loss =
+  failover is this one line"*. Both were false and the first caused this. Failover is now documented
+  as two ordered changes.
+
+**Design note — why a node label and not a `cluster-vars` knob.** The first attempt added
+`WG_INGRESS_NODE` to `clusters/base/cluster-vars.yaml` and substituted it into both the policy and
+the alert. The alert half is **impossible**: `platform/monitoring/tests/extract_rules.py` rejects any
+alert whose `expr` contains a `${...}` placeholder, because the rule it would unit-test is not the
+rule deployed. Rather than duplicate the node name into the alert as a literal (two sources that can
+skew), the label is readable from both sides — the selector matches on it and `kube_node_labels`
+carries it into Prometheus. One source of truth, and as a side effect the change touches no
+sensitive path, so it merges on a green gate rather than needing a human review.
+
+The DaemonSet deliberately gained **no** probe for this. A readinessProbe on VIP reachability would
+wedge a rollout behind an unrelated Gateway outage (`maxUnavailable: 1`); a liveness probe would tear
+down a tunnel carrying LAN traffic correctly.
+
+## What I verified
+
+Measured live against prod, `--context k3s-prod`, on 2026-10-08:
+
+* **The whole tunnel is healthy**, which is what made this hard: `wg show` on `k3s-prod-1` shows the
+  peer with a real endpoint and bidirectional transfer (346 KiB in / 200 KiB out);
+  `net.ipv4.ip_forward=1`; `wg0` up at `10.90.0.1/24` MTU 1420; `iptables -t nat -S POSTROUTING`
+  carries the `MASQUERADE` rule with 26 packets; `FORWARD -i/-o wg0` ACCEPT with 108 / 171 packets;
+  DaemonSet generation 2, 3/3 ready on the content-hashed `wg-peers-m9kcckcb5t`.
+* **The ARP failure and its symmetry.** On `k3s-prod-1`: `ip neigh show 192.168.2.20` → `FAILED`,
+  while `.21`/`.23` resolve to prod-2's MAC `04:0e:3c:96:b0:94`. On prod-2/prod-3, `.20` resolves to
+  prod-1's `bc:e9:2f:7e:c4:a6`. TCP agrees: `.20:443` and `.20:80` FAIL from prod-1, OK from the
+  other two. Repeated across 3 nodes × 5 VIPs — each node reached every VIP except the ones it alone
+  announced (prod-2 ✗ `.22`/`.21`; prod-3 ✗ `.24`).
+* **Not an artefact of probing from the host netns.** `curl --interface wg0` from prod-1 fails the
+  same way, and a neighbour entry is node-global state, so the forwarded path cannot differ in ARP
+  resolution.
+* **The Gateway itself was never broken:** from the workstation (192.168.2.133)
+  `home.keithjackson.dev` → `200`, `plex.home.keithjackson.dev` → `401`, both at `192.168.2.20`.
+  From prod-1 the router (`.1`) and TrueNAS (`.30`) are `REACHABLE`.
+* **`NotIn` semantics, empirically** — the basis for the fail-back claim in the policy's comment:
+  `kubectl get nodes -l 'kubernetes.io/hostname notin (k3s-prod-1)'` returns prod-2 and prod-3, and
+  `NotIn` on a key no node carries returns **all three**. So an unapplied label excludes nothing and
+  degrades to the old behaviour rather than taking every VIP off the LAN.
+* **The alert metric is really scraped:** `kube_lease_owner` is served by kube-state-metrics (55
+  series, every `cilium-l2announce-*` lease with its holder in `lease_holder`), and `kube_node_labels`
+  already carries the two `egress-*` labels through the same allowlist (`k3s-prod-3` shows
+  `egress-gateway=true`), so the third entry follows a proven path.
+* **The four promtool cases pass** (`platform/monitoring/tests/run.sh` → `SUCCESS`, 55 rules
+  checked), including that `VpnIngressNodeAnnouncesL2VIP` is inert when the label is absent.
+* All edited YAML parses; the rendered `nodeSelector` is the intended two-expression list.
+
+## What I did NOT verify
+
+* **The fix is UNAPPLIED and therefore UNPROVEN end to end.** Applying it needs a mutating
+  `kubectl`/`flux reconcile` plus an Ansible run, which are operator actions (CLAUDE.md §5). I have
+  **not** observed `k3s-prod-1` resolve `192.168.2.20` after the exclusion lands, nor a VPN client
+  reaching an app. The prediction rests on the measured symmetry (non-announcing nodes resolve every
+  VIP, 15/15 data points), not on a test of the changed state. **Verify after merge, in this order:**
+  re-run `node_baseline --tags vpn-ingress`; `kubectl get nodes -L
+  homelab.keithjackson.dev/vpn-ingress` (exactly one node); `kubectl -n kube-system get lease | grep
+  l2announce` (no holder is that node); `ip neigh show 192.168.2.20` from the prod-1 wireguard pod
+  (expect a `lladdr`, not `FAILED`); then a client.
+* **That Cilium actually releases a lease when its holder stops matching the `nodeSelector`.** I
+  assumed re-election and did not test it. If Cilium keeps renewing, the affected Leases must be
+  deleted once by hand to force an election. Check this first if the fix appears not to work.
+* **The brief interruption on apply.** Moving the Gateway and `pihole-dns-udp` leases off prod-1 will
+  blip `192.168.2.20` and `192.168.2.23` for as long as re-election plus gratuitous ARP takes. Not
+  measured, and I did not test whether in-flight LAN sessions survive it.
+* **The Ansible tasks have never been executed.** They parse as YAML and follow
+  `roles/vpn_egress_gateway`'s shape, but neither the label task nor the label-removal task has run,
+  so the `changed_when`/`failed_when` conditions and the `k3s kubectl label ... -` removal path are
+  unproven. The removal task's `failed_when` tolerates `'not found'` in stderr; that string was
+  assumed from kubectl's behaviour, not observed.
+* **The residual off-repo coupling is NOT closed and cannot be, as built.** The router's forward is
+  unreadable from the cluster. If someone re-points it without moving the label, the alert watches
+  the wrong node and stays silent while the VPN breaks identically against a different set of VIPs.
+  Only the runbook guards it. Closing it needs a WireGuard exporter publishing which node is
+  actually terminating peer traffic (`wg show` knows — only the forwarded node has a peer
+  `endpoint`); listed as "Not built" in the component README and **recommended follow-up, not done
+  here**.
+* **Why Cilium behaves this way** — intended, a known upstream issue, or fixed in a later version. I
+  measured the behaviour and did not read Cilium's source or issue tracker, so I cannot say whether
+  an upgrade would remove the need for this exclusion.
+* **`192.168.2.21:8443` (unifi) fails from all three nodes**, including ones where ARP resolves, so
+  it is a different, pre-existing fault. Not investigated, not addressed.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1 offline
+  subset only; none of the above is covered by a test that runs in the gate.
+* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).

docs/BULK-NFS-STORAGE.md

@@ -1,61 +1,43 @@
 # Bulk NFS storage — the media library and the legacy Nextcloud archive
 
-The operator-facing companion to the `class: bulk` storage entries in `catalog/{plex,sonarr,radarr,nextcloud}.yaml`.
-Everything here was **measured** on the storage host on 2026-09-07, over read-only SSH; nothing is
-assumed. It resolves the long-standing **OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
+The operator-facing companion to the `class: bulk` storage entries in
+`catalog/{plex,sonarr,radarr,qbittorrent,nextcloud}.yaml`.
+
+**Rewritten 2026-10-07 for the move off Proxmox onto a standalone TrueNAS at 192.168.2.30.** The
+layout (§2), the mount design (§3, §4) and the uid model (§6) are unchanged by the move and were
+measured on the old host on 2026-09-07. Everything about the storage host itself — §1, §5, §10 —
+was re-measured on the NAS on 2026-10-07. Nothing here is assumed. It resolves the long-standing
+**OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
 case that actually exists here — attaching datasets that already hold data — and leaves dynamic bulk
 provisioning unimplemented, because nothing needs it.
 
 ---
 
-## 1. The storage host, and why prod cannot reach it
+## 1. The storage host
 
-The bulk data lives on the **Proxmox host's ZFS pool**. That host has exactly one interface:
+The bulk data lives on a **standalone TrueNAS** (Community Edition) at **192.168.2.30** — the same
+`192.168.2.0/24` as prod's nodes (`192.168.2.10/.11/.12`), so it is reached directly, with no NAT
+and no routing decision.
 
-```
-vmbr0  192.168.1.9/24        default via 192.168.1.1   (pfSense LAN)
-```
-
-| | nodes | reaches 192.168.1.9? |
-|---|---|---|
-| **dev** | 192.168.1.40 / .41 / .42 | **Yes** — same L2 segment, and it already holds an NFS export from this host (`/mnt/Storage/k8s-volumes`, in `/etc/exports`) |
-| **prod** | 192.168.2.10 / .11 / .12 | **No** — 192.168.2.0/24 is pfSense's **WAN**. Ports 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. 2049 and 111 do not. |
-
-Opening the storage path for prod is a **pending operator decision**. Its two candidate outcomes give
-different values for the `BULK_NFS_SERVER` cluster-var:
-
-| outcome | `BULK_NFS_SERVER` for prod |
-|---|---|
-| port-forward `192.168.2.102:2049` → `192.168.1.9:2049` | `192.168.2.102` |
-| a WAN→LAN route to 192.168.1.0/24 | `192.168.1.9` |
-
-Until it lands, `clusters/prod` holds `TBD-PROD-BULK-NFS-SERVER`. **See §7 for why that placeholder
-behaves differently from the repo's other ones.**
+**This section used to be the hard part of this page.** The datasets were on the Proxmox host, whose
+only interface was `vmbr0 192.168.1.9/24` on pfSense's **LAN**, while prod sits on pfSense's **WAN**.
+tcp/2049 did not traverse that, so prod mounted through a DNAT at `192.168.2.102`, and which address
+belonged in `BULK_NFS_SERVER` depended on an undecided operator choice. Both pools were physically
+moved on 2026-10-07 (`docs/plans/proxmox-to-n5-air-migration.md`, Phase C/D) and all of that is
+simply gone. Measured 2026-10-07: **192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and
+:111 do.**
 
 ### It only needs ONE port
 
 The PVs mount `nfsvers=4.1`. **NFSv4 carries mount and lock in-protocol**, so the only thing that
 must be reachable is **tcp/2049** — no rpcbind/111, no mountd, no statd. NFSv3 would have needed
-four ports, three of them on ephemeral numbers, which is a materially worse firewall story. The
-server advertises `+3 +4 +4.1 +4.2` (`/proc/fs/nfsd/versions`), so v4.1 is available and is the
-conservative floor.
+four ports, three of them on ephemeral numbers. The NAS advertises `+3 +4 +4.1 +4.2`
+(`/proc/fs/nfsd/versions`, measured 2026-10-07), so 4.1 is available and is the conservative floor.
 
-### The export's client list is the SAME under both outcomes — measured, not assumed
-
-**pfSense DNATs the existing port-forward WITHOUT source NAT.** Connecting from this workstation
-(192.168.2.132) to `192.168.2.102:22`, the storage host logged the peer as `192.168.2.132`:
-
-```
-SSH_CONNECTION=192.168.2.132 39768 192.168.1.9 22
-ss -tn  ->  192.168.1.9:22   192.168.2.132:39768
-```
-
-The original WAN-side source address survives the translation (return traffic finds its way back via
-the host's default route, 192.168.1.1). So under **either** outcome the NFS server will see prod's
-**real** node addresses — 192.168.2.10/.11/.12 — and the `sharenfs` change in §5 is the same either
-way. **It does not have to wait for the routing decision.**
-
----
+**NFSv4 is not on by default and its absence does not look like a storage problem.** TrueNAS ships
+NFSv3-only until NFSv4 is added to *Services → NFS → Enabled Protocols*. Without it the mount never
+completes and the pods sit in `ContainerCreating` — no permission error, nothing in the app logs,
+just no volume.
 
 ## 2. The layout, as found
 
@@ -107,16 +89,18 @@ why it is mounted read-only; see §6.
 | plex | `/mnt/Storage/plex` | `/media` | — | **ro** | ReadOnlyMany |
 | sonarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
 | radarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
+| qbittorrent | `/mnt/Storage/plex` | `/downloads` | `downloads` | rw | ReadWriteMany |
 | nextcloud | `/mnt/Storage/nextcloud` | `/mnt/legacy-nextcloud` | `nextcloud` | **ro** | ReadOnlyMany |
 
 In-app paths that follow: Plex libraries at `/media/movies`, `/media/tv`, `/media/photos`; Sonarr
 Root Folder `/media/tv`; Radarr Root Folder `/media/movies`; both importing from
 `/media/downloads/complete/{tv-sonarr,radarr}`.
 
-**All three media consumers use the identical mountPath on purpose.** When qbittorrent is eventually
-given the same export (it is not in this change), the arrs and the download client will already agree
-on absolute paths, which is the second precondition for hardlinked imports — otherwise every import
-needs a Remote Path Mapping and quietly falls back to copying when one is missing.
+**The three library consumers use the identical mountPath on purpose**, and qbittorrent has since
+been given the same export (narrowed by `subPath: downloads`, presented at `/downloads` to match its
+existing `Session\DefaultSavePath`). The arrs and the download client therefore agree on absolute
+paths, which is the second precondition for hardlinked imports — otherwise every import needs a
+Remote Path Mapping and quietly falls back to copying when one is missing.
 
 **`/media` and not `/data`, and that is not cosmetic.** The arrs' Longhorn config PVC renders at the
 renderer default `/data/config`. A media mount at `/data` would be mounted **over** it and mask the
@@ -147,62 +131,123 @@ Narrowing that would cost hardlinks, which is the worse trade.
 
 ---
 
-## 5. The export change the operator must apply — STRICTLY ADDITIVE
+## 5. The exports, as configured on TrueNAS — and the settings that decide permissions
 
-**Do not apply this from the repo.** It is an operator step on the storage host, recorded here so the
-exact value is reviewable. The existing consumers **must keep working**: 192.168.1.6, 10.123.1.10 and
-192.168.1.18 on the media tree, and 192.168.1.6 and 192.168.1.5 on the Nextcloud dataset, are the
-live non-k3s prod stack.
+**Do not apply this from the repo.** It is operator configuration on the NAS, recorded here so the
+exact values are reviewable. Everything below was **measured on 192.168.2.30 on 2026-10-07**, after
+the move.
 
-Current values (`zfs get sharenfs Storage`):
+### The exports
 
 ```
-Storage/plex       rw=@192.168.1.6:@10.123.1.10:@192.168.1.18,no_subtree_check,no_root_squash
-Storage/nextcloud  [email protected]:@192.168.1.5,no_subtree_check,no_root_squash
+/mnt/Storage/plex       192.168.2.10 192.168.2.11 192.168.2.12
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)
+/mnt/Storage/nextcloud  192.168.2.10 192.168.2.11 192.168.2.12
+                        (sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,root_squash,no_all_squash)
 ```
 
-### `Storage/plex` — append the six k3s node addresses to the existing `rw=` list
+Per-node addresses rather than `@192.168.2.0/24`, following the precedent of the old host. A `/24`
+would work and would survive adding a node; it also grants the whole subnet.
+
+**All three nodes must appear on BOTH shares.** Any media pod can be scheduled on any node, so a
+missing node is a latent failure, not an immediate one: it surfaces the first time a pod reschedules
+there, as `access denied by server while mounting`, with the cause weeks stale. One was caught
+exactly this way during setup — the plex share listed `192.168.2.13`, an address no node holds, and
+so was missing `192.168.2.12` (k3s-prod-3). Nothing was failing at the time.
+
+### The option flags ARE the permission contract
+
+| flag | why it matters here |
+|---|---|
+| **`no_all_squash`** | The one the whole model rests on. Client uids pass through untouched, so the arrs' `PUID=1000` arrives as the owner of a `0755 1000:3000` tree. In the UI this is **Mapall User/Group left EMPTY** — setting either squashes every client to one identity and erases the reader/writer distinction entirely. |
+| **`sec=sys`** | AUTH_SYS. Numeric uid/gid, not Kerberos principals. §6 is only true because of this. |
+| **`root_squash`** | Maproot left empty. The Proxmox export carried `no_root_squash`; **nothing here needs it** — every write to the tree happens after the LSIO drop to uid 1000, and the images only chown their Longhorn `/config`. This is a deliberate tightening over the old export, not parity. |
+| **`rw` / `ro`** | The archive's read-only posture is **server-enforced**, as it was before. The media tree cannot get the same treatment — see below. |
+| **`secure`** | Reserved source ports required; "Allow non-root mount" off. The kubelet mounts as root from a reserved port and does not need the exception. |
+
+**Plex's read-only posture is still a CLIENT-side guarantee.** Plex is ro and the arrs are rw, but
+all four can land on any node, so the server sees the same three source addresses for all of them
+and cannot distinguish the reader from the writers. `spec.nfs.readOnly: true` on `plex-media` plus
+the volumeMount flag is the whole enforcement. Unchanged by the move; an honest limitation, not an
+oversight. If it ever needs to be server-enforced, the lever is a dedicated node pool for Plex.
+
+### Service settings that are NOT obvious from the export list
+
+- **Enabled Protocols must include NFSv4** (above).
+- **Require Kerberos for NFSv4: off.** The numeric AUTH_SYS pass-through is the model.
+- **Manage Groups Server-side: off.** This is `rpc.mountd --manage-gids`: the server *discards the
+  client's supplied group list* and looks that uid's groups up in TrueNAS's own user database
+  instead. With no local uid 1000 defined, the arrs' `PGID=3000` would simply vanish. Writes would
+  survive anyway — uid 1000 **owns** the tree, so the owner bits grant access without the group —
+  but it is a free way to lose group semantics.
+- **NFSv4 DNS Domain: blank.** It only feeds `idmapd.conf`, which numeric mapping bypasses.
+- **Bind IP 192.168.2.30** means loopback does not answer: check with `showmount -e 192.168.2.30`,
+  not `localhost`.
+
+There is **no** "NFSv3 ownership model for NFSv4" control on current TrueNAS — it was an older
+TrueNAS/CORE option, and no replacement is needed: on a Linux client against a Linux server over
+AUTH_SYS, both the `nfs` and `nfsd` modules default to `nfs4_disable_idmapping=Y`, so NFSv4 carries
+raw numeric ids rather than `user@domain` principals. §9's `ls -n` probe is how you confirm it.
+
+### The ZFS `sharenfs` properties must be CLEARED, not edited
+
+The pools arrived with the old host's exports still set as dataset properties:
+
+```
+Storage/plex              [email protected]:@10.123.1.10:@192.168.1.18:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
+Storage/nextcloud         [email protected]:@192.168.1.5,[email protected]:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
+Storage/docker-volumes    [email protected]:@192.168.1.5:@192.168.1.10:@192.168.1.18,no_subtree_check,no_root_squash
+```
+
+TrueNAS generates `/etc/exports` from its own middleware database and will fight a property it did
+not set — the failure mode is two differently-configured exports of one path, with no indication
+which is in force. Cleared 2026-10-07:
 
 ```sh
-zfs set sharenfs='[email protected]:@10.123.1.10:@192.168.1.18:@192.168.1.40:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/plex
+zfs inherit sharenfs Storage/plex
+zfs inherit sharenfs Storage/nextcloud
+zfs inherit sharenfs Storage/docker-volumes
 ```
 
-Every pre-existing host is preserved verbatim and no option changes; only clients are added.
+`Storage` is `off`, so each resolves to `off`.
 
-### `Storage/nextcloud` — add the k3s nodes to a **`ro=`** list, not to `rw=`
+### Do NOT touch the dataset's permissions or ACLs
 
-```sh
-zfs set sharenfs='[email protected]:@192.168.1.5,[email protected]:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/nextcloud
-```
+The tree's ownership **is** the access story (§6), and TrueNAS offers two ways to destroy it
+irreversibly across 8.33 T.
 
-This is the one place read-only can be enforced **server-side**, and it is worth taking: only
-Nextcloud consumes this dataset from k3s, so the export itself can refuse writes. Even a
-misconfigured client cannot then damage the archive.
+`Storage/plex` carries `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`) and
+`aclinherit=passthrough` (inherited). Measured 2026-10-07: the directories carry **trivial ACLs
+only** — `nfs4xdr_getfacl` shows just `owner@`/`group@`/`everyone@` matching the mode — so `0755
+1000:3000` governs exactly as it did under Proxmox.
 
-**The media tree cannot get the same treatment.** Plex is read-only and the arrs are read-write, but
-all three may be scheduled on any node, so the server sees the same source addresses for all of
-them and cannot distinguish the reader from the writers. **Plex's read-only posture is therefore a
-client-side guarantee** (`spec.nfs.readOnly: true` on `plex-media`, plus the volumeMount flag) — an
-honest limitation, not an oversight. If it ever needs to be server-enforced, the lever is a
-dedicated node pool for Plex, which is a much larger change.
+- **Stay out of the dataset's Edit Permissions / Edit ACL screen.** On an `acltype=nfsv4` dataset
+  that is a full ACL editor, and its presets write *real* NFSv4 ACLs, recursively if permitted.
+  There is no undo.
+- `aclinherit=passthrough` means a single ACL written at `/mnt/Storage/plex` would be **inherited by
+  everything created under it afterwards** — the blast radius is the whole future library, not one
+  directory.
+- Note GNU `ls`'s `+` marker is **not** a reliable ACL indicator here: it looks for a POSIX ACL
+  xattr, and this dataset stores NFSv4 ACLs elsewhere. Use `nfs4xdr_getfacl`.
 
-Per-node addresses rather than `@192.168.2.0/24` follows the precedent already on this host
-(`/etc/exports` lists the dev nodes individually, as `/32`). A `/24` would work and would survive
-adding a node; it also grants the whole subnet. Per-node is the tighter default.
+`xattr=on` rather than `sa` is a performance detail, not a permissions one, and changing it would
+not affect existing files.
 
 ### Verifying, without changing anything
 
 ```sh
-exportfs -v                  # the new client lines appear, existing ones unchanged
-zfs get sharenfs Storage/plex Storage/nextcloud
+cat /proc/fs/nfsd/versions          # +4.1 must be present
+exportfs -v                         # the two shares, three nodes each, flags as tabulated above
+showmount -e 192.168.2.30           # NOT localhost -- the service is bound to one address
+zfs get -r -o name,property,value,source sharenfs Storage    # every row off or -
 ```
 
----
-
 ## 6. uid / gid — the failure this design exists to avoid
 
-The export uses `sec=sys` and `no_root_squash`, so **the client's numeric uid is passed through
-verbatim** and the server's on-disk ownership is the entire access story. The media tree is
+The export uses `sec=sys` and `no_all_squash`, so **the client's numeric uid is passed through
+verbatim** and the server's on-disk ownership is the entire access story. (It no longer carries
+`no_root_squash`, which the Proxmox export did — see §5. Nothing here needs it, and the uid that
+matters is 1000, not 0.) The media tree is
 `0755 1000:3000` throughout (some directories `1000:1000`), so **only uid 1000 can write it**.
 
 **`fsGroup` cannot fix this, and reaching for it is how this gets misdiagnosed.** The in-tree NFS
@@ -233,24 +278,27 @@ Nextcloud needs no uid work at all, which is one more reason its mount is read-o
 
 ---
 
-## 7. The prod placeholder does NOT fail at apply time — read this before merging
+## 7. Why this knob is unvalidated at create AND immutable at update
+
+**The `TBD-PROD-BULK-NFS-SERVER` placeholder is gone** (2026-10-07) — prod now carries a real
+address, because the NAS is on prod's own subnet and there is no longer a decision to defer. The
+hazard that made it load-bearing is worth keeping on the page, because it is the half that explains
+why §10 exists.
 
 Every other `TBD-PROD-*` in `clusters/prod/kustomization.yaml` is rejected loudly by CRD validation
-(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one is not.**
+(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one was not.**
 
-A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepts
-`TBD-PROD-BULK-NFS-SERVER`, the PV is created, and the PVC **binds** to it — binding is pure
-control-plane and never touches the server. The failure appears only when a pod mounts:
+A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepted
+`TBD-PROD-BULK-NFS-SERVER`, the PV was created, and the PVC **bound** to it — binding is pure
+control-plane and never touches the server. The failure appeared only when a pod mounted:
 
 ```
 MountVolume.SetUp failed ... Failed to resolve server TBD-PROD-BULK-NFS-SERVER
 ```
 
-so **plex, sonarr, radarr and nextcloud would sit in ContainerCreating on prod.**
-
-**Consequently: set `BULK_NFS_SERVER` in `clusters/prod/kustomization.yaml` BEFORE prod reconciles a
-revision that carries these bulk volumes.** Dev renders a working value and is where these manifests
-are meant to be proven first.
+**The same field is immutable on UPDATE** (§10), which is the other half of the same awkwardness: a
+wrong value is accepted silently at create, and then cannot be corrected in place. Together they are
+why repointing this knob is a delete-and-recreate rather than a reconcile.
 
 ### The static guard this convention leans on is missing from `main` — but the gate now refuses the value anyway
 
@@ -287,10 +335,9 @@ Without the arm, HALF A is what holds the line, by design.
    EPHEMERAL layer"). Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it
    means those apps start from an empty config volume, so it belongs in its own change with the
    operator present. **This change does not make it worse** — that is why the media mount is `/media`.
-2. **qbittorrent has no bulk mount.** Today the arrs' import source is written by the **legacy
-   non-k3s** download client, which is coherent while the legacy stack stays live. Giving the k3s
-   qbittorrent the same export at the same `/media` path is the natural follow-up, and the paths were
-   chosen so it needs no Remote Path Mapping.
+2. **RESOLVED — qbittorrent now has the bulk mount** (`subPath: downloads` at `/downloads`), so the
+   import source is written by the k3s download client rather than the legacy non-k3s one. See
+   `catalog/qbittorrent.yaml`.
 3. **`/mnt/Storage/plex/downloads/downloads` is a symlink to `/shared/downloads`**, which is outside
    the export. Over NFS it is a dangling symlink on every client. Harmless, but expect Sonarr to log
    it while scanning `downloads/`.
@@ -300,10 +347,10 @@ Without the arm, HALF A is what holds the line, by design.
 5. **`nconnect` was not set.** It would likely help throughput on the media tree, but it is a tuning
    change that should be measured, not guessed, and it changes client behaviour under server
    failure.
-6. **Nothing was verified against a live cluster.** No `kubectl` context is usable from this
-   workstation (browser-OIDC only), and this change is manifest-authoring under an explicit
-   do-not-apply constraint. Every claim about the *host* is measured; every claim about *binding and
-   mounting* is reasoned from the manifests.
+6. **Every claim about the NAS in §5 is measured (2026-10-07); the post-move MOUNT is not.** The PVs
+   still have to be recreated (§10) before any pod mounts 192.168.2.30, so the §9 probes are the
+   outstanding verification, not a formality. The pre-move equivalents did pass against the old host
+   on 2026-09-12.
 7. **The bulk-tier requirement ID is deliberately not typed on any non-`.md` line of this change.**
    `ci/touched-requirements.sh` attributes requirements from R-IDs appearing in added lines, and the
    in-cluster conformance test for the bulk tier is still a gap, so typing it would select a
@@ -312,36 +359,95 @@ Without the arm, HALF A is what holds the line, by design.
 
 ---
 
-## 9. Validating on dev, once the export is extended
+## 9. Validating, after the PVs are recreated
 
-Dev can prove all of this before prod's network path exists. Read-only checks first:
+These run against **prod**; dev is decommissioned. They are the outstanding verification for the
+move — nothing below has been run against 192.168.2.30. Read-only checks first:
 
 ```sh
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pv
-infra/devcontainer/dev.sh run kubectl --context k3s-dev get pvc -A | grep -E 'media|legacy-archive'
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr get pod -o wide
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pv -o custom-columns=\
+NAME:.metadata.name,SERVER:.spec.nfs.server,PATH:.spec.nfs.path,STATUS:.status.phase'
 ```
 
-Expect **four** PVs — `plex-media`, `sonarr-media`, `radarr-media`, `nextcloud-legacy-archive` (three
-of them naming the *same* export, one per namespace, which is the cross-namespace model) — each
-`Bound` to the same-named PVC, with `RECLAIM POLICY Retain` and an empty `STORAGECLASS`.
+Expect **five** NFS PVs — `plex-media`, `sonarr-media`, `radarr-media`, `qbittorrent-downloads`,
+`nextcloud-legacy-archive` (four of them naming the *same* export, one per namespace, which is the
+cross-namespace model) — each `Bound`, `RECLAIM POLICY Retain`, empty `STORAGECLASS`, and
+**`SERVER 192.168.2.30`**. A PV still showing `192.168.2.102` means the recreate in §10 did not
+happen; Flux will NOT have updated it in place.
 
-Then the two things that actually matter, from inside a pod:
+Then the four things that actually matter, from inside a pod:
 
 ```sh
-# (a) the writer can write — this is the uid check, and it is the one that fails in practice
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
-  sh -c 'id; touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+# (a) numeric ids survived the wire -- the idmapping check, new since the TrueNAS move
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- sh -c 'ls -n /media | head -3'
+#     expect uid 1000 / gid 3000 on the entries. 65534 / nobody means ids are being mapped, not
+#     passed through, and every import will fail EACCES while the pod still reports Ready.
 
-# (b) hardlinks work BETWEEN the library and the download path — the root-mount check
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
+# (b) the writer can write -- the uid check, and the one that fails in practice
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
+  sh -c 'touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
+
+# (c) hardlinks work BETWEEN the library and the download path -- the root-mount check
+kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
   sh -c 'set -e; d=/media/downloads/complete/tv-sonarr/.probe; t=/media/tv/.probe-link;
          echo x > "$d"; ln "$d" "$t"; ls -i "$d" "$t"; rm -f "$d" "$t"; echo HARDLINK-OK'
 
-# (c) the reader really is read-only
-infra/devcontainer/dev.sh run kubectl --context k3s-dev -n plex exec deploy/plex -- \
-  sh -c 'touch /media/.probe 2>&1 | head -1'      # expect "Read-only file system"
+# (d) the reader really is read-only
+kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c 'touch /media/.probe 2>&1 | head -1'
+#     expect "Read-only file system"
 ```
 
-`(b)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
+`(c)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
 prints `EXDEV`, something has split the single mount back into two.
+
+Reading the failures together is what localises the fault: **(b) EACCES with (a) showing `nobody`**
+is idmapping (§5 service settings). **(b) EACCES with (a) showing `1000`** is on-disk ownership —
+stop and check `ls -ldn` on the NAS before changing anything else.
+
+---
+
+## 10. Repointing BULK_NFS_SERVER means DELETING AND RECREATING THE PVs
+
+**A bound PersistentVolume's NFS source is immutable.** Measured by server-side dry-run on
+2026-10-07:
+
+```
+$ kubectl patch pv sonarr-media --type=json --dry-run=server \
+    -p '[{"op":"replace","path":"/spec/nfs/server","value":"192.168.2.30"}]'
+The PersistentVolume "sonarr-media" is invalid: spec.persistentvolumesource: Forbidden:
+  spec.persistentvolumesource is immutable after creation
+```
+
+Note the pairing with §7: `spec.nfs.server` is **unvalidated at create** (any string is accepted and
+the PVC binds) and **immutable at update**. So editing `BULK_NFS_SERVER` alone does not fail loudly
+and does not work either — Flux's apply errors on the five PVs and the pods keep their previous,
+now-dead mount.
+
+This is safe to do because `persistentVolumeReclaimPolicy: Retain` means deleting a PV never touches
+the data, and `catalog/render.py` emits a **pre-set `claimRef`** on each one, so a recreated PV
+rebinds its own PVC with no manual step.
+
+**Operator sequence** (mutating `kubectl` is an operator action, not an agent action):
+
+```sh
+# 1. Stop the consumers. This also releases the hung hard-mounts against the old server:
+#    the pods report Ready, but any real I/O on /media blocks indefinitely once 2049 stops
+#    answering, because the PVs mount `hard`.
+for ns in plex sonarr radarr qbittorrent nextcloud; do
+  kubectl --context k3s-prod -n $ns scale deploy --all --replicas=0
+done
+
+# 2. Delete the five PVs. Retain + no live claim means nothing is reclaimed or erased.
+kubectl --context k3s-prod delete pv plex-media sonarr-media radarr-media \
+  qbittorrent-downloads nextcloud-legacy-archive
+
+# 3. Let Flux reconcile the revision carrying BULK_NFS_SERVER=192.168.2.30, which recreates them.
+#    Confirm .status.lastAppliedRevision matches the GitRepository artifact before continuing.
+
+# 4. Bring the consumers back, then run every probe in §9.
+```
+
+The PVCs pass through `Lost` between steps 2 and 3; that is expected and resolves when the PV with
+the matching `claimRef` reappears. If a PVC is still `Lost` after Flux reconciles, the PV name or
+its `claimRef` namespace does not match — do **not** delete the PVC to "fix" it.
+

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -420,3 +420,22 @@ flux_bootstrap_env_path: "clusters/prod"
 # rather than leaving an inert drop-in behind, so this group-wide `true` is still correct
 # for it.
 etcd_io_priority_enabled: true
+
+# --- The node terminating remote-access WireGuard — DECIDED (measured 2026-10-08) ---------------
+# k3s-prod-1 (192.168.2.10) is what the SOHO router forwards WAN udp/51820 to, confirmed from the
+# cluster rather than assumed: only prod-1's `wg show` had a peer with an `endpoint` and a recent
+# handshake, the other two listeners were idle.
+#
+# node_baseline labels it `homelab.keithjackson.dev/vpn-ingress=true`, and
+# platform-config/cilium/l2-announcement-policy.yaml excludes that label from announcing any
+# LoadBalancer VIP -- because a node CANNOT reach a VIP whose Cilium L2 announcement lease it holds
+# itself (the announcer does not answer its own ARP request; the neighbour entry goes to `FAILED`).
+# prod-1 held the `shared` Gateway lease, and since vpn-ingress MASQUERADEs clients to the node's
+# own address, all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved.
+#
+# [parity] MUST equal the node named by the router's forward (runbooks/vpn-remote-access.md step 3).
+# Nothing in CI can read that forward, so re-pointing it means changing BOTH: this value plus a
+# node_baseline re-run, and the router line. `VpnIngressNodeAnnouncesL2VIP` catches the case where
+# the label lands on a node that then takes a lease; it CANNOT catch the forward being moved
+# without this value following, which is why the runbook spells the two-step out.
+node_baseline_vpn_ingress_node: k3s-prod-1

infra/ansible/roles/node_baseline/defaults/main.yml

@@ -259,3 +259,30 @@ node_baseline_dns_options:
 # Written as ONE marked block, so removing an entry here removes it from the node, and
 # an empty list removes the block entirely.
 node_baseline_hosts_pins: []
+
+# --- The node that terminates remote-access WireGuard (tags: vpn-ingress) ----------------------
+# Hostname of the node the SOHO router forwards WAN udp/51820 to, i.e. the one platform/vpn-ingress
+# actually serves remote clients from. Set it in group_vars for the environment; EMPTY by default,
+# because dev has no such forward and because a wrong value is worse than none.
+#
+# WHAT IT IS FOR, AND WHY IT IS NOT COSMETIC. The matching node gets
+# `homelab.keithjackson.dev/vpn-ingress=true`, and
+# platform-config/cilium/l2-announcement-policy.yaml EXCLUDES that label from announcing any
+# LoadBalancer VIP. MEASURED on prod 2026-10-08: a node cannot reach a VIP whose Cilium L2
+# announcement lease it holds itself (the announcer does not answer its own ARP request -- the
+# neighbour entry goes to `FAILED`). Because vpn-ingress runs on `hostNetwork` and MASQUERADEs
+# clients to the node's own address, a VPN client inherits that: with the forward on the node
+# holding the `shared` Gateway lease, all 16 Gateway hostnames were dead over the VPN while PiHole
+# DNS still resolved. Names worked, every app hung, nothing alerted.
+#
+# [parity] MUST name the node the router's forward actually points at. The forward is off-repo and
+# no check can read it (runbooks/vpn-remote-access.md step 3). Changing the forward is therefore a
+# two-part change: this value (then re-run the role), and the router line.
+#
+# Only ONE node should ever carry the label: it removes that node from VIP announcement candidacy,
+# so labelling all three would leave nothing announcing any VIP at all. The task below removes the
+# label from a node that no longer matches, so moving the value moves the label.
+node_baseline_vpn_ingress_node: ""
+node_baseline_vpn_ingress_label_key: homelab.keithjackson.dev/vpn-ingress
+# k3s ships its kubeconfig here; every prod node is a k3s server, so `k3s kubectl` works locally.
+node_baseline_vpn_ingress_kubeconfig: /etc/rancher/k3s/k3s.yaml

infra/ansible/roles/node_baseline/tasks/main.yml

@@ -1109,3 +1109,45 @@
     state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
     backup: true
   tags: [dns, hosts]
+
+# ── vpn-ingress node label — tag `vpn-ingress` ────────────────────────────────────────────────
+# Marks the node terminating remote-access WireGuard so Cilium's L2 announcement policy can EXCLUDE
+# it from announcing LoadBalancer VIPs (a node cannot reach a VIP it announces itself -- see
+# defaults/main.yml). Same shape as roles/vpn_egress_gateway's egress-gateway label, and for the
+# same reason: the label is the one place BOTH a nodeSelector and a PromQL alert can read it from.
+#
+# Skipped entirely when node_baseline_vpn_ingress_node is empty (dev, or before the forward exists),
+# so this adds nothing to an environment that has no WAN forward.
+- name: Label this node as the vpn-ingress node
+  ansible.builtin.command:
+    cmd: >-
+      k3s kubectl label node {{ inventory_hostname }}
+      {{ node_baseline_vpn_ingress_label_key }}=true --overwrite
+  environment:
+    KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}"
+  register: node_baseline_vpn_ingress_label_apply
+  changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_apply.stdout"
+  when:
+    - node_baseline_vpn_ingress_node | length > 0
+    - inventory_hostname == node_baseline_vpn_ingress_node
+  tags: [vpn-ingress]
+
+# The other half: moving the knob must MOVE the label, not add a second one. Two labelled nodes
+# would remove two of three nodes from announcement candidacy; three would leave no announcer at
+# all and take every VIP off the LAN.
+- name: Remove the vpn-ingress label from a node that no longer holds the forward
+  ansible.builtin.command:
+    cmd: >-
+      k3s kubectl label node {{ inventory_hostname }}
+      {{ node_baseline_vpn_ingress_label_key }}-
+  environment:
+    KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}"
+  register: node_baseline_vpn_ingress_label_remove
+  changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_remove.stdout"
+  failed_when:
+    - node_baseline_vpn_ingress_label_remove.rc != 0
+    - "'not found' not in node_baseline_vpn_ingress_label_remove.stderr"
+  when:
+    - node_baseline_vpn_ingress_node | length > 0
+    - inventory_hostname != node_baseline_vpn_ingress_node
+  tags: [vpn-ingress]

platform-config/cilium/l2-announcement-policy.yaml

@@ -16,6 +16,71 @@ spec:
   externalIPs: false
   interfaces:
     - ^e(n|th)
+  # ===============================================================================================
+  # WHY THIS IS A matchExpressions LIST AND NOT `matchLabels: {kubernetes.io/os: linux}`
+  # ===============================================================================================
+  # The second expression excludes every node labelled `homelab.keithjackson.dev/vpn-ingress=true`
+  # from announcing ANY LoadBalancer VIP. That is the node the SOHO router forwards WAN udp/51820 to.
+  # It is not tuning; it is the fix for a measured, silent, total failure of remote access.
+  #
+  # MEASURED ON PROD 2026-10-08: A NODE CANNOT REACH A VIP IT ANNOUNCES ITSELF. Cilium's L2
+  # announcer answers ARP for the VIP on behalf of OTHER LAN hosts, but not for the announcing node
+  # itself. On the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP
+  # is dropped before it leaves the node:
+  #
+  #   k3s-prod-1, holding cilium-l2announce-gateway-cilium-gateway-shared:
+  #     ip neigh show 192.168.2.20  ->  192.168.2.20 dev eno1 FAILED       (its own VIP)
+  #     nc -z 192.168.2.20 443      ->  FAIL          nc -z 192.168.2.20 80 -> FAIL
+  #   k3s-prod-2 and k3s-prod-3, holding no lease for it:
+  #     ip neigh show 192.168.2.20  ->  lladdr bc:e9:2f:7e:c4:a6 (prod-1)   nc 443/80 -> OK
+  #
+  # Symmetric across all 3 nodes x 5 VIPs: every node reached every VIP EXCEPT the ones it alone
+  # announced (prod-2 could not reach crafty 192.168.2.22 or unifi 192.168.2.21; prod-3 could not
+  # reach plex 192.168.2.24). So this is a property of L2 announcement, not of one Service.
+  #
+  # WHY IT ONLY HURTS THE VPN NODE. For an ordinary workload this is harmless: pods reach Services
+  # through the cluster datapath and never ARP for a VIP. platform/vpn-ingress is the exception --
+  # it runs WireGuard with `hostNetwork: true` and MASQUERADEs remote clients to the node's own
+  # address, so a VPN client's packets take the node's LAN path and inherit exactly this ARP
+  # failure. On 2026-10-08 the forward pointed at k3s-prod-1, which held the Gateway lease, and all
+  # 16 Gateway hostnames were unreachable over the VPN while PiHole DNS still answered -- so names
+  # resolved and every app hung. Nothing in the cluster reported it; the alerts below now do.
+  #
+  # WHY A NODE LABEL AND NOT A cluster-vars KNOB. A knob would put the node's identity in git, which
+  # is more reviewable -- but it CANNOT BE USED BY THE ALERT that guards this, and an unguarded
+  # invariant here is worth less than a reviewable one. The alert harness
+  # (platform/monitoring/tests/extract_rules.py) REFUSES any alert whose `expr` contains a
+  # dollar-brace postBuild placeholder (spelled without braces here ON PURPOSE: this file is itself
+  # envsubst'd, and a literal dollar-brace-dot-dot-dot is UNPARSEABLE to envsubst, which fails the
+  # WHOLE Kustomization -- ci: flux-postbuild-substitution caught exactly this comment), because the
+  # rule it would unit-test is not the rule that gets
+  # deployed. A label is readable from BOTH sides: this selector matches on it, and
+  # `kube_node_labels` carries it into Prometheus so `VpnIngressNodeAnnouncesL2VIP` can join lease
+  # holders against it. One source of truth, no git-vs-cluster skew, and the policy stays
+  # env-agnostic as its header above claims. The label is set by
+  # infra/ansible/roles/node_baseline (node_baseline_vpn_ingress_node), the same shape
+  # roles/vpn_egress_gateway already uses for `egress-gateway`.
+  #
+  # THE COST, STATED: VIP announcement candidacy drops from 3 nodes to 2. That is still redundant,
+  # and the trade is deliberate -- the excluded node is the one whose *outbound* reach to every VIP
+  # matters most, because it carries every remote client. If BOTH remaining nodes are down the VIPs
+  # go dark where previously one node might have served them, but at that point the cluster has one
+  # node left and far worse problems.
+  #
+  # `NotIn` with an ABSENT key MATCHES (k8s apimachinery: a missing label is "not in the set"), so
+  # the other two nodes need no label and nothing has to be applied to them. If the label is never
+  # applied anywhere, this excludes nothing and the policy degrades to its pre-2026-10-08 behaviour
+  # -- the status quo, not an outage: it fails back, not down. That silent-degradation path is
+  # exactly why `VpnIngressNodeLabelMissing` exists as a watchdog: without it, the alert below
+  # would be structurally INERT (its join would match nothing) and would look identical to healthy.
   nodeSelector:
-    matchLabels:
-      kubernetes.io/os: linux
+    matchExpressions:
+      - key: kubernetes.io/os
+        operator: In
+        values:
+          - linux
+      # The node terminating remote-access WireGuard must announce nothing -- see above.
+      - key: homelab.keithjackson.dev/vpn-ingress
+        operator: NotIn
+        values:
+          - "true"

platform/monitoring/alerts.yaml

@@ -1147,6 +1147,97 @@ spec:
               metricLabelsAllowlist.
             runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
         # -----------------------------------------------------------------------------------------
+        # REMOTE-ACCESS VPN (platform/vpn-ingress) -- NOT the egress tunnel above. Different
+        # direction, different workload: this is inbound WireGuard for remote devices.
+        # -----------------------------------------------------------------------------------------
+        - alert: VpnIngressNodeAnnouncesL2VIP
+          # THE FAILURE THIS CATCHES. MEASURED on prod 2026-10-08: a node CANNOT reach a
+          # LoadBalancer VIP whose Cilium L2 announcement lease it holds itself -- the announcer
+          # does not answer its own ARP request, so the neighbour entry settles at `FAILED` and
+          # packets die before leaving the node (`ip neigh show 192.168.2.20` -> FAILED on the
+          # lease holder, resolved on both others). platform/vpn-ingress runs WireGuard on
+          # `hostNetwork` and MASQUERADEs remote clients to the node's own address, so if the node
+          # the router forwards udp/51820 to is also a lease holder, VPN clients cannot reach the
+          # VIPs it announces. On 2026-10-08 that was the `shared` Gateway: all 16 app hostnames
+          # dead over the VPN, while PiHole DNS kept answering (its VIP has two Services and the
+          # other lease sat on prod-2) -- so the tunnel handshook, names resolved, and every app
+          # hung. Nothing reported it for ~2 days.
+          #
+          # platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled
+          # `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so this should be
+          # permanently unsatisfiable. The alert exists because the exclusion has ways to stop
+          # working that no static check on that file's TEXT can see: the label being removed or
+          # moved to the wrong node, a Cilium upgrade changing `NotIn` semantics or the
+          # nodeSelector field, and the selector silently failing to match. This watches the
+          # OUTCOME -- who actually holds the lease -- so all of them surface the same way.
+          #
+          # WHY THE LABEL JOIN RATHER THAN A NODE NAME. The node's identity is NOT written into
+          # this expr, because the harness (tests/extract_rules.py) refuses any alert whose expr
+          # contains a dollar-brace placeholder -- spelled in words because a literal one here is
+          # unparseable to envsubst and would fail this whole Kustomization. Rightly so: the rule it
+          # unit-tests would not be the
+          # rule deployed. Joining `kube_lease_owner` against `kube_node_labels` keeps ONE source
+          # of truth (the label itself, which is also what the policy selects on), so this alert
+          # cannot drift out of agreement with the exclusion it guards. `label_replace` is needed
+          # because kube_lease_owner names the holder `lease_holder` while kube_node_labels names
+          # it `node`; the join is then on a shared `node` label. The label is in
+          # kube-state-metrics' metricLabelsAllowlist (kube-prometheus-stack.yaml) -- without that
+          # line this rule is inert, which `VpnIngressNodeLabelMissing` below exists to catch.
+          #
+          # `for: 15m` because leases legitimately move during a node reboot or a Cilium roll and
+          # Cilium re-elects within seconds; a real collision persists, since the lease holder
+          # keeps renewing and it never self-heals.
+          expr: |
+            max by (lease) (
+              label_replace(
+                kube_lease_owner{namespace="kube-system", lease=~"cilium-l2announce-.*"},
+                "node", "$1", "lease_holder", "(.+)"
+              )
+              * on (node) group_left ()
+              kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"}
+            ) == 1
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "VPN ingress node is announcing a LoadBalancer VIP ({{ $labels.lease }})"
+            description: >-
+              The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access
+              WireGuard, and it now holds the Cilium L2 announcement lease `{{ $labels.lease }}`. A
+              node cannot reach a VIP it announces itself, so every remote VPN client has just lost
+              all access to that VIP's Service -- silently, with DNS still resolving. Check that
+              platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that
+              the label is still on the node the router forwards udp/51820 to.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
+        - alert: VpnIngressNodeLabelMissing
+          # THE WATCHDOG for the rule above, and the reason it is not optional. The exclusion in
+          # l2-announcement-policy.yaml is a `NotIn` on this label, and `NotIn` with an ABSENT key
+          # MATCHES -- so if the label is on NO node, the policy excludes nothing, every node is an
+          # announcement candidate again, and the 2026-10-08 failure is fully re-armed. The alert
+          # above would not fire either: its join would match nothing and return no series. Present,
+          # evaluating, and incapable of firing -- the exact "inert critical alert" shape that
+          # VPNEgressUplinkLabelMissing exists for on the egress side, and that this repo has been
+          # burned by before.
+          #
+          # The label is applied by infra/ansible/roles/node_baseline
+          # (node_baseline_vpn_ingress_node); the most likely cause of it going absent is a node
+          # rebuild where that role has not been re-run.
+          expr: absent(kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"})
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF"
+            description: >-
+              No node publishes homelab.keithjackson.dev/vpn-ingress=true, so
+              l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none:
+              any node may now announce a LoadBalancer VIP, including the one terminating
+              remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so
+              the warning matters more than its severity suggests. Re-run the node_baseline role on
+              the forwarded node, and confirm the label is in kube-state-metrics'
+              metricLabelsAllowlist.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
+        # -----------------------------------------------------------------------------------------
         # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
         # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
         # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`

platform/monitoring/kube-prometheus-stack.yaml

@@ -281,8 +281,20 @@ spec:
       # without anyone having to remember to re-enable it later. Scoped to one label on one resource:
       # 3 extra series, no cardinality concern. Label name is sanitized by KSM to
       # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
+      # `vpn-ingress` (added 2026-10-17) marks the node the SOHO router forwards WAN udp/51820 to.
+      # platform-config/cilium/l2-announcement-policy.yaml excludes that node from announcing any
+      # LoadBalancer VIP, because a node cannot reach a VIP whose L2 announcement lease it holds
+      # itself and vpn-ingress MASQUERADEs remote clients to the node's own address (measured
+      # 2026-10-08: all 16 Gateway hostnames dead over the VPN while DNS still resolved). It is
+      # allowlisted HERE because that is the only way the invariant can be WATCHED:
+      # `VpnIngressNodeAnnouncesL2VIP` joins `kube_lease_owner` against this label, and the alert
+      # unit-test harness refuses a dollar-brace placeholder in an alert expr (written out in words:
+      # a literal one here is unparseable to envsubst and fails the whole Kustomization), so a knob
+      # could not have been used. Without this line BOTH that alert and its
+      # `VpnIngressNodeLabelMissing` watchdog are structurally inert. Same reasoning, and the same
+      # 3-extra-series cost, as the two egress labels below.
       metricLabelsAllowlist:
-        - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
+        - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink,homelab.keithjackson.dev/vpn-ingress]"
       # --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
       # Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
       # `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that

platform/monitoring/node-pressure-dashboard.yaml

@@ -199,7 +199,7 @@ data:
           "id": 15,
           "type": "timeseries",
           "title": "CAUSE 4 — NFS RPC retransmission ratio, per node",
-          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.102, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
+          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.30, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
           "gridPos": {"h": 8, "w": 12, "x": 12, "y": 39},
           "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
             {"color": "green", "value": null}, {"color": "red", "value": 0.01}

platform/monitoring/tests/vpn-ingress-l2.test.yaml

@@ -0,0 +1,203 @@
+# promtool rule-unit tests for VpnIngressNodeAnnouncesL2VIP + VpnIngressNodeLabelMissing
+# (platform/monitoring/alerts.yaml)
+#
+# Run with: platform/monitoring/tests/run.sh
+#           (the merge gate runs the same harness via
+#            conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS
+# MEASURED on prod 2026-10-08: a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
+# lease it holds ITSELF -- the announcer does not answer its own ARP request, so on the lease holder
+# the neighbour entry settles at `FAILED` and packets to that VIP die before leaving the node.
+# platform/vpn-ingress runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
+# own address, so a VPN client takes that node's LAN path and inherits the failure. With the
+# router's udp/51820 forward on k3s-prod-1, which held the `shared` Gateway lease, all 16 Gateway
+# hostnames were dead over the VPN -- while PiHole DNS kept answering, because its VIP has two
+# Services and k3s-prod-2 held the other lease. The tunnel handshook, names resolved, every app
+# hung, and nothing reported it for ~2 days.
+#
+# platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled
+# `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so VpnIngressNodeAnnouncesL2VIP
+# should be permanently unsatisfiable. THAT IS EXACTLY WHY IT NEEDS A TEST: an alert expected never
+# to fire is indistinguishable, in production, from an alert that CANNOT fire. Nothing would ever
+# reveal a mistyped label, a broken `label_replace`, or a join on the wrong label. Test 1 proves it
+# fires on the real failure; test 2 proves it is quiet in the fixed state, so it is not a rule that
+# fires on anything; test 3 pins the `for: 15m` against routine lease churn.
+#
+# THE JOIN IS THE FRAGILE PART AND IS TESTED DELIBERATELY. `kube_lease_owner` names the holder in
+# `lease_holder`; `kube_node_labels` names it `node`. The rule bridges them with `label_replace`
+# and joins `on (node) group_left ()`. Fixtures therefore carry BOTH series shapes with real label
+# sets, because a join that silently matches nothing returns no series -- which looks exactly like
+# "healthy" and is the failure mode this whole group exists to prevent.
+#
+# Fixture values are prod's real shape (measured 2026-10-08): leases
+# `cilium-l2announce-gateway-cilium-gateway-shared` (k3s-prod-1) and
+# `cilium-l2announce-pihole-pihole-dns-tcp` (k3s-prod-2), in namespace kube-system, with
+# k3s-prod-1 the vpn-ingress node.
+
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. THE BUG, EXACTLY AS IT HAPPENED -- the vpn-ingress node holds the Gateway lease: FIRES.
+  # =============================================================================================
+  - interval: 1m
+    name: vpn-ingress node holds the shared Gateway L2 lease - VpnIngressNodeAnnouncesL2VIP fires
+    input_series:
+      # The collision. A lease holder keeps renewing, so this series does not lapse on its own --
+      # which is why the condition never self-heals and why a 15m `for:` is safe.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
+        values: '1+0x40'
+      # A second lease held by a DIFFERENT node, present throughout. This is the discriminator that
+      # stops the rule from being "any l2announce lease exists anywhere": if the join were dropped,
+      # this series alone would fire the alert, and test 2 would fail.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-pihole-pihole-dns-tcp",lease_holder="k3s-prod-2"}'
+        values: '1+0x40'
+      # k3s-prod-1 is the vpn-ingress node. Only this node carries the label.
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
+        values: '1+0x40'
+      # The other two nodes publish kube_node_labels WITHOUT the vpn-ingress label. Present so the
+      # join is exercised against non-matching nodes rather than against nothing at all.
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
+        values: '1+0x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
+        values: '1+0x40'
+    alert_rule_test:
+      # Held 15m, so nothing before then.
+      - eval_time: 14m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+      - eval_time: 20m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts:
+          # EXACTLY ONE alert: the pihole lease on prod-2 must NOT produce a second one.
+          - exp_labels:
+              severity: critical
+              lease: cilium-l2announce-gateway-cilium-gateway-shared
+            exp_annotations:
+              summary: "VPN ingress node is announcing a LoadBalancer VIP (cilium-l2announce-gateway-cilium-gateway-shared)"
+              description: >-
+                The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access
+                WireGuard, and it now holds the Cilium L2 announcement lease `cilium-l2announce-gateway-cilium-gateway-shared`. A
+                node cannot reach a VIP it announces itself, so every remote VPN client has just lost
+                all access to that VIP's Service -- silently, with DNS still resolving. Check that
+                platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that
+                the label is still on the node the router forwards udp/51820 to.
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
+      # The label IS present, so the watchdog must stay quiet here.
+      - eval_time: 40m
+        alertname: VpnIngressNodeLabelMissing
+        exp_alerts: []
+
+  # =============================================================================================
+  # 2. THE FIXED STATE -- every lease is held by a node that is NOT the vpn-ingress node: QUIET.
+  # This is what l2-announcement-policy.yaml's exclusion produces, and the assertion that keeps
+  # test 1 meaningful rather than an alert that fires on anything.
+  # =============================================================================================
+  - interval: 1m
+    name: all l2 leases held by other nodes - VpnIngressNodeAnnouncesL2VIP stays quiet
+    input_series:
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-2"}'
+        values: '1+0x40'
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-plex-plex-l4-tcp",lease_holder="k3s-prod-3"}'
+        values: '1+0x40'
+      # A lease the vpn-ingress node DOES legitimately hold, in a different family -- proves the
+      # `cilium-l2announce-.*` matcher is doing work. Every node holds operator/controller leases;
+      # only the ANNOUNCEMENT ones matter here, and a matcher broad enough to catch this would page
+      # on a healthy cluster forever.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-operator-resource-lock",lease_holder="k3s-prod-1"}'
+        values: '1+0x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
+        values: '1+0x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
+        values: '1+0x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
+        values: '1+0x40'
+    alert_rule_test:
+      - eval_time: 40m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+      - eval_time: 40m
+        alertname: VpnIngressNodeLabelMissing
+        exp_alerts: []
+
+  # =============================================================================================
+  # 3. A LEGITIMATE, TRANSIENT LEASE MOVE -- the vpn-ingress node holds it for 11m during a node
+  # reboot or a Cilium roll, then Cilium re-elects away: QUIET. This pins the `for: 15m` as
+  # deliberate. Without it, every rolling restart of the cilium DaemonSet would page, and a paging
+  # alert that cries wolf on routine maintenance gets muted -- which is how the real collision
+  # would go unseen again.
+  # =============================================================================================
+  - interval: 1m
+    name: transient lease move shorter than for 15m - VpnIngressNodeAnnouncesL2VIP stays quiet
+    input_series:
+      # Held by the vpn-ingress node for 11 samples, then the series goes absent (the lease moved,
+      # so kube-state-metrics retires this label set and starts a new one).
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
+        values: '1+0x10 _x30'
+      # Re-elected onto prod-3 from t=11m. A label VALUE change retires one series and starts
+      # another, which is why this is a second series and not a changing value.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-3"}'
+        values: '_x11 1+0x29'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
+        values: '1+0x40'
+    alert_rule_test:
+      # Inside the transient window: pending at most, never firing.
+      - eval_time: 10m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+      # Well past it, with the lease settled on another node.
+      - eval_time: 30m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. THE WATCHDOG -- no node carries the label at all. The policy's `NotIn` then excludes
+  # NOTHING (a missing key matches), so the 2026-10-08 failure is fully re-armed, AND
+  # VpnIngressNodeAnnouncesL2VIP is structurally INERT: its join matches nothing and returns no
+  # series, which is indistinguishable from healthy. Both halves are asserted here, because an
+  # inert critical alert is the exact shape this repo has been burned by before.
+  # =============================================================================================
+  - interval: 1m
+    name: vpn-ingress label absent everywhere - watchdog fires and the critical alert is inert
+    input_series:
+      # A lease held by the node that IS (in reality) the forwarded one -- but with no label
+      # anywhere, nothing can tell. This is the dangerous state.
+      - series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
+        values: '1+0x60'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}'
+        values: '1+0x60'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
+        values: '1+0x60'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
+        values: '1+0x60'
+    alert_rule_test:
+      # The critical alert CANNOT fire -- this is the inertness being pinned, not an aside.
+      - eval_time: 45m
+        alertname: VpnIngressNodeAnnouncesL2VIP
+        exp_alerts: []
+      # ... which is precisely why the watchdog must.
+      - eval_time: 45m
+        alertname: VpnIngressNodeLabelMissing
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              # absent() propagates the matcher's own labels onto the series it returns, so the
+              # alert inherits this one. Asserted rather than worked around: it is harmless, and
+              # pinning it means a future change to the matcher cannot alter the alert's identity
+              # (and therefore its Alertmanager grouping) unnoticed.
+              label_homelab_keithjackson_dev_vpn_ingress: "true"
+            exp_annotations:
+              summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF"
+              description: >-
+                No node publishes homelab.keithjackson.dev/vpn-ingress=true, so
+                l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none:
+                any node may now announce a LoadBalancer VIP, including the one terminating
+                remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so
+                the warning matters more than its severity suggests. Re-run the node_baseline role on
+                the forwarded node, and confirm the label is in kube-state-metrics'
+                metricLabelsAllowlist.
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"

platform/oauth2-proxy/helmrelease.yaml

@@ -63,6 +63,75 @@ spec:
       # gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
       skip-provider-button: "true"
       scope: "openid email profile"
+    # ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ──────────
+    # MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at
+    # startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so
+    # the Gateway answers 503 and the proxy dies:
+    #
+    #   [provider.go:55] Performing OIDC Discovery...
+    #   [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC
+    #                configuration: unexpected status "503": no healthy upstream
+    #
+    # It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was
+    # still down minutes later and only a manual pod delete cleared it. There is no retry flag; the
+    # process simply does not survive a cold start. A readinessProbe cannot help (the container is
+    # already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+    #
+    # `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in
+    # values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init
+    # container for its built-in waitForRedis gate, so this adds no new base image to the platform.
+    # A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole
+    # point.
+    #
+    # BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst
+    # case is then exactly today's behaviour, never something new. An unbounded wait would trade a
+    # CrashLoop for a pod wedged in Init forever, which is not an improvement.
+    #
+    # SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is
+    # rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which
+    # CONSUMES braced expansions -- so a braced shell variable renders to the empty string and
+    # silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars
+    # knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud
+    # OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.)
+    extraInitContainers:
+      - name: wait-for-idp
+        image: docker.io/library/alpine:3.22
+        imagePullPolicy: IfNotPresent
+        command: ["/bin/sh", "-c"]
+        args:
+          - |
+            set -u
+            url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
+            i=0
+            while [ "$i" -lt 60 ]; do
+              # busybox wget validates against alpine's ca-certificates-bundle, so a real TLS
+              # failure is a non-zero exit here rather than a false pass.
+              if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then
+                echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy"
+                exit 0
+              fi
+              i=$(( i + 1 ))
+              sleep 5
+            done
+            echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2
+            echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2
+            exit 0
+        securityContext:
+          allowPrivilegeEscalation: false
+          runAsNonRoot: true
+          runAsUser: 65534
+          runAsGroup: 65534
+          readOnlyRootFilesystem: true
+          capabilities:
+            drop:
+              - ALL
+        resources:
+          requests:
+            cpu: 10m
+            memory: 16Mi
+          limits:
+            memory: 32Mi
+
     # Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
     metrics:
       enabled: true

platform/vpn-ingress/README.md

@@ -117,8 +117,19 @@ boundary around this port is WireGuard's peer authentication, full stop.
 The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
 forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
 A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
-keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no
-re-issued client profiles. The two idle listeners cost ~15 MiB each.
+keys — so losing the forwarded node never costs a re-issued client profile. The two idle listeners
+cost ~15 MiB each.
+
+> **This section used to say "losing the forwarded node is a one-line router change, with no cluster
+> work". It was wrong, and it cost two days of silently broken remote access on 2026-10-08.** The
+> nodes are warm, but they are **not interchangeable**: the node the forward points at must be the
+> one labelled `homelab.keithjackson.dev/vpn-ingress=true`, because that is the node
+> `platform-config/cilium/l2-announcement-policy.yaml` excludes from announcing LoadBalancer VIPs.
+> **A node cannot reach a VIP whose Cilium L2 announcement lease it holds itself** (see "Measured on
+> prod, 2026-10-08" below), and since this pod MASQUERADEs clients to the node's own address, a VPN
+> client inherits that. Failover is therefore **two** changes: move
+> `node_baseline_vpn_ingress_node` and re-run that role, then re-point the router —
+> `runbooks/vpn-remote-access.md` step 3.
 
 A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
 replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
@@ -186,6 +197,45 @@ right way; one was a defect in my own manifest.
 | Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
 | Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
 
+## Measured on prod, 2026-10-08 — why the forwarded node cannot be any node
+
+The first device enrolled, handshook, resolved names, and could not open a single connection to any
+cluster web app. The tunnel was not at fault — every part of it measured healthy:
+`net.ipv4.ip_forward=1`, `wg0` up at `10.90.0.1/24` MTU 1420, the peer authorised with real
+bidirectional transfer (346 KiB in / 200 KiB out), `MASQUERADE` present and counting 26 flows,
+`FORWARD -i/-o wg0 ACCEPT` counting 108 / 171 packets, and PiHole DNS answering from the node.
+
+**The fault was that a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
+holds itself.** Cilium's announcer answers ARP for the VIP on behalf of other LAN hosts, but not for
+the announcing node's own request, so the neighbour entry settles at `FAILED`:
+
+| from | `192.168.2.20` (Gateway) | `192.168.2.22` (crafty) | `192.168.2.24` (plex) | announces |
+|---|---|---|---|---|
+| `k3s-prod-1` | **FAILED** | — | — | `.20` |
+| `k3s-prod-2` | ok | **INCOMPLETE** | ok | `.21`, `.22`, `.23-tcp` |
+| `k3s-prod-3` | ok | ok | **INCOMPLETE** | `.24` |
+
+Symmetric across 3 nodes × 5 VIPs: each node reached every VIP **except** the ones it alone
+announced. It is a property of L2 announcement, not of any one Service, and it is harmless for
+ordinary pods — they reach Services through the cluster datapath and never ARP for a VIP. This
+component is the exception, because `hostNetwork` + `MASQUERADE` put remote clients on the node's
+own LAN path.
+
+The forward pointed at `k3s-prod-1`, which held the `shared` Gateway lease, so **all 16 Gateway
+hostnames were dead over the VPN**. Two things hid it:
+
+* **PiHole survived by accident.** `192.168.2.23` is announced by *two* Services
+  (`pihole-dns-udp` on prod-1, `pihole-dns-tcp` on prod-2), so prod-2 answered the ARP prod-1 could
+  not. Names resolved perfectly and every app hung — the hardest shape to diagnose from a client.
+* **Nothing reported it.** No probe in this repo compares the forwarded node against the lease
+  holders, and from outside WireGuard is silent to unauthenticated traffic anyway.
+
+Closed by labelling the forwarded node `homelab.keithjackson.dev/vpn-ingress=true` and excluding
+that label in `platform-config/cilium/l2-announcement-policy.yaml` (the node then ARP-resolves every
+VIP), with `VpnIngressNodeAnnouncesL2VIP` watching the lease holders so a recurrence is loud and
+`VpnIngressNodeLabelMissing` watching for the label going absent, which would silently re-arm the
+whole failure. Cost: VIP announcement candidacy drops from three nodes to two.
+
 ## Prerequisites outside this directory
 
 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`

platform/vpn-ingress/daemonset.yaml

@@ -62,9 +62,39 @@
 # as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to
 # whichever node the scheduler picked and silently invalidate that forward on every reschedule. A
 # DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of
-# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no
-# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own
-# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each.
+# them, and if that node is lost every other node is already warm and answers the same keys, so no
+# client profile ever changes. Each node binds udp/51820 in its own netns, so there is no conflict,
+# and the two idle listeners cost ~15 MiB each.
+#
+# BUT THE NODES ARE NOT INTERCHANGEABLE, AND THIS COMMENT USED TO CLAIM THEY WERE.
+# It said "FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work". That was wrong, and the
+# wording cost two days of silently broken remote access on 2026-10-08. MEASURED that day: a node
+# cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds ITSELF — the announcer
+# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and packets to
+# that VIP die before leaving the node:
+#
+#   k3s-prod-1, holding the shared Gateway lease:  ip neigh show 192.168.2.20 -> FAILED
+#   k3s-prod-2 / k3s-prod-3:                       ip neigh show 192.168.2.20 -> lladdr …(prod-1)
+#
+# Because this pod MASQUERADEs clients to the node's own address (see PostUp below), a remote client
+# inherits exactly that failure. With the forward on the Gateway's announcing node, all 16 Gateway
+# hostnames were unreachable over the VPN while PiHole DNS still resolved — a tunnel that
+# handshakes, resolves names, and then hangs on every single app.
+#
+# THE FIX IS DECLARATIVE AND LIVES OUTSIDE THIS FILE: the forwarded node is labelled
+# homelab.keithjackson.dev/vpn-ingress=true by infra/ansible/roles/node_baseline (from
+# node_baseline_vpn_ingress_node), and platform-config/cilium/l2-announcement-policy.yaml excludes
+# that label from announcing ANY VIP, so the node can ARP-resolve all of them. A label rather than a
+# cluster-vars knob because the alert that guards this has to read it too, and the alert unit-test
+# harness refuses a postBuild placeholder in an expr. Failover is therefore TWO steps — move
+# node_baseline_vpn_ingress_node and re-run the role, THEN re-point the router
+# (runbooks/vpn-remote-access.md step 3).
+# `VpnIngressNodeAnnouncesL2VIP` (platform/monitoring/alerts.yaml) fires if the two collide again.
+#
+# This pod deliberately does NOT probe that itself. A readinessProbe on VIP reachability would wedge
+# a DaemonSet rollout behind an unrelated Gateway outage (maxUnavailable: 1 below), and a liveness
+# probe would tear down a tunnel that is carrying LAN traffic perfectly well. The condition is a
+# property of the CLUSTER's announcement layout, so it is watched there.
 #
 # All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the
 # DaemonSet to cover the cluster; add them if a node is ever tainted.

runbooks/kustomization.yaml

@@ -119,6 +119,11 @@ configMapGenerator:
       - update-rollback=update-rollback.md
       - vpn-down=vpn-down.md
       - vpn-egress-stale=vpn-egress-stale.md
+      # `VpnIngressNodeAnnouncesL2VIP` — the INBOUND remote-access VPN, not the egress tunnel the two
+      # pages above cover. A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease
+      # it holds itself, and vpn-ingress MASQUERADEs clients to the node's own address, so the
+      # forwarded node must announce nothing (measured 2026-10-08).
+      - vpn-ingress-l2-collision=vpn-ingress-l2-collision.md
       # Operator procedure, not an alert page. Served because the component it completes is INERT
       # until three off-repo manual steps are done (kernel module, server key, WAN port-forward) —
       # and because the one check that actually proves it cannot be run from inside the house.

runbooks/vpn-ingress-l2-collision.md

@@ -0,0 +1,165 @@
+# vpn-ingress-l2-collision — the VPN ingress node is announcing a LoadBalancer VIP
+
+**Alerts:** `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) and `VpnIngressNodeLabelMissing`
+(warning, 30m)
+**File:** `platform/monitoring/alerts.yaml`
+**Related:** `/runbooks/vpn-remote-access` is the setup and enrolment runbook for this component.
+`/runbooks/vpn-egress-stale` and `/runbooks/vpn-down` are the **other** VPN — the outbound PIA
+tunnel — and have nothing to do with this.
+
+## What this means here
+
+**A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
+announcer answers ARP for the VIP on behalf of other LAN hosts, but not for the announcing node's
+own request. On the lease holder the neighbour entry settles at `FAILED`, and every packet to that
+VIP is dropped before it leaves the node. Measured on prod 2026-10-08, symmetric across all three
+nodes and five VIPs — each node reached every VIP **except** the ones it alone announced.
+
+For an ordinary workload this is harmless: pods reach Services through the cluster datapath and
+never ARP for a VIP. `platform/vpn-ingress` is the exception. It runs WireGuard with
+`hostNetwork: true` and MASQUERADEs remote clients to the node's own address, so a remote client's
+packets take that node's LAN path and inherit its ARP failures. **If the node the router forwards
+`udp/51820` to is also an L2 lease holder, every remote VPN client silently loses all access to the
+Services behind the VIPs it announces.**
+
+The node terminating WireGuard is marked with `homelab.keithjackson.dev/vpn-ingress=true`, and
+`platform-config/cilium/l2-announcement-policy.yaml` excludes that label from announcing anything,
+so this should be unreachable.
+
+**The symptom your users report will not sound like a network fault.** On 2026-10-08 the forward sat
+on `k3s-prod-1`, which held the `shared` Gateway lease, so all 16 Gateway hostnames were dead over
+the VPN — while PiHole DNS kept answering, because `192.168.2.23` is announced by *two* Services and
+`k3s-prod-2` held the other lease. So the tunnel handshakes, `wg show` looks perfect, names resolve,
+and **every app hangs on connect**. Expect "the VPN connects but nothing loads", not "the VPN is
+down".
+
+Which Services are affected depends on which lease moved — read `{{ $labels.lease }}` from the
+alert. `cilium-l2announce-gateway-cilium-gateway-shared` is the worst case: it is every web app at
+once.
+
+### If it is `VpnIngressNodeLabelMissing` that fired
+
+That is the watchdog, and it means the protection is **switched off**, not that something is merely
+untidy. The exclusion is a `NotIn` on the label, and `NotIn` with an absent key **matches** — so
+with the label on no node, the policy excludes nothing and every node is an announcement candidate
+again. `VpnIngressNodeAnnouncesL2VIP` is simultaneously **inert**: its join matches nothing and
+returns no series, which is indistinguishable from healthy. Treat a quiet critical alert as
+meaningless until this warning is cleared. Usual cause: a node was rebuilt and `node_baseline` has
+not been re-run on it.
+
+## First commands
+
+Run cluster commands through the pinned toolbox, which supplies the kube context:
+
+```bash
+# 1. WHO HOLDS WHAT, and WHO IS THE VPN NODE — the whole question, in two lines.
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n kube-system get lease | grep l2announce'
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/vpn-ingress'
+# Exactly ONE node should carry vpn-ingress=true, and NO l2announce lease should name it.
+
+# 2. IS THE EXCLUSION STILL IN THE LIVE POLICY? Expect a matchExpressions list with a
+#    homelab.keithjackson.dev/vpn-ingress NotIn [true] entry — not a bare matchLabels.
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get ciliuml2announcementpolicy default-l2 -o jsonpath="{.spec.nodeSelector}"; echo'
+
+# 3. IS THE LABEL REACHING PROMETHEUS? If this is empty the alerts are inert even when
+#    the label is on the node (kube-state-metrics metricLabelsAllowlist).
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n monitoring exec deploy/kube-prometheus-stack-kube-state-metrics -- \
+     wget -qO- localhost:8080/metrics | grep vpn_ingress | head'
+```
+
+**Confirm the actual damage**, which is the one measurement that is unambiguous. From the WireGuard
+pod on the affected node (it shares the host netns, so this is the path a VPN client takes):
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress get pods -o wide'   # pick the pod on that node
+
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- \
+     sh -c "ping -c1 -W1 192.168.2.20 >/dev/null 2>&1; ip neigh show 192.168.2.20"'
+# BROKEN:  192.168.2.20 dev eno1 FAILED
+# HEALTHY: 192.168.2.20 dev eno1 lladdr bc:e9:2f:7e:c4:a6 STALE|REACHABLE|DELAY
+```
+
+The `ping` is only there to trigger ARP — the VIP does not answer ICMP even when healthy, so
+**ignore its 100% packet loss**; the neighbour state is the answer. Substitute the VIP that belongs
+to `{{ $labels.lease }}` if the Gateway is not the one that moved.
+
+**Which node actually holds the forward** (the router cannot be queried, but the cluster knows):
+only the forwarded node's `wg show` has a peer with an `endpoint` and a recent `latest handshake`.
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'for p in $(kubectl --context k3s-prod -n vpn-ingress get pods -o name); do
+     echo "== $p"; kubectl --context k3s-prod -n vpn-ingress exec ${p#pod/} -c wireguard -- wg show; done'
+```
+
+## Likely causes, most likely first
+
+1. **The label is on the wrong node** — someone re-pointed the router's forward during a node
+   failover and did not move `node_baseline_vpn_ingress_node`. This is the most likely cause by a
+   wide margin, because the router is off-repo and nothing can check it. Confirm with the `wg show`
+   loop above: the node with the live peer endpoint is the real forward target.
+   *Fix:* merge a PR setting `node_baseline_vpn_ingress_node`
+   (`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`) to that node, then re-run the role —
+   `ansible-playbook -i inventory/hosts.ini <playbook> -e k3s_hosts=k3s_prod --tags vpn-ingress`.
+   The role labels the new node and **removes** the label from the old one. Then confirm Cilium
+   re-elected the lease off the newly-labelled node.
+
+2. **The exclusion was edited out of `platform-config/cilium/l2-announcement-policy.yaml`**, or the
+   file was reverted to a bare `matchLabels: {kubernetes.io/os: linux}`. Command 2 above shows this
+   directly. *Fix:* restore the `matchExpressions` form; the file's own header explains why it is
+   shaped that way.
+
+3. **The label is missing entirely** (`VpnIngressNodeLabelMissing` firing) — a node rebuild without
+   a `node_baseline` re-run. *Fix:* re-run the role as above.
+
+4. **The label is on the node but not in Prometheus** — `metricLabelsAllowlist` in
+   `platform/monitoring/kube-prometheus-stack.yaml` lost the `vpn-ingress` entry. Then the critical
+   alert is inert *and* the watchdog fires. Command 3 above distinguishes this from cause 3.
+
+5. **Cilium kept renewing a lease its holder is no longer eligible for.** The exclusion is correct
+   and in place, but the lease never re-elected. This was never verified when the fix was authored
+   (see `decisions/2026-10-17-vpn-ingress-l2-lease-collision.md`). *Fix:* delete that one Lease
+   object to force an election — `kubectl -n kube-system delete lease <name>` — and watch it come
+   back on a different node.
+
+6. **A Cilium upgrade changed `NotIn` semantics or the `nodeSelector` field.** Least likely, but it
+   is the reason this is a runtime alert rather than a static check on the policy text.
+
+## When to escalate or stop
+
+**Act now, don't wait for a window.** Remote access is fully broken for the affected VIPs while this
+is firing, and it does **not** self-heal: a lease holder keeps renewing, so this condition persists
+indefinitely once entered. That is also why the 15m `for:` is safe — a transient move during a node
+reboot or a Cilium roll resolves on its own well inside it.
+
+**The fastest honest mitigation is to move the forward, not the leases.** Re-pointing the router's
+`udp/51820` at a node that holds no lease for the affected VIP restores service in seconds, with no
+cluster change and no new client profiles — all three nodes answer the same keys. Follow it with the
+`node_baseline_vpn_ingress_node` PR and role re-run so the two agree again; until you do, this alert
+is watching the wrong node. Note each node's own announcements when choosing: as of 2026-10-08,
+prod-2 announced crafty `192.168.2.22` and unifi `192.168.2.21`, prod-3 announced plex
+`192.168.2.24`, so neither is a cost-free target.
+
+**Do not** `kubectl edit` the `CiliumL2AnnouncementPolicy` to fix this — Flux owns it and will revert
+you, and the drift is the kind that outlives the memory of making it. The two acceptable hand
+actions are deleting a single Lease to force re-election (cause 5), and `kubectl label node` as a
+*temporary* stopgap while the PR lands — note that a node rebuild or a `node_baseline` run will undo
+a hand-applied label, so it is not a fix.
+
+**Stop and escalate to the operator if** the leases will not leave the excluded node after a forced
+re-election, or if the intended forward target is unclear. Both need a decision about which node
+carries remote access, which is an operator call, not a runbook step.
+
+**Known gap, so you are not misled:** if someone re-points the router without moving the label,
+`VpnIngressNodeAnnouncesL2VIP` watches the wrong node and **stays silent** while the VPN breaks in
+exactly the way described above. There is no in-cluster signal for that today; closing it needs a
+WireGuard exporter publishing which node is actually terminating peer traffic
+(`platform/vpn-ingress/README.md`, "Not built") — the `wg show` loop above is the manual substitute.
+Until then, if remote access is broken and both alerts are quiet, **check the forward against the
+label by hand before believing the alerts.**

runbooks/vpn-remote-access.md

@@ -171,14 +171,57 @@ server without a `wg0` "ready".
 On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
 `192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.
 
-Pick any node — all three run an identical listener. Note which one you chose here:
+> ### You may NOT pick any node. The forward target must be the `vpn-ingress`-labelled node.
+>
+> This step used to say "pick any node — all three run an identical listener". **That sentence
+> caused a two-day silent outage of all remote access on 2026-10-08**, and it is why there is now a
+> cluster-vars knob instead of a free choice here.
+>
+> **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
+> announcer answers ARP for the VIP on behalf of other LAN hosts but not for itself, so on the lease
+> holder the neighbour entry settles at `FAILED` and packets to that VIP are dropped before they
+> leave the node. Measured on prod 2026-10-08:
+>
+> ```text
+> on k3s-prod-1 (held the shared Gateway lease):   ip neigh show 192.168.2.20 -> FAILED
+> on k3s-prod-2 / k3s-prod-3:                      ip neigh show 192.168.2.20 -> lladdr … (prod-1)
+> ```
+>
+> WireGuard runs on `hostNetwork` and MASQUERADEs remote clients to the node's own address, so a VPN
+> client inherits that node's ARP failure. With the forward on `k3s-prod-1`, which held the Gateway
+> lease, **all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved** — so
+> the tunnel handshook, names looked fine, and every single app hung on connect. Nothing reported
+> it.
+>
+`infra/ansible/roles/node_baseline` now labels the forwarded node
+> `homelab.keithjackson.dev/vpn-ingress=true` (from `node_baseline_vpn_ingress_node` in
+> `group_vars/k3s_prod/vars.yml`), and `platform-config/cilium/l2-announcement-policy.yaml` excludes
+> that label from announcing any VIP. **The forward must point at the node carrying that label.** CI
+> cannot read your router, so this parity is yours to hold — `VpnIngressNodeAnnouncesL2VIP` catches a
+> labelled node taking a lease, but it CANNOT catch the forward being moved without the label.
+
+Forward to the node named by `node_baseline_vpn_ingress_node` (`k3s-prod-1` → `192.168.2.10`).
+Note it here:
 
 ```
 # WAN udp/51820 -> 192.168.2.__  (k3s-prod-_)   set <date>
+# MUST equal node_baseline_vpn_ingress_node in group_vars/k3s_prod/vars.yml
 ```
 
-**Node loss = failover is this one line.** Re-point the forward at another node address; every
-device keeps working with the same profile, because all three nodes answer the same keys.
+**Node loss = failover is TWO changes, not one.** It used to be advertised as one router line; that
+was wrong, because re-pointing at another node lands the VPN on a node that announces VIPs and
+reintroduces the failure above against a different set of addresses (prod-2 announces crafty and
+unifi; prod-3 announces plex). Do both, in this order:
+
+1. **Merge a PR** setting `node_baseline_vpn_ingress_node` to the new node, then re-run the role
+   (`--tags vpn-ingress`). It labels the new node and removes the label from the old one; Cilium
+   then moves the leases off the newly-labelled node and onto its peers.
+2. **Re-point the router** forward at the new node's address.
+
+Every device keeps working with the same profile either way; all three nodes answer the same keys,
+and the listener stays warm on all three. Doing step 2 without step 1 gives you a tunnel that
+handshakes and routes nothing — and `VpnIngressNodeAnnouncesL2VIP` will **not** catch it, because
+that alert watches the node the knob names, not the node your router actually points at.
 
 Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
 already silent to unauthenticated probes (README, "Why this is safe to expose").