revert(vpn-ingress): withdraw the L2 lease-collision fix — a node CAN reach a VIP it announces

PR #69 claimed a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
lease it holds, and excluded the VPN ingress node from announcing on that basis.
The claim makes a sharp prediction; today's power-cycle put ALL EIGHT
cilium-l2announce-* leases on k3s-prod-1, and the prediction failed.

Measured 2026-10-17 from the wireguard pod's host netns on each node: the holder
reaches 192.168.2.20/.22/.23/.24 normally (curl 200 in 18ms on .20) while its
ip neigh for every one of them sits at FAILED. The asserted cause is present
without the asserted effect, which falsifies it. .21 (unifi) fails from all three
nodes and is a separate pre-existing fault.

It is inert because no path resolves a VIP by ARP: cilium-dbg reports
KubeProxyReplacement on [eno1, wg0] and programs 192.168.2.20:443 =>
127.0.0.1:13606, so VIPs are DNAT'd in BPF before any neighbour lookup, on the
forwarded path as well as the host one.

Both probes behind the original diagnosis were bad. `ip neigh` FAILED is
permanent and harmless, and a ping artefact. `curl --interface wg0` — the control
used to rule out a host-netns artefact — fails on all three nodes, including ones
whose ARP resolves, so it discriminates nothing. The deeper error: a MASQUERADEd
client is FORWARDED while every pod-runnable probe is socket-originated. Sharing
a netns is not sharing a hook.

Reverted the policy nodeSelector, the Ansible node label, both alerts, their
promtool tests and the kube-state-metrics allowlist entry. Kept documentation
only: the measurements, which probes lie, and the fact that the forwarded path
remains unmeasured without a connected client.

Also recorded, for the reported "LAN and cluster work, internet does not":
new-peer.sh has emitted a FULL tunnel (0.0.0.0/0) by default since f92707f, so
there is no full-tunnel option to add and no code change is made. The server side
is capable — ip_forward=1, the 10.90.0.0/24 MASQUERADE present with an empty
CILIUM_POST_nat ahead of it, FORWARD accepts, default route intact, nodes reach
1.1.1.1. A full tunnel egresses the HOUSE WAN, not PIA: the egress policy selects
pods by egress-class=vpn, this pod is labelled internet, and tun0 exists on
k3s-prod-3 only. Root cause NOT found; no client has connected since the
power-cycle (0 forwarded packets, no handshake), so it is not reproducible here.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

decisions/2026-10-17-oauth2-proxy-oidc-startup-race.md

@@ -0,0 +1,99 @@
+---
+date: 2026-10-17
+branch: fix/oauth2-proxy-oidc-startup-race
+title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
+---
+
+## What changed
+
+Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
+discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
+serving yet, so the shared Gateway answers 503 and the proxy dies:
+
+```
+[provider.go:55] Performing OIDC Discovery...
+[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
+             failed to discover OIDC configuration: unexpected status "503": no healthy upstream
+```
+
+It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
+replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
+retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
+is already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+
+`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
+polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
+accrue CrashLoopBackOff backoff and retries promptly, which is the point.
+
+**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
+case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
+for a pod wedged in `Init` forever, which is not an improvement.
+
+## Why this shape and not the alternatives
+
+* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
+  example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
+  pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
+  at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
+  "reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
+* **No new base image.** The chart already runs an `alpine` init container for its built-in
+  `initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
+  `3.22` (verified to exist upstream) rather than the chart's own `latest`.
+* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
+  this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
+  the chart already has a hook for.
+* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
+  more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
+  oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
+  the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
+  the operator out of all of them, so the reversible, additive option wins.
+
+The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
+with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
+and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
+silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
+the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.
+
+## What I verified
+
+* **The failure is measured, not theorised.** The log above is from the live pod
+  (`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
+  with `--previous` after the cold boot. The surviving replica's log shows the same startup
+  succeeding once Keycloak was up, which is what identifies this as a race rather than a config
+  fault.
+* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
+  clean and both replicas have been healthy since.
+* **The chart key is real**, at the pinned version range: `helm search repo` resolves
+  `>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
+  `extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
+* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
+  assumed from memory.
+* The edited HelmRelease parses as YAML.
+
+## What I did NOT verify
+
+* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
+  seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
+  replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
+  seven apps, so **do it when you can watch it**, not unattended.
+* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
+  oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
+  from the measured failure, not demonstrated against it.
+* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
+  `ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
+  failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
+  degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
+  anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
+  init container's log on first apply**: it should print "Keycloak discovery reachable" within
+  seconds on a warm cluster, not the give-up message.
+* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
+  re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
+  good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
+  (`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
+  Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
+  this should hold — unverified.
+* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
+  not a measured bound.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
+  subset only.
+* I did not re-measure branch protection or runner state (CLAUDE.md §2b).

decisions/2026-10-17-vpn-ingress-l2-lease-collision.md

@@ -0,0 +1,198 @@
+---
+date: 2026-10-17
+branch: fix/vpn-ingress-l2-lease-collision
+title: "WITHDRAWN: a node CAN reach a VIP it announces. My L2-lease diagnosis was wrong"
+---
+
+## What changed
+
+**This branch previously proposed a fix. That fix has been reverted in full. The diagnosis behind it
+was wrong.** What is left is documentation of what was actually measured, so the wrong conclusion is
+not re-derived — it is very easy to re-derive from `ip neigh` alone.
+
+### The claim, and how it died
+
+The earlier version of this entry claimed: *a node cannot reach a LoadBalancer VIP whose Cilium L2
+announcement lease it holds itself* — the announcer does not answer its own ARP request, the
+neighbour entry settles at `FAILED`, and packets to that VIP die before leaving the node. Because
+`platform/vpn-ingress` runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
+own address, a VPN client was said to inherit that. The fix labelled the forwarded node and excluded
+that label from announcing, with two alerts and four promtool tests behind it.
+
+The claim makes a sharp prediction. After today's UPS power-cycle **all eight** `cilium-l2announce-*`
+leases are held by `k3s-prod-1` (gateway, pihole tcp+udp, crafty tcp+udp, unifi tcp+udp, plex) —
+the extreme case. The prediction was therefore that `k3s-prod-1` should reach **none** of the five
+VIPs and the other two nodes should reach **all**.
+
+**The prediction failed.** Measured from the host netns of the `wireguard` pod on each node:
+
+| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
+|---|---|---|
+| `ip neigh show 192.168.2.20` | `FAILED`, no lladdr | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
+| `curl -k https://192.168.2.20/` | **`200` in 18 ms** | `200` in 13–40 ms |
+| `nc -z 192.168.2.20 443` | **OK** | OK |
+| `192.168.2.22:25565`, `.23:53`, `.24:32400` | **all OK / `200`** | all OK / `200` |
+| `192.168.2.21:8443` | FAIL | **FAIL** (both) |
+
+The asserted **cause is still present** today — `FAILED` on the holder, for all five VIPs — while the
+asserted **effect is absent**. That is sufficient to falsify the causal claim without having to
+explain anything else. The ARP asymmetry is real and reproducible; it is also **inert**.
+
+### Why it is inert
+
+No traffic path resolves a LoadBalancer VIP by ARP:
+
+* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
+  `Masquerading: BPF [eno1, wg0]` (Cilium v1.20.1, TCX attach, tunnel/vxlan). Cilium's LB datapath
+  is attached to **`wg0` as well as `eno1`**, so a remote client's packet arriving on `wg0` for a VIP
+  is DNAT'd **in BPF, before any route or neighbour lookup for the VIP**.
+* `cilium-dbg service list` programs `192.168.2.20:443 => 127.0.0.1:13606` — the shared Gateway's
+  backend is a **node-local** Envoy socket, so every node serves that VIP without a LAN packet.
+* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
+  which is why `curl` succeeds on the holder. Discriminator: a real service port connects while a
+  bogus one (`nc -z 192.168.2.20 9999`) fails, identically on all three nodes — service-aware DNAT,
+  not wire traffic.
+
+### The two probes that misled me
+
+Both were cited as evidence in the previous version of this entry.
+
+* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless, and is also a probing
+  artefact: VIPs answer no ICMP, so a `ping` leaves a failed entry whether or not anything is wrong.
+* **`curl --interface wg0`.** The previous entry used this as the control that ruled out a
+  host-netns artefact. It is not a control: it fails (`rc=28`) from **all three** nodes, including
+  the two whose ARP for the VIP resolves perfectly. It is uncorrelated with lease holding, so it
+  discriminates nothing.
+
+**The deeper error:** a MASQUERADEd client's packets are **forwarded** (netfilter `FORWARD`, then
+routing); everything runnable from the pod is **socket-originated**. Different hooks. A host-netns
+probe cannot measure the forwarded path in either direction — it reports success where the forwarded
+path might fail, and `--interface wg0` reports failure where it does not. Sharing a netns is not
+sharing a hook, and that is the assumption the original diagnosis rested on.
+
+### Files
+
+Reverted to `origin/main`: `platform-config/cilium/l2-announcement-policy.yaml`,
+`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`,
+`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`, `platform/monitoring/alerts.yaml`,
+`platform/monitoring/kube-prometheus-stack.yaml`, `runbooks/kustomization.yaml`. Deleted:
+`platform/monitoring/tests/vpn-ingress-l2.test.yaml`, `runbooks/vpn-ingress-l2-collision.md`.
+
+Kept, rewritten from scratch:
+
+* **`platform/vpn-ingress/README.md`** — two measured sections: the L2 table above with why it is
+  inert and which probes lie, and the internet-egress facts below.
+* **`runbooks/vpn-remote-access.md`** — a triage section for the reported symptom ("LAN and cluster
+  work, internet does not") that works outwards from the client profile, and a short section saying
+  the L2 lease is a dead end, with the unifi `.21` note.
+
+**Nothing in `platform/vpn-ingress` or `infra/vpn-ingress` changed behaviourally. No code fix is
+proposed by this branch any more.**
+
+### The internet-routing report: no server-side fix was needed, and none was made
+
+The brief's prior was that `new-peer.sh` renders a split tunnel and that the operator wants a full
+one, to be added as a documented option. **That prior does not match the tree.** `new-peer.sh` has
+emitted `AllowedIPs = 0.0.0.0/0` **by default since the component's first commit** (`f92707f`,
+2026-10-05, confirmed with `git log -S`); `--split` is the opt-in, and
+`runbooks/vpn-remote-access.md:252-253` already documents both forms. So there is no full-tunnel
+feature to add, and adding a `WG_FULL_TUNNEL` knob would be a second switch for a default that
+already exists. **I deliberately shipped no code change here** — writing one would have meant
+inventing a fix for a symptom I could not reproduce (see below).
+
+The server side was measured end to end and is capable and correct, so the cause is in the client
+profile or the client. The triage section names the two candidates that fit the symptom exactly —
+a `--split` profile, and the IPv4-only `AllowedIPs` letting a dual-stack client prefer AAAA and
+leak public traffic outside the tunnel — each with a one-line test that settles it.
+
+## What I verified
+
+Measured live against prod, `kubectl --context k3s-prod --request-timeout=15s`, read-only, on
+2026-10-17:
+
+* **The lease distribution.** All eight `cilium-l2announce-*` leases held by `k3s-prod-1`
+  (`.spec.holderIdentity`). The brief's observation reproduced; it had not moved again.
+* **The falsification, in full**, as tabulated above: 3 nodes x 5 VIPs, `ip neigh` + `nc -z` + a
+  real `curl` for each. The holder reaches four of five VIPs; `.21` fails from everywhere.
+* **The socket-LB discriminator.** Real service port OK, bogus port FAIL, identically on all three
+  nodes — so the successes are service-aware DNAT, not wire delivery.
+* **`curl --interface wg0` fails on all three nodes**, including non-announcing ones with
+  `REACHABLE` neighbour entries. This is the measurement that destroys the original control.
+* **The datapath facts** quoted above from `cilium-dbg status --verbose` and
+  `cilium-dbg service list` on `cilium-55jv8` (`k3s-prod-1`), plus `cilium-config`:
+  `kube-proxy-replacement=true`, `enable-l2-announcements=true`, `enable-bpf-masquerade=true`,
+  `routing-mode=tunnel`, `devices=` (empty, auto-detected). Cilium `v1.20.1` on all three.
+* **The VIP is not a local address and routes on-link.** `ip route get 192.168.2.20` →
+  `dev eno1 src 192.168.2.10` on the holder; nothing in `table local`; `ip -4 addr` shows no VIP.
+  So the 200 is not loopback delivery by the host stack — it is BPF.
+* **The internet path, server side, on all three nodes:** `/proc/sys/net/ipv4/ip_forward` = `1`
+  (read, never written); `iptables -t nat -S POSTROUTING` carries
+  `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE`; the preceding `CILIUM_POST_nat` chain is **empty with 0
+  packets**, so it cannot short-circuit that rule; `FORWARD -i wg0` and `-o wg0` ACCEPT both
+  present; `default via 192.168.2.1 dev eno1 onlink` intact; `ip route get 1.1.1.1` resolves via it;
+  `curl http://1.1.1.1/` returns `301` from each node.
+* **The PIA / egress-gateway answer.** `platform/vpn-egress/egress-steering.yaml` selects by
+  `podSelector` on `egress-class: vpn`; `platform/vpn-ingress/daemonset.yaml:130` labels this pod
+  `egress-class: internet`; `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`), absent on
+  `-1` and `-2`. So a full tunnel egresses the **house WAN**, and a client's apparent address is the
+  home WAN IP, not PIA's.
+* **PiHole resolves public names** — `nslookup google.com 192.168.2.23` from the pod returned
+  `142.250.113.138`.
+* **No client is connected, and none has been since the power-cycle.** `wg show wg0` lists the one
+  enrolled peer (`10.90.0.2/32`) with **no endpoint and no handshake** on all three nodes, and both
+  `FORWARD -i wg0` and the `MASQUERADE` rule read **0 packets / 0 bytes** on all three. The
+  `wireguard` pods restarted 153 min ago (2 restarts, 11d old), which reset the counters.
+* **The full-tunnel default's age**, from `git log -S` on `new-peer.sh`: introduced in `f92707f`
+  (2026-10-05) and never changed.
+* **The branch rebased cleanly** onto `origin/main` (`1a0b208`, PR #70 merged) with no conflicts.
+
+## What I did NOT verify
+
+* **THE FORWARDED PATH IS STILL UNMEASURED, in both directions.** No client is connected, so I could
+  not test what a real remote client does, and — as established above — **no probe runnable from the
+  pod can substitute for it**. Specifically I did **not** prove that a forwarded client packet to a
+  VIP is DNAT'd by Cilium's `wg0` program. That inference rests on `cilium-dbg` listing `wg0` as a
+  LB device plus the node-local Gateway backend, not on an observed client packet. **It is the one
+  remaining way the withdrawn diagnosis could be partly right** — if `wg0` BPF were somehow not in
+  the forwarded path, forwarded traffic would need ARP and would die on the holder. I judge this
+  unlikely but it is unproven. *To close it: connect a client, then from the node whose `wg show`
+  shows the peer endpoint, watch `FORWARD -i wg0` and the `MASQUERADE` counters move while the
+  client loads a cluster app — and repeat it on a node that holds leases and one that does not.*
+* **WHICH NODE THE ROUTER FORWARDS `udp/51820` TO IS UNDETERMINABLE RIGHT NOW**, for the same
+  reason. **I need a client connected** to learn it; the router is unreadable from the cluster.
+* **I cannot reproduce or explain the 2026-10-08 measurements.** That entry recorded `nc -z` TCP
+  failures correlated with lease holding across 15 node x VIP pairs. Today the same test passes.
+  My leading hypothesis is **stale or incorrectly-programmed BPF LB state on `k3s-prod-1`**,
+  rebuilt by today's full power-cycle — which would make the lease correlation coincidental. A
+  competing hypothesis is that Cilium had not auto-detected `wg0` then (`devices` is empty, and
+  `wg0` is created by this DaemonSet after Cilium starts). **Both are unproven, and neither is
+  needed for the falsification** — the asserted cause is present today without its effect. I did not
+  retro-measure anything, and I did not read Cilium's source or issue tracker.
+* **The operator's report cannot be a description of "right now."** No client has connected since
+  the power-cycle (0 forwarded packets, no handshake on any node), so "clients can reach cluster
+  services and the LAN but not the internet" must describe a session before the power-cycle or one
+  that has since dropped. **I did not establish when it was observed, or with which profile.**
+* **THE INTERNET ROOT CAUSE IS NOT DIAGNOSED.** I proved the server side is *capable*; I did **not**
+  find the fault, and I shipped no fix for it. The two candidates in the triage section (a
+  `--split` profile; IPv6 preference leaking outside an IPv4-only `AllowedIPs`) are **reasoned, not
+  measured**. *What I need:* the `AllowedIPs` line from the client's own profile — that one line
+  decides split vs full and halves the search — and a connected client for step 2.
+* **Whether `--split` was in fact used for the enrolled peer.** The profile holds a private key and
+  lives only on the operator's machine and the device; it is not in the repo, and I did not look for
+  it. Server-side `AllowedIPs` is a `/32` source filter and says nothing about the client's routing.
+* **Whether `MTU = 1420` holds on a full-tunnel WAN path.** Not measured, and not measurable without
+  a client. It is the standard 1500 − 60; a client behind PPPoE or CGNAT with a smaller path MTU
+  would see large transfers hang while small requests succeed. The runbook's existing MTU note
+  (drop to 1280) already covers the remedy.
+* **Adding `::/0` to `AllowedIPs` is untested advice.** The triage section offers it to blackhole
+  IPv6 rather than leak it, and states that the tunnel has no IPv6 egress. I did not try it on a
+  client, and I do not know how each WireGuard client behaves when handed an IPv6 route on an
+  interface with no IPv6 address.
+* **`192.168.2.21:8443` (unifi) fails from all three nodes**, as it did on 2026-10-08 — a separate,
+  pre-existing fault. Not investigated, not addressed, and now recorded in the runbook as such.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so the gate proves the L0/L1
+  offline subset only. Nothing above is covered by a test that runs in CI.
+* **Nothing was applied, and nothing needs to be.** The branch is now documentation-only, so there
+  is no Flux reconcile, Ansible run or `kubectl` step to perform after merge. I ran no mutating
+  `kubectl`, `flux` or Ansible, and did not run `new-peer.sh` (it mints key material).
+* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).

platform/oauth2-proxy/helmrelease.yaml

@@ -63,6 +63,75 @@ spec:
       # gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
       skip-provider-button: "true"
       scope: "openid email profile"
+    # ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ──────────
+    # MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at
+    # startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so
+    # the Gateway answers 503 and the proxy dies:
+    #
+    #   [provider.go:55] Performing OIDC Discovery...
+    #   [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC
+    #                configuration: unexpected status "503": no healthy upstream
+    #
+    # It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was
+    # still down minutes later and only a manual pod delete cleared it. There is no retry flag; the
+    # process simply does not survive a cold start. A readinessProbe cannot help (the container is
+    # already dead), and CrashLoopBackOff backoff is not tunable per-pod.
+    #
+    # `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in
+    # values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init
+    # container for its built-in waitForRedis gate, so this adds no new base image to the platform.
+    # A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole
+    # point.
+    #
+    # BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst
+    # case is then exactly today's behaviour, never something new. An unbounded wait would trade a
+    # CrashLoop for a pod wedged in Init forever, which is not an improvement.
+    #
+    # SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is
+    # rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which
+    # CONSUMES braced expansions -- so a braced shell variable renders to the empty string and
+    # silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars
+    # knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud
+    # OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.)
+    extraInitContainers:
+      - name: wait-for-idp
+        image: docker.io/library/alpine:3.22
+        imagePullPolicy: IfNotPresent
+        command: ["/bin/sh", "-c"]
+        args:
+          - |
+            set -u
+            url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
+            i=0
+            while [ "$i" -lt 60 ]; do
+              # busybox wget validates against alpine's ca-certificates-bundle, so a real TLS
+              # failure is a non-zero exit here rather than a false pass.
+              if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then
+                echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy"
+                exit 0
+              fi
+              i=$(( i + 1 ))
+              sleep 5
+            done
+            echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2
+            echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2
+            exit 0
+        securityContext:
+          allowPrivilegeEscalation: false
+          runAsNonRoot: true
+          runAsUser: 65534
+          runAsGroup: 65534
+          readOnlyRootFilesystem: true
+          capabilities:
+            drop:
+              - ALL
+        resources:
+          requests:
+            cpu: 10m
+            memory: 16Mi
+          limits:
+            memory: 32Mi
+
     # Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
     metrics:
       enabled: true

platform/vpn-ingress/README.md

@@ -186,6 +186,86 @@ right way; one was a defect in my own manifest.
 | Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
 | Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
 
+## Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach
+
+A change was proposed (PR #69) on the belief that **a node cannot reach a LoadBalancer VIP whose
+Cilium L2 announcement lease it holds**, and that the VPN node therefore had to be excluded from
+announcing. **That belief is wrong, and the change was withdrawn.** The half of it that is true is
+real but inert, and the distinction matters enough to record, because the wrong version is very
+easy to re-derive from `ip neigh` alone.
+
+Measured with all **eight** `cilium-l2announce-*` leases held by `k3s-prod-1` — the extreme case,
+after the UPS power-cycle — from the host netns of the `wireguard` pod on each node:
+
+| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
+|---|---|---|
+| `ip neigh show 192.168.2.20` | `FAILED` (no lladdr) | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
+| `curl -k https://192.168.2.20/` | **`200`, 18 ms** | `200`, 13–40 ms |
+| `nc -z 192.168.2.20 443` | **OK** | OK |
+| plex `192.168.2.24:32400` | **`200`** | `200` |
+
+So the announcer really does **not** answer its own ARP request — that part of the observation is
+reproducible. It changes nothing, because **no traffic path resolves a LoadBalancer VIP by ARP**:
+
+* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
+  `Masquerading: BPF [eno1, wg0]`. Cilium's LB datapath is attached to **`wg0` as well as `eno1`**,
+  so a remote client's packet arriving on `wg0` for a VIP is DNAT'd to a backend **in BPF, before
+  any route or neighbour lookup for the VIP happens**.
+* `cilium-dbg service list` on `k3s-prod-1` programs `192.168.2.20:443 => 127.0.0.1:13606` — the
+  shared Gateway's backend is a **node-local** Envoy socket. Every node can serve that VIP without
+  a packet crossing the LAN at all.
+* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
+  which is why `curl` succeeds on the lease holder despite `FAILED`. The discriminator: a **real**
+  service port connects, a **bogus** one (`nc -z 192.168.2.20 9999`) fails, on all three nodes —
+  that is service-aware DNAT, not wire traffic.
+
+**Two tests that look like proof and are not.** Both were used to support the withdrawn diagnosis:
+
+* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless. It is also an
+  artefact of probing: VIPs answer no ICMP, so a `ping` leaves a failed entry behind whether or not
+  anything is wrong.
+* **`curl --interface wg0`.** This fails (`rc=28`, timeout) from **all three** nodes, including the
+  two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it
+  discriminates nothing — and it is not the client path either (see below).
+
+**Why neither probe can see the client path.** A MASQUERADEd remote client's packets are
+**forwarded** — netfilter `FORWARD`, then routing — whereas everything runnable from the pod is
+**socket-originated**. The two traverse different hooks, so a host-netns probe cannot measure the
+forwarded path in either direction: it reports success where the forwarded path might fail, and
+`--interface wg0` reports failure where it does not. **The forwarded path can only be measured with
+a client actually connected**, by watching the `FORWARD -i wg0` and `POSTROUTING … MASQUERADE`
+counters move.
+
+## Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses
+
+`new-peer.sh` has emitted a **full tunnel** (`AllowedIPs = 0.0.0.0/0`) **by default since the
+component's first commit**; `--split` is the opt-in for LAN-only. There is no full-tunnel feature
+to add. The server side was checked end to end and is **capable and correct** — if a full-tunnel
+client cannot reach the internet, nothing in the list below is the reason:
+
+| checked | result on all three nodes |
+|---|---|
+| `net.ipv4.ip_forward` (read from `/proc/sys`, never written) | `1` |
+| `iptables -t nat -S POSTROUTING` | carries `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` — NATs to **any** outbound interface, so internet NAT needs no change |
+| `CILIUM_POST_nat`, which precedes it in the chain | **empty, 0 packets** — it cannot short-circuit the MASQUERADE |
+| `iptables -L FORWARD` | `-i wg0` and `-o wg0` ACCEPT both present |
+| default route | `default via 192.168.2.1 dev eno1 onlink`, intact |
+| node's own internet reachability | `http://1.1.1.1/` returns `301` |
+
+**A full tunnel egresses the HOUSE WAN, not PIA.** This was the trap worth checking, and it is
+closed: `platform/vpn-egress/egress-steering.yaml`'s `CiliumEgressGatewayPolicy` selects by
+`podSelector` on `egress-class: vpn`, and this DaemonSet is deliberately labelled
+`egress-class: internet`, so it is not selected. Forwarded `wg0` traffic is not pod-identity
+traffic at all, and `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`) — the egress-gateway
+node. A remote client's apparent public address is therefore the **home WAN IP**, which is the
+intended behaviour and the thing to state when enrolling a device.
+
+**Not measurable without a connected client.** At the time of writing `wg show wg0` listed the one
+enrolled peer with **no endpoint and no handshake** on every node, and both the `FORWARD -i wg0` and
+the `MASQUERADE` counters read **0 packets** — so no client traffic had been forwarded since the
+power-cycle, and which node the router forwards `udp/51820` to was **undeterminable**. Only the
+forwarded node shows a peer with an `endpoint` and a recent `latest handshake`.
+
 ## Prerequisites outside this directory
 
 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`

runbooks/vpn-remote-access.md

@@ -273,6 +273,84 @@ bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
 by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
 if the router is not delivering the packets, and a `handshake` with no completed session if it is.
 
+## Triage — "the client reaches the LAN and cluster apps, but not the internet"
+
+Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom
+almost always lives, and because the server side has been measured and is not the cause.
+
+**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.**
+
+| the profile says | meaning |
+|---|---|
+| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. |
+| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. |
+
+`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit —
+`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client
+profile only**, so an already-issued profile keeps whatever it was rendered with until it is
+re-rendered and re-imported on the device. The two forms differ by one flag:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet>'           # full tunnel
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet> --split'   # LAN only
+```
+
+Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into
+`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is
+editing `AllowedIPs` in the profile **on the device** — the server does not care.
+
+**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the
+client connected, then watch the two counters that only forwarded traffic moves:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c "
+     wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5;
+     iptables -t nat -L POSTROUTING -n -v | head -5"'
+```
+
+* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which
+  node the router's `udp/51820` forward points at (the router is unreadable from the cluster).
+* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is
+  upstream of the cluster (router forward, client, or carrier), not here.
+* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side
+  failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there,
+  which breaks internet while node-local cluster VIPs still work. It has not been observed.
+
+**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly:
+
+* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack
+  client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel —
+  so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4:
+  `curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it,
+  and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is
+  blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it).
+* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified
+  2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a
+  blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on
+  a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all.
+
+**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the
+`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain
+**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route
+`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the
+PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled
+`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the
+**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6).
+
+## Triage — which L2 announcement lease a node holds does NOT matter
+
+Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node
+could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`:
+that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh`
+for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer
+VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend
+is a node-local socket. Full measurements and the two misleading probes
+(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`.
+
+`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a
+separate, pre-existing fault in that service and has nothing to do with the VPN.
+
 ## Revoking a device
 
 Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the