revert(vpn-ingress): withdraw the L2 lease-collision fix — a node CAN reach a VIP it announces

PR #69 claimed a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
lease it holds, and excluded the VPN ingress node from announcing on that basis.
The claim makes a sharp prediction; today's power-cycle put ALL EIGHT
cilium-l2announce-* leases on k3s-prod-1, and the prediction failed.

Measured 2026-10-17 from the wireguard pod's host netns on each node: the holder
reaches 192.168.2.20/.22/.23/.24 normally (curl 200 in 18ms on .20) while its
ip neigh for every one of them sits at FAILED. The asserted cause is present
without the asserted effect, which falsifies it. .21 (unifi) fails from all three
nodes and is a separate pre-existing fault.

It is inert because no path resolves a VIP by ARP: cilium-dbg reports
KubeProxyReplacement on [eno1, wg0] and programs 192.168.2.20:443 =>
127.0.0.1:13606, so VIPs are DNAT'd in BPF before any neighbour lookup, on the
forwarded path as well as the host one.

Both probes behind the original diagnosis were bad. `ip neigh` FAILED is
permanent and harmless, and a ping artefact. `curl --interface wg0` — the control
used to rule out a host-netns artefact — fails on all three nodes, including ones
whose ARP resolves, so it discriminates nothing. The deeper error: a MASQUERADEd
client is FORWARDED while every pod-runnable probe is socket-originated. Sharing
a netns is not sharing a hook.

Reverted the policy nodeSelector, the Ansible node label, both alerts, their
promtool tests and the kube-state-metrics allowlist entry. Kept documentation
only: the measurements, which probes lie, and the fact that the forwarded path
remains unmeasured without a connected client.

Also recorded, for the reported "LAN and cluster work, internet does not":
new-peer.sh has emitted a FULL tunnel (0.0.0.0/0) by default since f92707f, so
there is no full-tunnel option to add and no code change is made. The server side
is capable — ip_forward=1, the 10.90.0.0/24 MASQUERADE present with an empty
CILIUM_POST_nat ahead of it, FORWARD accepts, default route intact, nodes reach
1.1.1.1. A full tunnel egresses the HOUSE WAN, not PIA: the egress policy selects
pods by egress-class=vpn, this pod is labelled internet, and tun0 exists on
k3s-prod-3 only. Root cause NOT found; no client has connected since the
power-cycle (0 forwarded packets, no handshake), so it is not reproducible here.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

runbooks/vpn-remote-access.md

@@ -171,57 +171,14 @@ server without a `wg0` "ready".
 On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
 `192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.
 
-> ### You may NOT pick any node. The forward target must be the `vpn-ingress`-labelled node.
->
-> This step used to say "pick any node — all three run an identical listener". **That sentence
-> caused a two-day silent outage of all remote access on 2026-10-08**, and it is why there is now a
-> cluster-vars knob instead of a free choice here.
->
-> **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
-> announcer answers ARP for the VIP on behalf of other LAN hosts but not for itself, so on the lease
-> holder the neighbour entry settles at `FAILED` and packets to that VIP are dropped before they
-> leave the node. Measured on prod 2026-10-08:
->
-> ```text
-> on k3s-prod-1 (held the shared Gateway lease):   ip neigh show 192.168.2.20 -> FAILED
-> on k3s-prod-2 / k3s-prod-3:                      ip neigh show 192.168.2.20 -> lladdr … (prod-1)
-> ```
->
-> WireGuard runs on `hostNetwork` and MASQUERADEs remote clients to the node's own address, so a VPN
-> client inherits that node's ARP failure. With the forward on `k3s-prod-1`, which held the Gateway
-> lease, **all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved** — so
-> the tunnel handshook, names looked fine, and every single app hung on connect. Nothing reported
-> it.
->
-`infra/ansible/roles/node_baseline` now labels the forwarded node
-> `homelab.keithjackson.dev/vpn-ingress=true` (from `node_baseline_vpn_ingress_node` in
-> `group_vars/k3s_prod/vars.yml`), and `platform-config/cilium/l2-announcement-policy.yaml` excludes
-> that label from announcing any VIP. **The forward must point at the node carrying that label.** CI
-> cannot read your router, so this parity is yours to hold — `VpnIngressNodeAnnouncesL2VIP` catches a
-> labelled node taking a lease, but it CANNOT catch the forward being moved without the label.
-
-Forward to the node named by `node_baseline_vpn_ingress_node` (`k3s-prod-1` → `192.168.2.10`).
-Note it here:
+Pick any node — all three run an identical listener. Note which one you chose here:
 
 ```
 # WAN udp/51820 -> 192.168.2.__  (k3s-prod-_)   set <date>
-# MUST equal node_baseline_vpn_ingress_node in group_vars/k3s_prod/vars.yml
 ```
 
-**Node loss = failover is TWO changes, not one.** It used to be advertised as one router line; that
-was wrong, because re-pointing at another node lands the VPN on a node that announces VIPs and
-reintroduces the failure above against a different set of addresses (prod-2 announces crafty and
-unifi; prod-3 announces plex). Do both, in this order:
-
-1. **Merge a PR** setting `node_baseline_vpn_ingress_node` to the new node, then re-run the role
-   (`--tags vpn-ingress`). It labels the new node and removes the label from the old one; Cilium
-   then moves the leases off the newly-labelled node and onto its peers.
-2. **Re-point the router** forward at the new node's address.
-
-Every device keeps working with the same profile either way; all three nodes answer the same keys,
-and the listener stays warm on all three. Doing step 2 without step 1 gives you a tunnel that
-handshakes and routes nothing — and `VpnIngressNodeAnnouncesL2VIP` will **not** catch it, because
-that alert watches the node the knob names, not the node your router actually points at.
+**Node loss = failover is this one line.** Re-point the forward at another node address; every
+device keeps working with the same profile, because all three nodes answer the same keys.
 
 Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
 already silent to unauthenticated probes (README, "Why this is safe to expose").
@@ -316,6 +273,84 @@ bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
 by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
 if the router is not delivering the packets, and a `handshake` with no completed session if it is.
 
+## Triage — "the client reaches the LAN and cluster apps, but not the internet"
+
+Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom
+almost always lives, and because the server side has been measured and is not the cause.
+
+**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.**
+
+| the profile says | meaning |
+|---|---|
+| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. |
+| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. |
+
+`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit —
+`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client
+profile only**, so an already-issued profile keeps whatever it was rendered with until it is
+re-rendered and re-imported on the device. The two forms differ by one flag:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet>'           # full tunnel
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet> --split'   # LAN only
+```
+
+Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into
+`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is
+editing `AllowedIPs` in the profile **on the device** — the server does not care.
+
+**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the
+client connected, then watch the two counters that only forwarded traffic moves:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c "
+     wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5;
+     iptables -t nat -L POSTROUTING -n -v | head -5"'
+```
+
+* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which
+  node the router's `udp/51820` forward points at (the router is unreadable from the cluster).
+* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is
+  upstream of the cluster (router forward, client, or carrier), not here.
+* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side
+  failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there,
+  which breaks internet while node-local cluster VIPs still work. It has not been observed.
+
+**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly:
+
+* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack
+  client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel —
+  so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4:
+  `curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it,
+  and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is
+  blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it).
+* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified
+  2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a
+  blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on
+  a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all.
+
+**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the
+`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain
+**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route
+`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the
+PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled
+`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the
+**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6).
+
+## Triage — which L2 announcement lease a node holds does NOT matter
+
+Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node
+could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`:
+that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh`
+for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer
+VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend
+is a node-local socket. Full measurements and the two misleading probes
+(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`.
+
+`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a
+separate, pre-existing fault in that service and has nothing to do with the VPN.
+
 ## Revoking a device
 
 Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the