docs(vpn-ingress): withdraw the L2-lease diagnosis — it was falsified by measurement

The claim this branch originally shipped (a node cannot reach a LoadBalancer VIP
whose Cilium L2 announcement lease it holds, so VPN clients inherit that via
hostNetwork MASQUERADE) is FALSE. Re-measured with all 8 l2announce leases on
k3s-prod-1:

  ip neigh show 192.168.2.20   -> INCOMPLETE   (cause present)
  nc -z 192.168.2.20 443       -> OK
  curl --resolve ... :192.168.2.20  -> 200 in 25ms   (effect absent)

Cause present, effect absent. prod-2 and prod-3 return the same 200.

Why it was never going to bite: cilium-dbg status shows
`KubeProxyReplacement: True [eno1 ..., wg0 10.90.0.1]` and
`Masquerading: BPF [eno1, wg0]` -- Cilium's LB datapath is attached to wg0, so a
client packet for a VIP is DNAT'd in BPF BEFORE any neighbour lookup, and the
Gateway backend is a node-local Envoy socket anyway.

The methodological error, which is the part worth keeping: every probe was run
from the wireguard pod on the grounds that sharing the host netns made it "the
path a MASQUERADEd client takes". False. A MASQUERADEd client is FORWARDED
(netfilter FORWARD + routing); a probe started in the pod is SOCKET-ORIGINATED
(Cilium socket-LB). Different hooks -- sharing a namespace is not sharing a hook.
The `curl --interface wg0` control cited to rule out a host-netns artefact fails
on ALL THREE nodes including ones whose ARP resolves, so it discriminated
nothing. A raw-IP `curl -k https://192.168.2.20/` returning rc=35 is missing SNI,
not unreachability.

Also corrected: new-peer.sh emits AllowedIPs = 0.0.0.0/0 (FULL tunnel) by
default; `--split` is the opt-in. An earlier reading had the branches backwards.

All code is withdrawn -- the nodeSelector exclusion, the node_baseline label
tasks, both alerts, the promtool tests and the KSM allowlist entry are gone.
Net change is three documentation files, purely additive (356 insertions, 0
deletions), recording what was measured so the next person does not redo it.

The reported internet-routing problem is NOT diagnosed here: no client had
connected since the power-cycle (no peer endpoint, 0 packets through FORWARD and
MASQUERADE), so the forwarded path was unmeasurable.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

runbooks/vpn-remote-access.md

@@ -273,6 +273,84 @@ bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
 by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
 if the router is not delivering the packets, and a `handshake` with no completed session if it is.
 
+## Triage — "the client reaches the LAN and cluster apps, but not the internet"
+
+Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom
+almost always lives, and because the server side has been measured and is not the cause.
+
+**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.**
+
+| the profile says | meaning |
+|---|---|
+| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. |
+| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. |
+
+`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit —
+`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client
+profile only**, so an already-issued profile keeps whatever it was rendered with until it is
+re-rendered and re-imported on the device. The two forms differ by one flag:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet>'           # full tunnel
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet> --split'   # LAN only
+```
+
+Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into
+`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is
+editing `AllowedIPs` in the profile **on the device** — the server does not care.
+
+**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the
+client connected, then watch the two counters that only forwarded traffic moves:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c "
+     wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5;
+     iptables -t nat -L POSTROUTING -n -v | head -5"'
+```
+
+* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which
+  node the router's `udp/51820` forward points at (the router is unreadable from the cluster).
+* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is
+  upstream of the cluster (router forward, client, or carrier), not here.
+* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side
+  failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there,
+  which breaks internet while node-local cluster VIPs still work. It has not been observed.
+
+**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly:
+
+* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack
+  client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel —
+  so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4:
+  `curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it,
+  and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is
+  blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it).
+* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified
+  2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a
+  blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on
+  a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all.
+
+**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the
+`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain
+**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route
+`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the
+PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled
+`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the
+**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6).
+
+## Triage — which L2 announcement lease a node holds does NOT matter
+
+Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node
+could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`:
+that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh`
+for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer
+VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend
+is a node-local socket. Full measurements and the two misleading probes
+(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`.
+
+`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a
+separate, pre-existing fault in that service and has nothing to do with the VPN.
+
 ## Revoking a device
 
 Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the