docs(vpn-ingress): withdraw the L2-lease diagnosis — it was falsified by measurement

The claim this branch originally shipped (a node cannot reach a LoadBalancer VIP
whose Cilium L2 announcement lease it holds, so VPN clients inherit that via
hostNetwork MASQUERADE) is FALSE. Re-measured with all 8 l2announce leases on
k3s-prod-1:

  ip neigh show 192.168.2.20   -> INCOMPLETE   (cause present)
  nc -z 192.168.2.20 443       -> OK
  curl --resolve ... :192.168.2.20  -> 200 in 25ms   (effect absent)

Cause present, effect absent. prod-2 and prod-3 return the same 200.

Why it was never going to bite: cilium-dbg status shows
`KubeProxyReplacement: True [eno1 ..., wg0 10.90.0.1]` and
`Masquerading: BPF [eno1, wg0]` -- Cilium's LB datapath is attached to wg0, so a
client packet for a VIP is DNAT'd in BPF BEFORE any neighbour lookup, and the
Gateway backend is a node-local Envoy socket anyway.

The methodological error, which is the part worth keeping: every probe was run
from the wireguard pod on the grounds that sharing the host netns made it "the
path a MASQUERADEd client takes". False. A MASQUERADEd client is FORWARDED
(netfilter FORWARD + routing); a probe started in the pod is SOCKET-ORIGINATED
(Cilium socket-LB). Different hooks -- sharing a namespace is not sharing a hook.
The `curl --interface wg0` control cited to rule out a host-netns artefact fails
on ALL THREE nodes including ones whose ARP resolves, so it discriminated
nothing. A raw-IP `curl -k https://192.168.2.20/` returning rc=35 is missing SNI,
not unreachability.

Also corrected: new-peer.sh emits AllowedIPs = 0.0.0.0/0 (FULL tunnel) by
default; `--split` is the opt-in. An earlier reading had the branches backwards.

All code is withdrawn -- the nodeSelector exclusion, the node_baseline label
tasks, both alerts, the promtool tests and the KSM allowlist entry are gone.
Net change is three documentation files, purely additive (356 insertions, 0
deletions), recording what was measured so the next person does not redo it.

The reported internet-routing problem is NOT diagnosed here: no client had
connected since the power-cycle (no peer endpoint, 0 packets through FORWARD and
MASQUERADE), so the forwarded path was unmeasurable.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

platform/vpn-ingress/README.md

@@ -186,6 +186,86 @@ right way; one was a defect in my own manifest.
 | Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
 | Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
 
+## Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach
+
+A change was proposed (PR #69) on the belief that **a node cannot reach a LoadBalancer VIP whose
+Cilium L2 announcement lease it holds**, and that the VPN node therefore had to be excluded from
+announcing. **That belief is wrong, and the change was withdrawn.** The half of it that is true is
+real but inert, and the distinction matters enough to record, because the wrong version is very
+easy to re-derive from `ip neigh` alone.
+
+Measured with all **eight** `cilium-l2announce-*` leases held by `k3s-prod-1` — the extreme case,
+after the UPS power-cycle — from the host netns of the `wireguard` pod on each node:
+
+| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
+|---|---|---|
+| `ip neigh show 192.168.2.20` | `FAILED` (no lladdr) | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
+| `curl -k https://192.168.2.20/` | **`200`, 18 ms** | `200`, 13–40 ms |
+| `nc -z 192.168.2.20 443` | **OK** | OK |
+| plex `192.168.2.24:32400` | **`200`** | `200` |
+
+So the announcer really does **not** answer its own ARP request — that part of the observation is
+reproducible. It changes nothing, because **no traffic path resolves a LoadBalancer VIP by ARP**:
+
+* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
+  `Masquerading: BPF [eno1, wg0]`. Cilium's LB datapath is attached to **`wg0` as well as `eno1`**,
+  so a remote client's packet arriving on `wg0` for a VIP is DNAT'd to a backend **in BPF, before
+  any route or neighbour lookup for the VIP happens**.
+* `cilium-dbg service list` on `k3s-prod-1` programs `192.168.2.20:443 => 127.0.0.1:13606` — the
+  shared Gateway's backend is a **node-local** Envoy socket. Every node can serve that VIP without
+  a packet crossing the LAN at all.
+* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
+  which is why `curl` succeeds on the lease holder despite `FAILED`. The discriminator: a **real**
+  service port connects, a **bogus** one (`nc -z 192.168.2.20 9999`) fails, on all three nodes —
+  that is service-aware DNAT, not wire traffic.
+
+**Two tests that look like proof and are not.** Both were used to support the withdrawn diagnosis:
+
+* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless. It is also an
+  artefact of probing: VIPs answer no ICMP, so a `ping` leaves a failed entry behind whether or not
+  anything is wrong.
+* **`curl --interface wg0`.** This fails (`rc=28`, timeout) from **all three** nodes, including the
+  two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it
+  discriminates nothing — and it is not the client path either (see below).
+
+**Why neither probe can see the client path.** A MASQUERADEd remote client's packets are
+**forwarded** — netfilter `FORWARD`, then routing — whereas everything runnable from the pod is
+**socket-originated**. The two traverse different hooks, so a host-netns probe cannot measure the
+forwarded path in either direction: it reports success where the forwarded path might fail, and
+`--interface wg0` reports failure where it does not. **The forwarded path can only be measured with
+a client actually connected**, by watching the `FORWARD -i wg0` and `POSTROUTING … MASQUERADE`
+counters move.
+
+## Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses
+
+`new-peer.sh` has emitted a **full tunnel** (`AllowedIPs = 0.0.0.0/0`) **by default since the
+component's first commit**; `--split` is the opt-in for LAN-only. There is no full-tunnel feature
+to add. The server side was checked end to end and is **capable and correct** — if a full-tunnel
+client cannot reach the internet, nothing in the list below is the reason:
+
+| checked | result on all three nodes |
+|---|---|
+| `net.ipv4.ip_forward` (read from `/proc/sys`, never written) | `1` |
+| `iptables -t nat -S POSTROUTING` | carries `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` — NATs to **any** outbound interface, so internet NAT needs no change |
+| `CILIUM_POST_nat`, which precedes it in the chain | **empty, 0 packets** — it cannot short-circuit the MASQUERADE |
+| `iptables -L FORWARD` | `-i wg0` and `-o wg0` ACCEPT both present |
+| default route | `default via 192.168.2.1 dev eno1 onlink`, intact |
+| node's own internet reachability | `http://1.1.1.1/` returns `301` |
+
+**A full tunnel egresses the HOUSE WAN, not PIA.** This was the trap worth checking, and it is
+closed: `platform/vpn-egress/egress-steering.yaml`'s `CiliumEgressGatewayPolicy` selects by
+`podSelector` on `egress-class: vpn`, and this DaemonSet is deliberately labelled
+`egress-class: internet`, so it is not selected. Forwarded `wg0` traffic is not pod-identity
+traffic at all, and `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`) — the egress-gateway
+node. A remote client's apparent public address is therefore the **home WAN IP**, which is the
+intended behaviour and the thing to state when enrolling a device.
+
+**Not measurable without a connected client.** At the time of writing `wg show wg0` listed the one
+enrolled peer with **no endpoint and no handshake** on every node, and both the `FORWARD -i wg0` and
+the `MASQUERADE` counters read **0 packets** — so no client traffic had been forwarded since the
+power-cycle, and which node the router forwards `udp/51820` to was **undeterminable**. Only the
+forwarded node shows a peer with an `endpoint` and a recent `latest handshake`.
+
 ## Prerequisites outside this directory
 
 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`