docs(vpn-ingress): withdraw the L2-lease diagnosis — it was falsified by measurement

The claim this branch originally shipped (a node cannot reach a LoadBalancer VIP
whose Cilium L2 announcement lease it holds, so VPN clients inherit that via
hostNetwork MASQUERADE) is FALSE. Re-measured with all 8 l2announce leases on
k3s-prod-1:

  ip neigh show 192.168.2.20   -> INCOMPLETE   (cause present)
  nc -z 192.168.2.20 443       -> OK
  curl --resolve ... :192.168.2.20  -> 200 in 25ms   (effect absent)

Cause present, effect absent. prod-2 and prod-3 return the same 200.

Why it was never going to bite: cilium-dbg status shows
`KubeProxyReplacement: True [eno1 ..., wg0 10.90.0.1]` and
`Masquerading: BPF [eno1, wg0]` -- Cilium's LB datapath is attached to wg0, so a
client packet for a VIP is DNAT'd in BPF BEFORE any neighbour lookup, and the
Gateway backend is a node-local Envoy socket anyway.

The methodological error, which is the part worth keeping: every probe was run
from the wireguard pod on the grounds that sharing the host netns made it "the
path a MASQUERADEd client takes". False. A MASQUERADEd client is FORWARDED
(netfilter FORWARD + routing); a probe started in the pod is SOCKET-ORIGINATED
(Cilium socket-LB). Different hooks -- sharing a namespace is not sharing a hook.
The `curl --interface wg0` control cited to rule out a host-netns artefact fails
on ALL THREE nodes including ones whose ARP resolves, so it discriminated
nothing. A raw-IP `curl -k https://192.168.2.20/` returning rc=35 is missing SNI,
not unreachability.

Also corrected: new-peer.sh emits AllowedIPs = 0.0.0.0/0 (FULL tunnel) by
default; `--split` is the opt-in. An earlier reading had the branches backwards.

All code is withdrawn -- the nodeSelector exclusion, the node_baseline label
tasks, both alerts, the promtool tests and the KSM allowlist entry are gone.
Net change is three documentation files, purely additive (356 insertions, 0
deletions), recording what was measured so the next person does not redo it.

The reported internet-routing problem is NOT diagnosed here: no client had
connected since the power-cycle (no peer endpoint, 0 packets through FORWARD and
MASQUERADE), so the forwarded path was unmeasurable.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

decisions/2026-10-17-vpn-ingress-l2-lease-collision.md

@@ -0,0 +1,198 @@
+---
+date: 2026-10-17
+branch: fix/vpn-ingress-l2-lease-collision
+title: "WITHDRAWN: a node CAN reach a VIP it announces. My L2-lease diagnosis was wrong"
+---
+
+## What changed
+
+**This branch previously proposed a fix. That fix has been reverted in full. The diagnosis behind it
+was wrong.** What is left is documentation of what was actually measured, so the wrong conclusion is
+not re-derived — it is very easy to re-derive from `ip neigh` alone.
+
+### The claim, and how it died
+
+The earlier version of this entry claimed: *a node cannot reach a LoadBalancer VIP whose Cilium L2
+announcement lease it holds itself* — the announcer does not answer its own ARP request, the
+neighbour entry settles at `FAILED`, and packets to that VIP die before leaving the node. Because
+`platform/vpn-ingress` runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
+own address, a VPN client was said to inherit that. The fix labelled the forwarded node and excluded
+that label from announcing, with two alerts and four promtool tests behind it.
+
+The claim makes a sharp prediction. After today's UPS power-cycle **all eight** `cilium-l2announce-*`
+leases are held by `k3s-prod-1` (gateway, pihole tcp+udp, crafty tcp+udp, unifi tcp+udp, plex) —
+the extreme case. The prediction was therefore that `k3s-prod-1` should reach **none** of the five
+VIPs and the other two nodes should reach **all**.
+
+**The prediction failed.** Measured from the host netns of the `wireguard` pod on each node:
+
+| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
+|---|---|---|
+| `ip neigh show 192.168.2.20` | `FAILED`, no lladdr | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
+| `curl -k https://192.168.2.20/` | **`200` in 18 ms** | `200` in 13–40 ms |
+| `nc -z 192.168.2.20 443` | **OK** | OK |
+| `192.168.2.22:25565`, `.23:53`, `.24:32400` | **all OK / `200`** | all OK / `200` |
+| `192.168.2.21:8443` | FAIL | **FAIL** (both) |
+
+The asserted **cause is still present** today — `FAILED` on the holder, for all five VIPs — while the
+asserted **effect is absent**. That is sufficient to falsify the causal claim without having to
+explain anything else. The ARP asymmetry is real and reproducible; it is also **inert**.
+
+### Why it is inert
+
+No traffic path resolves a LoadBalancer VIP by ARP:
+
+* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
+  `Masquerading: BPF [eno1, wg0]` (Cilium v1.20.1, TCX attach, tunnel/vxlan). Cilium's LB datapath
+  is attached to **`wg0` as well as `eno1`**, so a remote client's packet arriving on `wg0` for a VIP
+  is DNAT'd **in BPF, before any route or neighbour lookup for the VIP**.
+* `cilium-dbg service list` programs `192.168.2.20:443 => 127.0.0.1:13606` — the shared Gateway's
+  backend is a **node-local** Envoy socket, so every node serves that VIP without a LAN packet.
+* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
+  which is why `curl` succeeds on the holder. Discriminator: a real service port connects while a
+  bogus one (`nc -z 192.168.2.20 9999`) fails, identically on all three nodes — service-aware DNAT,
+  not wire traffic.
+
+### The two probes that misled me
+
+Both were cited as evidence in the previous version of this entry.
+
+* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless, and is also a probing
+  artefact: VIPs answer no ICMP, so a `ping` leaves a failed entry whether or not anything is wrong.
+* **`curl --interface wg0`.** The previous entry used this as the control that ruled out a
+  host-netns artefact. It is not a control: it fails (`rc=28`) from **all three** nodes, including
+  the two whose ARP for the VIP resolves perfectly. It is uncorrelated with lease holding, so it
+  discriminates nothing.
+
+**The deeper error:** a MASQUERADEd client's packets are **forwarded** (netfilter `FORWARD`, then
+routing); everything runnable from the pod is **socket-originated**. Different hooks. A host-netns
+probe cannot measure the forwarded path in either direction — it reports success where the forwarded
+path might fail, and `--interface wg0` reports failure where it does not. Sharing a netns is not
+sharing a hook, and that is the assumption the original diagnosis rested on.
+
+### Files
+
+Reverted to `origin/main`: `platform-config/cilium/l2-announcement-policy.yaml`,
+`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`,
+`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`, `platform/monitoring/alerts.yaml`,
+`platform/monitoring/kube-prometheus-stack.yaml`, `runbooks/kustomization.yaml`. Deleted:
+`platform/monitoring/tests/vpn-ingress-l2.test.yaml`, `runbooks/vpn-ingress-l2-collision.md`.
+
+Kept, rewritten from scratch:
+
+* **`platform/vpn-ingress/README.md`** — two measured sections: the L2 table above with why it is
+  inert and which probes lie, and the internet-egress facts below.
+* **`runbooks/vpn-remote-access.md`** — a triage section for the reported symptom ("LAN and cluster
+  work, internet does not") that works outwards from the client profile, and a short section saying
+  the L2 lease is a dead end, with the unifi `.21` note.
+
+**Nothing in `platform/vpn-ingress` or `infra/vpn-ingress` changed behaviourally. No code fix is
+proposed by this branch any more.**
+
+### The internet-routing report: no server-side fix was needed, and none was made
+
+The brief's prior was that `new-peer.sh` renders a split tunnel and that the operator wants a full
+one, to be added as a documented option. **That prior does not match the tree.** `new-peer.sh` has
+emitted `AllowedIPs = 0.0.0.0/0` **by default since the component's first commit** (`f92707f`,
+2026-10-05, confirmed with `git log -S`); `--split` is the opt-in, and
+`runbooks/vpn-remote-access.md:252-253` already documents both forms. So there is no full-tunnel
+feature to add, and adding a `WG_FULL_TUNNEL` knob would be a second switch for a default that
+already exists. **I deliberately shipped no code change here** — writing one would have meant
+inventing a fix for a symptom I could not reproduce (see below).
+
+The server side was measured end to end and is capable and correct, so the cause is in the client
+profile or the client. The triage section names the two candidates that fit the symptom exactly —
+a `--split` profile, and the IPv4-only `AllowedIPs` letting a dual-stack client prefer AAAA and
+leak public traffic outside the tunnel — each with a one-line test that settles it.
+
+## What I verified
+
+Measured live against prod, `kubectl --context k3s-prod --request-timeout=15s`, read-only, on
+2026-10-17:
+
+* **The lease distribution.** All eight `cilium-l2announce-*` leases held by `k3s-prod-1`
+  (`.spec.holderIdentity`). The brief's observation reproduced; it had not moved again.
+* **The falsification, in full**, as tabulated above: 3 nodes x 5 VIPs, `ip neigh` + `nc -z` + a
+  real `curl` for each. The holder reaches four of five VIPs; `.21` fails from everywhere.
+* **The socket-LB discriminator.** Real service port OK, bogus port FAIL, identically on all three
+  nodes — so the successes are service-aware DNAT, not wire delivery.
+* **`curl --interface wg0` fails on all three nodes**, including non-announcing ones with
+  `REACHABLE` neighbour entries. This is the measurement that destroys the original control.
+* **The datapath facts** quoted above from `cilium-dbg status --verbose` and
+  `cilium-dbg service list` on `cilium-55jv8` (`k3s-prod-1`), plus `cilium-config`:
+  `kube-proxy-replacement=true`, `enable-l2-announcements=true`, `enable-bpf-masquerade=true`,
+  `routing-mode=tunnel`, `devices=` (empty, auto-detected). Cilium `v1.20.1` on all three.
+* **The VIP is not a local address and routes on-link.** `ip route get 192.168.2.20` →
+  `dev eno1 src 192.168.2.10` on the holder; nothing in `table local`; `ip -4 addr` shows no VIP.
+  So the 200 is not loopback delivery by the host stack — it is BPF.
+* **The internet path, server side, on all three nodes:** `/proc/sys/net/ipv4/ip_forward` = `1`
+  (read, never written); `iptables -t nat -S POSTROUTING` carries
+  `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE`; the preceding `CILIUM_POST_nat` chain is **empty with 0
+  packets**, so it cannot short-circuit that rule; `FORWARD -i wg0` and `-o wg0` ACCEPT both
+  present; `default via 192.168.2.1 dev eno1 onlink` intact; `ip route get 1.1.1.1` resolves via it;
+  `curl http://1.1.1.1/` returns `301` from each node.
+* **The PIA / egress-gateway answer.** `platform/vpn-egress/egress-steering.yaml` selects by
+  `podSelector` on `egress-class: vpn`; `platform/vpn-ingress/daemonset.yaml:130` labels this pod
+  `egress-class: internet`; `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`), absent on
+  `-1` and `-2`. So a full tunnel egresses the **house WAN**, and a client's apparent address is the
+  home WAN IP, not PIA's.
+* **PiHole resolves public names** — `nslookup google.com 192.168.2.23` from the pod returned
+  `142.250.113.138`.
+* **No client is connected, and none has been since the power-cycle.** `wg show wg0` lists the one
+  enrolled peer (`10.90.0.2/32`) with **no endpoint and no handshake** on all three nodes, and both
+  `FORWARD -i wg0` and the `MASQUERADE` rule read **0 packets / 0 bytes** on all three. The
+  `wireguard` pods restarted 153 min ago (2 restarts, 11d old), which reset the counters.
+* **The full-tunnel default's age**, from `git log -S` on `new-peer.sh`: introduced in `f92707f`
+  (2026-10-05) and never changed.
+* **The branch rebased cleanly** onto `origin/main` (`1a0b208`, PR #70 merged) with no conflicts.
+
+## What I did NOT verify
+
+* **THE FORWARDED PATH IS STILL UNMEASURED, in both directions.** No client is connected, so I could
+  not test what a real remote client does, and — as established above — **no probe runnable from the
+  pod can substitute for it**. Specifically I did **not** prove that a forwarded client packet to a
+  VIP is DNAT'd by Cilium's `wg0` program. That inference rests on `cilium-dbg` listing `wg0` as a
+  LB device plus the node-local Gateway backend, not on an observed client packet. **It is the one
+  remaining way the withdrawn diagnosis could be partly right** — if `wg0` BPF were somehow not in
+  the forwarded path, forwarded traffic would need ARP and would die on the holder. I judge this
+  unlikely but it is unproven. *To close it: connect a client, then from the node whose `wg show`
+  shows the peer endpoint, watch `FORWARD -i wg0` and the `MASQUERADE` counters move while the
+  client loads a cluster app — and repeat it on a node that holds leases and one that does not.*
+* **WHICH NODE THE ROUTER FORWARDS `udp/51820` TO IS UNDETERMINABLE RIGHT NOW**, for the same
+  reason. **I need a client connected** to learn it; the router is unreadable from the cluster.
+* **I cannot reproduce or explain the 2026-10-08 measurements.** That entry recorded `nc -z` TCP
+  failures correlated with lease holding across 15 node x VIP pairs. Today the same test passes.
+  My leading hypothesis is **stale or incorrectly-programmed BPF LB state on `k3s-prod-1`**,
+  rebuilt by today's full power-cycle — which would make the lease correlation coincidental. A
+  competing hypothesis is that Cilium had not auto-detected `wg0` then (`devices` is empty, and
+  `wg0` is created by this DaemonSet after Cilium starts). **Both are unproven, and neither is
+  needed for the falsification** — the asserted cause is present today without its effect. I did not
+  retro-measure anything, and I did not read Cilium's source or issue tracker.
+* **The operator's report cannot be a description of "right now."** No client has connected since
+  the power-cycle (0 forwarded packets, no handshake on any node), so "clients can reach cluster
+  services and the LAN but not the internet" must describe a session before the power-cycle or one
+  that has since dropped. **I did not establish when it was observed, or with which profile.**
+* **THE INTERNET ROOT CAUSE IS NOT DIAGNOSED.** I proved the server side is *capable*; I did **not**
+  find the fault, and I shipped no fix for it. The two candidates in the triage section (a
+  `--split` profile; IPv6 preference leaking outside an IPv4-only `AllowedIPs`) are **reasoned, not
+  measured**. *What I need:* the `AllowedIPs` line from the client's own profile — that one line
+  decides split vs full and halves the search — and a connected client for step 2.
+* **Whether `--split` was in fact used for the enrolled peer.** The profile holds a private key and
+  lives only on the operator's machine and the device; it is not in the repo, and I did not look for
+  it. Server-side `AllowedIPs` is a `/32` source filter and says nothing about the client's routing.
+* **Whether `MTU = 1420` holds on a full-tunnel WAN path.** Not measured, and not measurable without
+  a client. It is the standard 1500 − 60; a client behind PPPoE or CGNAT with a smaller path MTU
+  would see large transfers hang while small requests succeed. The runbook's existing MTU note
+  (drop to 1280) already covers the remedy.
+* **Adding `::/0` to `AllowedIPs` is untested advice.** The triage section offers it to blackhole
+  IPv6 rather than leak it, and states that the tunnel has no IPv6 egress. I did not try it on a
+  client, and I do not know how each WireGuard client behaves when handed an IPv6 route on an
+  interface with no IPv6 address.
+* **`192.168.2.21:8443` (unifi) fails from all three nodes**, as it did on 2026-10-08 — a separate,
+  pre-existing fault. Not investigated, not addressed, and now recorded in the runbook as such.
+* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so the gate proves the L0/L1
+  offline subset only. Nothing above is covered by a test that runs in CI.
+* **Nothing was applied, and nothing needs to be.** The branch is now documentation-only, so there
+  is no Flux reconcile, Ansible run or `kubectl` step to perform after merge. I ran no mutating
+  `kubectl`, `flux` or Ansible, and did not run `new-peer.sh` (it mints key material).
+* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).