diff --git a/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md b/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md new file mode 100644 index 0000000..e0923b8 --- /dev/null +++ b/decisions/2026-10-17-vpn-ingress-l2-lease-collision.md @@ -0,0 +1,198 @@ +--- +date: 2026-10-17 +branch: fix/vpn-ingress-l2-lease-collision +title: "WITHDRAWN: a node CAN reach a VIP it announces. My L2-lease diagnosis was wrong" +--- + +## What changed + +**This branch previously proposed a fix. That fix has been reverted in full. The diagnosis behind it +was wrong.** What is left is documentation of what was actually measured, so the wrong conclusion is +not re-derived — it is very easy to re-derive from `ip neigh` alone. + +### The claim, and how it died + +The earlier version of this entry claimed: *a node cannot reach a LoadBalancer VIP whose Cilium L2 +announcement lease it holds itself* — the announcer does not answer its own ARP request, the +neighbour entry settles at `FAILED`, and packets to that VIP die before leaving the node. Because +`platform/vpn-ingress` runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's +own address, a VPN client was said to inherit that. The fix labelled the forwarded node and excluded +that label from announcing, with two alerts and four promtool tests behind it. + +The claim makes a sharp prediction. After today's UPS power-cycle **all eight** `cilium-l2announce-*` +leases are held by `k3s-prod-1` (gateway, pihole tcp+udp, crafty tcp+udp, unifi tcp+udp, plex) — +the extreme case. The prediction was therefore that `k3s-prod-1` should reach **none** of the five +VIPs and the other two nodes should reach **all**. + +**The prediction failed.** Measured from the host netns of the `wireguard` pod on each node: + +| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) | +|---|---|---| +| `ip neigh show 192.168.2.20` | `FAILED`, no lladdr | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` | +| `curl -k https://192.168.2.20/` | **`200` in 18 ms** | `200` in 13–40 ms | +| `nc -z 192.168.2.20 443` | **OK** | OK | +| `192.168.2.22:25565`, `.23:53`, `.24:32400` | **all OK / `200`** | all OK / `200` | +| `192.168.2.21:8443` | FAIL | **FAIL** (both) | + +The asserted **cause is still present** today — `FAILED` on the holder, for all five VIPs — while the +asserted **effect is absent**. That is sufficient to falsify the causal claim without having to +explain anything else. The ARP asymmetry is real and reproducible; it is also **inert**. + +### Why it is inert + +No traffic path resolves a LoadBalancer VIP by ARP: + +* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and + `Masquerading: BPF [eno1, wg0]` (Cilium v1.20.1, TCX attach, tunnel/vxlan). Cilium's LB datapath + is attached to **`wg0` as well as `eno1`**, so a remote client's packet arriving on `wg0` for a VIP + is DNAT'd **in BPF, before any route or neighbour lookup for the VIP**. +* `cilium-dbg service list` programs `192.168.2.20:443 => 127.0.0.1:13606` — the shared Gateway's + backend is a **node-local** Envoy socket, so every node serves that VIP without a LAN packet. +* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`), + which is why `curl` succeeds on the holder. Discriminator: a real service port connects while a + bogus one (`nc -z 192.168.2.20 9999`) fails, identically on all three nodes — service-aware DNAT, + not wire traffic. + +### The two probes that misled me + +Both were cited as evidence in the previous version of this entry. + +* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless, and is also a probing + artefact: VIPs answer no ICMP, so a `ping` leaves a failed entry whether or not anything is wrong. +* **`curl --interface wg0`.** The previous entry used this as the control that ruled out a + host-netns artefact. It is not a control: it fails (`rc=28`) from **all three** nodes, including + the two whose ARP for the VIP resolves perfectly. It is uncorrelated with lease holding, so it + discriminates nothing. + +**The deeper error:** a MASQUERADEd client's packets are **forwarded** (netfilter `FORWARD`, then +routing); everything runnable from the pod is **socket-originated**. Different hooks. A host-netns +probe cannot measure the forwarded path in either direction — it reports success where the forwarded +path might fail, and `--interface wg0` reports failure where it does not. Sharing a netns is not +sharing a hook, and that is the assumption the original diagnosis rested on. + +### Files + +Reverted to `origin/main`: `platform-config/cilium/l2-announcement-policy.yaml`, +`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`, +`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`, `platform/monitoring/alerts.yaml`, +`platform/monitoring/kube-prometheus-stack.yaml`, `runbooks/kustomization.yaml`. Deleted: +`platform/monitoring/tests/vpn-ingress-l2.test.yaml`, `runbooks/vpn-ingress-l2-collision.md`. + +Kept, rewritten from scratch: + +* **`platform/vpn-ingress/README.md`** — two measured sections: the L2 table above with why it is + inert and which probes lie, and the internet-egress facts below. +* **`runbooks/vpn-remote-access.md`** — a triage section for the reported symptom ("LAN and cluster + work, internet does not") that works outwards from the client profile, and a short section saying + the L2 lease is a dead end, with the unifi `.21` note. + +**Nothing in `platform/vpn-ingress` or `infra/vpn-ingress` changed behaviourally. No code fix is +proposed by this branch any more.** + +### The internet-routing report: no server-side fix was needed, and none was made + +The brief's prior was that `new-peer.sh` renders a split tunnel and that the operator wants a full +one, to be added as a documented option. **That prior does not match the tree.** `new-peer.sh` has +emitted `AllowedIPs = 0.0.0.0/0` **by default since the component's first commit** (`f92707f`, +2026-10-05, confirmed with `git log -S`); `--split` is the opt-in, and +`runbooks/vpn-remote-access.md:252-253` already documents both forms. So there is no full-tunnel +feature to add, and adding a `WG_FULL_TUNNEL` knob would be a second switch for a default that +already exists. **I deliberately shipped no code change here** — writing one would have meant +inventing a fix for a symptom I could not reproduce (see below). + +The server side was measured end to end and is capable and correct, so the cause is in the client +profile or the client. The triage section names the two candidates that fit the symptom exactly — +a `--split` profile, and the IPv4-only `AllowedIPs` letting a dual-stack client prefer AAAA and +leak public traffic outside the tunnel — each with a one-line test that settles it. + +## What I verified + +Measured live against prod, `kubectl --context k3s-prod --request-timeout=15s`, read-only, on +2026-10-17: + +* **The lease distribution.** All eight `cilium-l2announce-*` leases held by `k3s-prod-1` + (`.spec.holderIdentity`). The brief's observation reproduced; it had not moved again. +* **The falsification, in full**, as tabulated above: 3 nodes x 5 VIPs, `ip neigh` + `nc -z` + a + real `curl` for each. The holder reaches four of five VIPs; `.21` fails from everywhere. +* **The socket-LB discriminator.** Real service port OK, bogus port FAIL, identically on all three + nodes — so the successes are service-aware DNAT, not wire delivery. +* **`curl --interface wg0` fails on all three nodes**, including non-announcing ones with + `REACHABLE` neighbour entries. This is the measurement that destroys the original control. +* **The datapath facts** quoted above from `cilium-dbg status --verbose` and + `cilium-dbg service list` on `cilium-55jv8` (`k3s-prod-1`), plus `cilium-config`: + `kube-proxy-replacement=true`, `enable-l2-announcements=true`, `enable-bpf-masquerade=true`, + `routing-mode=tunnel`, `devices=` (empty, auto-detected). Cilium `v1.20.1` on all three. +* **The VIP is not a local address and routes on-link.** `ip route get 192.168.2.20` → + `dev eno1 src 192.168.2.10` on the holder; nothing in `table local`; `ip -4 addr` shows no VIP. + So the 200 is not loopback delivery by the host stack — it is BPF. +* **The internet path, server side, on all three nodes:** `/proc/sys/net/ipv4/ip_forward` = `1` + (read, never written); `iptables -t nat -S POSTROUTING` carries + `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE`; the preceding `CILIUM_POST_nat` chain is **empty with 0 + packets**, so it cannot short-circuit that rule; `FORWARD -i wg0` and `-o wg0` ACCEPT both + present; `default via 192.168.2.1 dev eno1 onlink` intact; `ip route get 1.1.1.1` resolves via it; + `curl http://1.1.1.1/` returns `301` from each node. +* **The PIA / egress-gateway answer.** `platform/vpn-egress/egress-steering.yaml` selects by + `podSelector` on `egress-class: vpn`; `platform/vpn-ingress/daemonset.yaml:130` labels this pod + `egress-class: internet`; `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`), absent on + `-1` and `-2`. So a full tunnel egresses the **house WAN**, and a client's apparent address is the + home WAN IP, not PIA's. +* **PiHole resolves public names** — `nslookup google.com 192.168.2.23` from the pod returned + `142.250.113.138`. +* **No client is connected, and none has been since the power-cycle.** `wg show wg0` lists the one + enrolled peer (`10.90.0.2/32`) with **no endpoint and no handshake** on all three nodes, and both + `FORWARD -i wg0` and the `MASQUERADE` rule read **0 packets / 0 bytes** on all three. The + `wireguard` pods restarted 153 min ago (2 restarts, 11d old), which reset the counters. +* **The full-tunnel default's age**, from `git log -S` on `new-peer.sh`: introduced in `f92707f` + (2026-10-05) and never changed. +* **The branch rebased cleanly** onto `origin/main` (`1a0b208`, PR #70 merged) with no conflicts. + +## What I did NOT verify + +* **THE FORWARDED PATH IS STILL UNMEASURED, in both directions.** No client is connected, so I could + not test what a real remote client does, and — as established above — **no probe runnable from the + pod can substitute for it**. Specifically I did **not** prove that a forwarded client packet to a + VIP is DNAT'd by Cilium's `wg0` program. That inference rests on `cilium-dbg` listing `wg0` as a + LB device plus the node-local Gateway backend, not on an observed client packet. **It is the one + remaining way the withdrawn diagnosis could be partly right** — if `wg0` BPF were somehow not in + the forwarded path, forwarded traffic would need ARP and would die on the holder. I judge this + unlikely but it is unproven. *To close it: connect a client, then from the node whose `wg show` + shows the peer endpoint, watch `FORWARD -i wg0` and the `MASQUERADE` counters move while the + client loads a cluster app — and repeat it on a node that holds leases and one that does not.* +* **WHICH NODE THE ROUTER FORWARDS `udp/51820` TO IS UNDETERMINABLE RIGHT NOW**, for the same + reason. **I need a client connected** to learn it; the router is unreadable from the cluster. +* **I cannot reproduce or explain the 2026-10-08 measurements.** That entry recorded `nc -z` TCP + failures correlated with lease holding across 15 node x VIP pairs. Today the same test passes. + My leading hypothesis is **stale or incorrectly-programmed BPF LB state on `k3s-prod-1`**, + rebuilt by today's full power-cycle — which would make the lease correlation coincidental. A + competing hypothesis is that Cilium had not auto-detected `wg0` then (`devices` is empty, and + `wg0` is created by this DaemonSet after Cilium starts). **Both are unproven, and neither is + needed for the falsification** — the asserted cause is present today without its effect. I did not + retro-measure anything, and I did not read Cilium's source or issue tracker. +* **The operator's report cannot be a description of "right now."** No client has connected since + the power-cycle (0 forwarded packets, no handshake on any node), so "clients can reach cluster + services and the LAN but not the internet" must describe a session before the power-cycle or one + that has since dropped. **I did not establish when it was observed, or with which profile.** +* **THE INTERNET ROOT CAUSE IS NOT DIAGNOSED.** I proved the server side is *capable*; I did **not** + find the fault, and I shipped no fix for it. The two candidates in the triage section (a + `--split` profile; IPv6 preference leaking outside an IPv4-only `AllowedIPs`) are **reasoned, not + measured**. *What I need:* the `AllowedIPs` line from the client's own profile — that one line + decides split vs full and halves the search — and a connected client for step 2. +* **Whether `--split` was in fact used for the enrolled peer.** The profile holds a private key and + lives only on the operator's machine and the device; it is not in the repo, and I did not look for + it. Server-side `AllowedIPs` is a `/32` source filter and says nothing about the client's routing. +* **Whether `MTU = 1420` holds on a full-tunnel WAN path.** Not measured, and not measurable without + a client. It is the standard 1500 − 60; a client behind PPPoE or CGNAT with a smaller path MTU + would see large transfers hang while small requests succeed. The runbook's existing MTU note + (drop to 1280) already covers the remedy. +* **Adding `::/0` to `AllowedIPs` is untested advice.** The triage section offers it to blackhole + IPv6 rather than leak it, and states that the tunnel has no IPv6 egress. I did not try it on a + client, and I do not know how each WireGuard client behaves when handed an IPv6 route on an + interface with no IPv6 address. +* **`192.168.2.21:8443` (unifi) fails from all three nodes**, as it did on 2026-10-08 — a separate, + pre-existing fault. Not investigated, not addressed, and now recorded in the runbook as such. +* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so the gate proves the L0/L1 + offline subset only. Nothing above is covered by a test that runs in CI. +* **Nothing was applied, and nothing needs to be.** The branch is now documentation-only, so there + is no Flux reconcile, Ansible run or `kubectl` step to perform after merge. I ran no mutating + `kubectl`, `flux` or Ansible, and did not run `new-peer.sh` (it mints key material). +* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b). diff --git a/platform/vpn-ingress/README.md b/platform/vpn-ingress/README.md index 3cfc916..b0598be 100644 --- a/platform/vpn-ingress/README.md +++ b/platform/vpn-ingress/README.md @@ -186,6 +186,86 @@ right way; one was a defect in my own manifest. | Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. | | Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. | +## Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach + +A change was proposed (PR #69) on the belief that **a node cannot reach a LoadBalancer VIP whose +Cilium L2 announcement lease it holds**, and that the VPN node therefore had to be excluded from +announcing. **That belief is wrong, and the change was withdrawn.** The half of it that is true is +real but inert, and the distinction matters enough to record, because the wrong version is very +easy to re-derive from `ip neigh` alone. + +Measured with all **eight** `cilium-l2announce-*` leases held by `k3s-prod-1` — the extreme case, +after the UPS power-cycle — from the host netns of the `wireguard` pod on each node: + +| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) | +|---|---|---| +| `ip neigh show 192.168.2.20` | `FAILED` (no lladdr) | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` | +| `curl -k https://192.168.2.20/` | **`200`, 18 ms** | `200`, 13–40 ms | +| `nc -z 192.168.2.20 443` | **OK** | OK | +| plex `192.168.2.24:32400` | **`200`** | `200` | + +So the announcer really does **not** answer its own ARP request — that part of the observation is +reproducible. It changes nothing, because **no traffic path resolves a LoadBalancer VIP by ARP**: + +* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and + `Masquerading: BPF [eno1, wg0]`. Cilium's LB datapath is attached to **`wg0` as well as `eno1`**, + so a remote client's packet arriving on `wg0` for a VIP is DNAT'd to a backend **in BPF, before + any route or neighbour lookup for the VIP happens**. +* `cilium-dbg service list` on `k3s-prod-1` programs `192.168.2.20:443 => 127.0.0.1:13606` — the + shared Gateway's backend is a **node-local** Envoy socket. Every node can serve that VIP without + a packet crossing the LAN at all. +* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`), + which is why `curl` succeeds on the lease holder despite `FAILED`. The discriminator: a **real** + service port connects, a **bogus** one (`nc -z 192.168.2.20 9999`) fails, on all three nodes — + that is service-aware DNAT, not wire traffic. + +**Two tests that look like proof and are not.** Both were used to support the withdrawn diagnosis: + +* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless. It is also an + artefact of probing: VIPs answer no ICMP, so a `ping` leaves a failed entry behind whether or not + anything is wrong. +* **`curl --interface wg0`.** This fails (`rc=28`, timeout) from **all three** nodes, including the + two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it + discriminates nothing — and it is not the client path either (see below). + +**Why neither probe can see the client path.** A MASQUERADEd remote client's packets are +**forwarded** — netfilter `FORWARD`, then routing — whereas everything runnable from the pod is +**socket-originated**. The two traverse different hooks, so a host-netns probe cannot measure the +forwarded path in either direction: it reports success where the forwarded path might fail, and +`--interface wg0` reports failure where it does not. **The forwarded path can only be measured with +a client actually connected**, by watching the `FORWARD -i wg0` and `POSTROUTING … MASQUERADE` +counters move. + +## Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses + +`new-peer.sh` has emitted a **full tunnel** (`AllowedIPs = 0.0.0.0/0`) **by default since the +component's first commit**; `--split` is the opt-in for LAN-only. There is no full-tunnel feature +to add. The server side was checked end to end and is **capable and correct** — if a full-tunnel +client cannot reach the internet, nothing in the list below is the reason: + +| checked | result on all three nodes | +|---|---| +| `net.ipv4.ip_forward` (read from `/proc/sys`, never written) | `1` | +| `iptables -t nat -S POSTROUTING` | carries `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` — NATs to **any** outbound interface, so internet NAT needs no change | +| `CILIUM_POST_nat`, which precedes it in the chain | **empty, 0 packets** — it cannot short-circuit the MASQUERADE | +| `iptables -L FORWARD` | `-i wg0` and `-o wg0` ACCEPT both present | +| default route | `default via 192.168.2.1 dev eno1 onlink`, intact | +| node's own internet reachability | `http://1.1.1.1/` returns `301` | + +**A full tunnel egresses the HOUSE WAN, not PIA.** This was the trap worth checking, and it is +closed: `platform/vpn-egress/egress-steering.yaml`'s `CiliumEgressGatewayPolicy` selects by +`podSelector` on `egress-class: vpn`, and this DaemonSet is deliberately labelled +`egress-class: internet`, so it is not selected. Forwarded `wg0` traffic is not pod-identity +traffic at all, and `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`) — the egress-gateway +node. A remote client's apparent public address is therefore the **home WAN IP**, which is the +intended behaviour and the thing to state when enrolling a device. + +**Not measurable without a connected client.** At the time of writing `wg show wg0` listed the one +enrolled peer with **no endpoint and no handshake** on every node, and both the `FORWARD -i wg0` and +the `MASQUERADE` counters read **0 packets** — so no client traffic had been forwarded since the +power-cycle, and which node the router forwards `udp/51820` to was **undeterminable**. Only the +forwarded node shows a peer with an `endpoint` and a recent `latest handshake`. + ## Prerequisites outside this directory 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules` diff --git a/runbooks/vpn-remote-access.md b/runbooks/vpn-remote-access.md index 9f123ab..cba1887 100644 --- a/runbooks/vpn-remote-access.md +++ b/runbooks/vpn-remote-access.md @@ -273,6 +273,84 @@ bytes of WireGuard encapsulation); drop the client to `1280` and re-test. by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes if the router is not delivering the packets, and a `handshake` with no completed session if it is. +## Triage — "the client reaches the LAN and cluster apps, but not the internet" + +Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom +almost always lives, and because the server side has been measured and is not the cause. + +**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.** + +| the profile says | meaning | +|---|---| +| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. | +| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. | + +`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit — +`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client +profile only**, so an already-issued profile keeps whatever it was rendered with until it is +re-rendered and re-imported on the device. The two forms differ by one flag: + +```bash +./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh ' # full tunnel +./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh --split' # LAN only +``` + +Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into +`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is +editing `AllowedIPs` in the profile **on the device** — the server does not care. + +**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the +client connected, then watch the two counters that only forwarded traffic moves: + +```bash +./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c " + wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5; + iptables -t nat -L POSTROUTING -n -v | head -5"' +``` + +* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which + node the router's `udp/51820` forward points at (the router is unreadable from the cluster). +* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is + upstream of the cluster (router forward, client, or carrier), not here. +* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side + failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there, + which breaks internet while node-local cluster VIPs still work. It has not been observed. + +**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly: + +* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack + client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel — + so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4: + `curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it, + and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is + blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it). +* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified + 2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a + blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on + a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all. + +**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the +`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain +**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route +`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the +PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled +`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the +**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6). + +## Triage — which L2 announcement lease a node holds does NOT matter + +Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node +could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`: +that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh` +for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer +VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend +is a node-local socket. Full measurements and the two misleading probes +(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`. + +`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a +separate, pre-existing fault in that service and has nothing to do with the VPN. + ## Revoking a device Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the