Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(vpn-ingress): withdraw the L2-lease diagnosis — it was falsified by measurement #69

Closed
gitea-admin wants to merge 1 commits from fix/vpn-ingress-l2-lease-collision into main
pull from: fix/vpn-ingress-l2-lease-collision
merge into: Home-Network:main
Conversation 1 Commits 1 Files Changed 3
+356
3 changed files with 356 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

decisions/2026-10-17-vpn-ingress-l2-lease-collision.md
+198
View File
@@ -0,0 +1,198 @@
---
date: 2026-10-17
branch: fix/vpn-ingress-l2-lease-collision
title: "WITHDRAWN: a node CAN reach a VIP it announces. My L2-lease diagnosis was wrong"
---
## What changed
**This branch previously proposed a fix. That fix has been reverted in full. The diagnosis behind it
was wrong.** What is left is documentation of what was actually measured, so the wrong conclusion is
not re-derived — it is very easy to re-derive from `ip neigh` alone.
### The claim, and how it died
The earlier version of this entry claimed: *a node cannot reach a LoadBalancer VIP whose Cilium L2
announcement lease it holds itself* — the announcer does not answer its own ARP request, the
neighbour entry settles at `FAILED`, and packets to that VIP die before leaving the node. Because
`platform/vpn-ingress` runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
own address, a VPN client was said to inherit that. The fix labelled the forwarded node and excluded
that label from announcing, with two alerts and four promtool tests behind it.
The claim makes a sharp prediction. After today's UPS power-cycle **all eight** `cilium-l2announce-*`
leases are held by `k3s-prod-1` (gateway, pihole tcp+udp, crafty tcp+udp, unifi tcp+udp, plex) —
the extreme case. The prediction was therefore that `k3s-prod-1` should reach **none** of the five
VIPs and the other two nodes should reach **all**.
**The prediction failed.** Measured from the host netns of the `wireguard` pod on each node:
| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
|---|---|---|
| `ip neigh show 192.168.2.20` | `FAILED`, no lladdr | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
| `curl -k https://192.168.2.20/` | **`200` in 18 ms** | `200` in 13–40 ms |
| `nc -z 192.168.2.20 443` | **OK** | OK |
| `192.168.2.22:25565`, `.23:53`, `.24:32400` | **all OK / `200`** | all OK / `200` |
| `192.168.2.21:8443` | FAIL | **FAIL** (both) |
The asserted **cause is still present** today — `FAILED` on the holder, for all five VIPs — while the
asserted **effect is absent**. That is sufficient to falsify the causal claim without having to
explain anything else. The ARP asymmetry is real and reproducible; it is also **inert**.
### Why it is inert
No traffic path resolves a LoadBalancer VIP by ARP:
* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
`Masquerading: BPF [eno1, wg0]` (Cilium v1.20.1, TCX attach, tunnel/vxlan). Cilium's LB datapath
is attached to **`wg0` as well as `eno1`**, so a remote client's packet arriving on `wg0` for a VIP
is DNAT'd **in BPF, before any route or neighbour lookup for the VIP**.
* `cilium-dbg service list` programs `192.168.2.20:443 => 127.0.0.1:13606` — the shared Gateway's
backend is a **node-local** Envoy socket, so every node serves that VIP without a LAN packet.
* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
which is why `curl` succeeds on the holder. Discriminator: a real service port connects while a
bogus one (`nc -z 192.168.2.20 9999`) fails, identically on all three nodes — service-aware DNAT,
not wire traffic.
### The two probes that misled me
Both were cited as evidence in the previous version of this entry.
* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless, and is also a probing
artefact: VIPs answer no ICMP, so a `ping` leaves a failed entry whether or not anything is wrong.
* **`curl --interface wg0`.** The previous entry used this as the control that ruled out a
host-netns artefact. It is not a control: it fails (`rc=28`) from **all three** nodes, including
the two whose ARP for the VIP resolves perfectly. It is uncorrelated with lease holding, so it
discriminates nothing.
**The deeper error:** a MASQUERADEd client's packets are **forwarded** (netfilter `FORWARD`, then
routing); everything runnable from the pod is **socket-originated**. Different hooks. A host-netns
probe cannot measure the forwarded path in either direction — it reports success where the forwarded
path might fail, and `--interface wg0` reports failure where it does not. Sharing a netns is not
sharing a hook, and that is the assumption the original diagnosis rested on.
### Files
Reverted to `origin/main`: `platform-config/cilium/l2-announcement-policy.yaml`,
`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`,
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`, `platform/monitoring/alerts.yaml`,
`platform/monitoring/kube-prometheus-stack.yaml`, `runbooks/kustomization.yaml`. Deleted:
`platform/monitoring/tests/vpn-ingress-l2.test.yaml`, `runbooks/vpn-ingress-l2-collision.md`.
Kept, rewritten from scratch:
* **`platform/vpn-ingress/README.md`** — two measured sections: the L2 table above with why it is
inert and which probes lie, and the internet-egress facts below.
* **`runbooks/vpn-remote-access.md`** — a triage section for the reported symptom ("LAN and cluster
work, internet does not") that works outwards from the client profile, and a short section saying
the L2 lease is a dead end, with the unifi `.21` note.
**Nothing in `platform/vpn-ingress` or `infra/vpn-ingress` changed behaviourally. No code fix is
proposed by this branch any more.**
### The internet-routing report: no server-side fix was needed, and none was made
The brief's prior was that `new-peer.sh` renders a split tunnel and that the operator wants a full
one, to be added as a documented option. **That prior does not match the tree.** `new-peer.sh` has
emitted `AllowedIPs = 0.0.0.0/0` **by default since the component's first commit** (`f92707f`,
2026-10-05, confirmed with `git log -S`); `--split` is the opt-in, and
`runbooks/vpn-remote-access.md:252-253` already documents both forms. So there is no full-tunnel
feature to add, and adding a `WG_FULL_TUNNEL` knob would be a second switch for a default that
already exists. **I deliberately shipped no code change here** — writing one would have meant
inventing a fix for a symptom I could not reproduce (see below).
The server side was measured end to end and is capable and correct, so the cause is in the client
profile or the client. The triage section names the two candidates that fit the symptom exactly —
a `--split` profile, and the IPv4-only `AllowedIPs` letting a dual-stack client prefer AAAA and
leak public traffic outside the tunnel — each with a one-line test that settles it.
## What I verified
Measured live against prod, `kubectl --context k3s-prod --request-timeout=15s`, read-only, on
2026-10-17:
* **The lease distribution.** All eight `cilium-l2announce-*` leases held by `k3s-prod-1`
(`.spec.holderIdentity`). The brief's observation reproduced; it had not moved again.
* **The falsification, in full**, as tabulated above: 3 nodes x 5 VIPs, `ip neigh` + `nc -z` + a
real `curl` for each. The holder reaches four of five VIPs; `.21` fails from everywhere.
* **The socket-LB discriminator.** Real service port OK, bogus port FAIL, identically on all three
nodes — so the successes are service-aware DNAT, not wire delivery.
* **`curl --interface wg0` fails on all three nodes**, including non-announcing ones with
`REACHABLE` neighbour entries. This is the measurement that destroys the original control.
* **The datapath facts** quoted above from `cilium-dbg status --verbose` and
`cilium-dbg service list` on `cilium-55jv8` (`k3s-prod-1`), plus `cilium-config`:
`kube-proxy-replacement=true`, `enable-l2-announcements=true`, `enable-bpf-masquerade=true`,
`routing-mode=tunnel`, `devices=` (empty, auto-detected). Cilium `v1.20.1` on all three.
* **The VIP is not a local address and routes on-link.** `ip route get 192.168.2.20` →
`dev eno1 src 192.168.2.10` on the holder; nothing in `table local`; `ip -4 addr` shows no VIP.
So the 200 is not loopback delivery by the host stack — it is BPF.
* **The internet path, server side, on all three nodes:** `/proc/sys/net/ipv4/ip_forward` = `1`
(read, never written); `iptables -t nat -S POSTROUTING` carries
`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE`; the preceding `CILIUM_POST_nat` chain is **empty with 0
packets**, so it cannot short-circuit that rule; `FORWARD -i wg0` and `-o wg0` ACCEPT both
present; `default via 192.168.2.1 dev eno1 onlink` intact; `ip route get 1.1.1.1` resolves via it;
`curl http://1.1.1.1/` returns `301` from each node.
* **The PIA / egress-gateway answer.** `platform/vpn-egress/egress-steering.yaml` selects by
`podSelector` on `egress-class: vpn`; `platform/vpn-ingress/daemonset.yaml:130` labels this pod
`egress-class: internet`; `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`), absent on
`-1` and `-2`. So a full tunnel egresses the **house WAN**, and a client's apparent address is the
home WAN IP, not PIA's.
* **PiHole resolves public names** — `nslookup google.com 192.168.2.23` from the pod returned
`142.250.113.138`.
* **No client is connected, and none has been since the power-cycle.** `wg show wg0` lists the one
enrolled peer (`10.90.0.2/32`) with **no endpoint and no handshake** on all three nodes, and both
`FORWARD -i wg0` and the `MASQUERADE` rule read **0 packets / 0 bytes** on all three. The
`wireguard` pods restarted 153 min ago (2 restarts, 11d old), which reset the counters.
* **The full-tunnel default's age**, from `git log -S` on `new-peer.sh`: introduced in `f92707f`
(2026-10-05) and never changed.
* **The branch rebased cleanly** onto `origin/main` (`1a0b208`, PR #70 merged) with no conflicts.
## What I did NOT verify
* **THE FORWARDED PATH IS STILL UNMEASURED, in both directions.** No client is connected, so I could
not test what a real remote client does, and — as established above — **no probe runnable from the
pod can substitute for it**. Specifically I did **not** prove that a forwarded client packet to a
VIP is DNAT'd by Cilium's `wg0` program. That inference rests on `cilium-dbg` listing `wg0` as a
LB device plus the node-local Gateway backend, not on an observed client packet. **It is the one
remaining way the withdrawn diagnosis could be partly right** — if `wg0` BPF were somehow not in
the forwarded path, forwarded traffic would need ARP and would die on the holder. I judge this
unlikely but it is unproven. *To close it: connect a client, then from the node whose `wg show`
shows the peer endpoint, watch `FORWARD -i wg0` and the `MASQUERADE` counters move while the
client loads a cluster app — and repeat it on a node that holds leases and one that does not.*
* **WHICH NODE THE ROUTER FORWARDS `udp/51820` TO IS UNDETERMINABLE RIGHT NOW**, for the same
reason. **I need a client connected** to learn it; the router is unreadable from the cluster.
* **I cannot reproduce or explain the 2026-10-08 measurements.** That entry recorded `nc -z` TCP
failures correlated with lease holding across 15 node x VIP pairs. Today the same test passes.
My leading hypothesis is **stale or incorrectly-programmed BPF LB state on `k3s-prod-1`**,
rebuilt by today's full power-cycle — which would make the lease correlation coincidental. A
competing hypothesis is that Cilium had not auto-detected `wg0` then (`devices` is empty, and
`wg0` is created by this DaemonSet after Cilium starts). **Both are unproven, and neither is
needed for the falsification** — the asserted cause is present today without its effect. I did not
retro-measure anything, and I did not read Cilium's source or issue tracker.
* **The operator's report cannot be a description of "right now."** No client has connected since
the power-cycle (0 forwarded packets, no handshake on any node), so "clients can reach cluster
services and the LAN but not the internet" must describe a session before the power-cycle or one
that has since dropped. **I did not establish when it was observed, or with which profile.**
* **THE INTERNET ROOT CAUSE IS NOT DIAGNOSED.** I proved the server side is *capable*; I did **not**
find the fault, and I shipped no fix for it. The two candidates in the triage section (a
`--split` profile; IPv6 preference leaking outside an IPv4-only `AllowedIPs`) are **reasoned, not
measured**. *What I need:* the `AllowedIPs` line from the client's own profile — that one line
decides split vs full and halves the search — and a connected client for step 2.
* **Whether `--split` was in fact used for the enrolled peer.** The profile holds a private key and
lives only on the operator's machine and the device; it is not in the repo, and I did not look for
it. Server-side `AllowedIPs` is a `/32` source filter and says nothing about the client's routing.
* **Whether `MTU = 1420` holds on a full-tunnel WAN path.** Not measured, and not measurable without
a client. It is the standard 1500 − 60; a client behind PPPoE or CGNAT with a smaller path MTU
would see large transfers hang while small requests succeed. The runbook's existing MTU note
(drop to 1280) already covers the remedy.
* **Adding `::/0` to `AllowedIPs` is untested advice.** The triage section offers it to blackhole
IPv6 rather than leak it, and states that the tunnel has no IPv6 egress. I did not try it on a
client, and I do not know how each WireGuard client behaves when handed an IPv6 route on an
interface with no IPv6 address.
* **`192.168.2.21:8443` (unifi) fails from all three nodes**, as it did on 2026-10-08 — a separate,
pre-existing fault. Not investigated, not addressed, and now recorded in the runbook as such.
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so the gate proves the L0/L1
offline subset only. Nothing above is covered by a test that runs in CI.
* **Nothing was applied, and nothing needs to be.** The branch is now documentation-only, so there
is no Flux reconcile, Ansible run or `kubectl` step to perform after merge. I ran no mutating
`kubectl`, `flux` or Ansible, and did not run `new-peer.sh` (it mints key material).
* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).
platform/vpn-ingress/README.md
+80
View File
@@ -186,6 +186,86 @@ right way; one was a defect in my own manifest.
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
## Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach
A change was proposed (PR #69) on the belief that **a node cannot reach a LoadBalancer VIP whose
Cilium L2 announcement lease it holds**, and that the VPN node therefore had to be excluded from
announcing. **That belief is wrong, and the change was withdrawn.** The half of it that is true is
real but inert, and the distinction matters enough to record, because the wrong version is very
easy to re-derive from `ip neigh` alone.
Measured with all **eight** `cilium-l2announce-*` leases held by `k3s-prod-1` — the extreme case,
after the UPS power-cycle — from the host netns of the `wireguard` pod on each node:
| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
|---|---|---|
| `ip neigh show 192.168.2.20` | `FAILED` (no lladdr) | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
| `curl -k https://192.168.2.20/` | **`200`, 18 ms** | `200`, 13–40 ms |
| `nc -z 192.168.2.20 443` | **OK** | OK |
| plex `192.168.2.24:32400` | **`200`** | `200` |
So the announcer really does **not** answer its own ARP request — that part of the observation is
reproducible. It changes nothing, because **no traffic path resolves a LoadBalancer VIP by ARP**:
* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
`Masquerading: BPF [eno1, wg0]`. Cilium's LB datapath is attached to **`wg0` as well as `eno1`**,
so a remote client's packet arriving on `wg0` for a VIP is DNAT'd to a backend **in BPF, before
any route or neighbour lookup for the VIP happens**.
* `cilium-dbg service list` on `k3s-prod-1` programs `192.168.2.20:443 => 127.0.0.1:13606` — the
shared Gateway's backend is a **node-local** Envoy socket. Every node can serve that VIP without
a packet crossing the LAN at all.
* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
which is why `curl` succeeds on the lease holder despite `FAILED`. The discriminator: a **real**
service port connects, a **bogus** one (`nc -z 192.168.2.20 9999`) fails, on all three nodes —
that is service-aware DNAT, not wire traffic.
**Two tests that look like proof and are not.** Both were used to support the withdrawn diagnosis:
* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless. It is also an
artefact of probing: VIPs answer no ICMP, so a `ping` leaves a failed entry behind whether or not
anything is wrong.
* **`curl --interface wg0`.** This fails (`rc=28`, timeout) from **all three** nodes, including the
two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it
discriminates nothing — and it is not the client path either (see below).
**Why neither probe can see the client path.** A MASQUERADEd remote client's packets are
**forwarded** — netfilter `FORWARD`, then routing — whereas everything runnable from the pod is
**socket-originated**. The two traverse different hooks, so a host-netns probe cannot measure the
forwarded path in either direction: it reports success where the forwarded path might fail, and
`--interface wg0` reports failure where it does not. **The forwarded path can only be measured with
a client actually connected**, by watching the `FORWARD -i wg0` and `POSTROUTING … MASQUERADE`
counters move.
## Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses
`new-peer.sh` has emitted a **full tunnel** (`AllowedIPs = 0.0.0.0/0`) **by default since the
component's first commit**; `--split` is the opt-in for LAN-only. There is no full-tunnel feature
to add. The server side was checked end to end and is **capable and correct** — if a full-tunnel
client cannot reach the internet, nothing in the list below is the reason:
| checked | result on all three nodes |
|---|---|
| `net.ipv4.ip_forward` (read from `/proc/sys`, never written) | `1` |
| `iptables -t nat -S POSTROUTING` | carries `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` — NATs to **any** outbound interface, so internet NAT needs no change |
| `CILIUM_POST_nat`, which precedes it in the chain | **empty, 0 packets** — it cannot short-circuit the MASQUERADE |
| `iptables -L FORWARD` | `-i wg0` and `-o wg0` ACCEPT both present |
| default route | `default via 192.168.2.1 dev eno1 onlink`, intact |
| node's own internet reachability | `http://1.1.1.1/` returns `301` |
**A full tunnel egresses the HOUSE WAN, not PIA.** This was the trap worth checking, and it is
closed: `platform/vpn-egress/egress-steering.yaml`'s `CiliumEgressGatewayPolicy` selects by
`podSelector` on `egress-class: vpn`, and this DaemonSet is deliberately labelled
`egress-class: internet`, so it is not selected. Forwarded `wg0` traffic is not pod-identity
traffic at all, and `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`) — the egress-gateway
node. A remote client's apparent public address is therefore the **home WAN IP**, which is the
intended behaviour and the thing to state when enrolling a device.
**Not measurable without a connected client.** At the time of writing `wg show wg0` listed the one
enrolled peer with **no endpoint and no handshake** on every node, and both the `FORWARD -i wg0` and
the `MASQUERADE` counters read **0 packets** — so no client traffic had been forwarded since the
power-cycle, and which node the router forwards `udp/51820` to was **undeterminable**. Only the
forwarded node shows a peer with an `endpoint` and a recent `latest handshake`.
## Prerequisites outside this directory
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
runbooks/vpn-remote-access.md
+78
View File
@@ -273,6 +273,84 @@ bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
## Triage — "the client reaches the LAN and cluster apps, but not the internet"
Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom
almost always lives, and because the server side has been measured and is not the cause.
**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.**
| the profile says | meaning |
|---|---|
| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. |
| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. |
`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit —
`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client
profile only**, so an already-issued profile keeps whatever it was rendered with until it is
re-rendered and re-imported on the device. The two forms differ by one flag:
```bash
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet>' # full tunnel
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet> --split' # LAN only
```
Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into
`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is
editing `AllowedIPs` in the profile **on the device** — the server does not care.
**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the
client connected, then watch the two counters that only forwarded traffic moves:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c "
wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5;
iptables -t nat -L POSTROUTING -n -v | head -5"'
```
* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which
node the router's `udp/51820` forward points at (the router is unreadable from the cluster).
* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is
upstream of the cluster (router forward, client, or carrier), not here.
* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side
failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there,
which breaks internet while node-local cluster VIPs still work. It has not been observed.
**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly:
* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack
client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel —
so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4:
`curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it,
and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is
blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it).
* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified
2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a
blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on
a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all.
**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the
`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain
**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route
`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the
PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled
`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the
**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6).
## Triage — which L2 announcement lease a node holds does NOT matter
Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node
could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`:
that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh`
for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer
VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend
is a node-local socket. Full measurements and the two misleading probes
(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`.
`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a
separate, pre-existing fault in that service and has nothing to do with the VPN.
## Revoking a device
Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the