docs(vpn-ingress): withdraw the L2-lease diagnosis — it was falsified by measurement #69

2026-10-18T03:42:57Z

gitea-admin commented

2026-10-18 03:42:57 +00:00

This PR previously claimed a bug that does not exist. The fix has been withdrawn.

Original claim: a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself (the announcer does not answer its own ARP request), and because vpn-ingress runs on hostNetwork and MASQUERADEs clients to the node's own address, VPN clients inherit that — so all 16 Gateway hostnames were dead over the VPN.

That causal claim is falsified. Re-measured on prod with all 8 cilium-l2announce-* leases held by k3s-prod-1:

from `k3s-prod-1` (holds the gateway lease)	result
`ip neigh show 192.168.2.20`	`INCOMPLETE` — unresolved, as the original claim predicts
`nc -z 192.168.2.20 443`	OK
`curl --resolve home.keithjackson.dev:443:192.168.2.20 https://home.keithjackson.dev/`	200 in 25 ms

The asserted cause is present and the asserted effect is absent. prod-2 and prod-3, holding no lease, return the same 200. That is sufficient to withdraw the claim.

Why it was never going to bite

cilium-dbg status on every node:

KubeProxyReplacement: True  [eno1 192.168.2.10 192.168.2.9 ... (Direct Routing), wg0 10.90.0.1]
Masquerading:         BPF   [eno1, wg0]

Cilium's LB datapath is attached to wg0. A client packet arriving on wg0 destined for a VIP is DNAT'd in BPF before any neighbour lookup happens, so the VIP's ARP state is irrelevant to forwarded client traffic. cilium-dbg service list also programs 192.168.2.20:443 => 127.0.0.1:13606 — the Gateway backend is a node-local Envoy socket, so every node serves that VIP without putting a packet on the LAN at all.

Why the original investigation went wrong — the methodological error

Every probe behind the original diagnosis was run from the wireguard pod on the grounds that it "shares the host netns, so this is the path a MASQUERADEd client takes". That reasoning is wrong. A MASQUERADEd client is forwarded (netfilter FORWARD + routing); a probe started inside the pod is socket-originated (Cilium socket-LB). Different hooks. Sharing a network namespace is not sharing a datapath hook.

Two specific bad probes:

ip neigh ... FAILED is permanent, harmless, and partly a ping artefact — VIPs answer no ICMP even when perfectly healthy.
curl --interface wg0 was used as the control to rule out a host-netns artefact. It fails (rc=28) on all three nodes, including ones whose ARP resolves perfectly. It discriminates nothing, so it never supported the conclusion it was cited for.
A raw-IP curl -k https://192.168.2.20/ returns rc=35 on all three nodes — that is missing SNI, not unreachability. Envoy needs SNI to select a listener.

What remains in this PR

Documentation only — no behaviour change. The Cilium nodeSelector edit, the node_baseline label tasks, the k3s_prod var, both alerts, the promtool tests, the KSM allowlist entry and the new runbook have all been reverted or deleted.

Net diff vs main is three files:

platform/vpn-ingress/README.md — records that Cilium is attached to wg0, why VIP ARP state is a red herring, and that forwarded ≠ socket-originated
runbooks/vpn-remote-access.md — the same, plus the pre-existing 192.168.2.21:8443 (unifi) fault, which fails from all three nodes including ones with resolving ARP
decisions/2026-10-17-vpn-ingress-l2-lease-collision.md — rewritten in place to record the falsification

Also corrected: the client profile is a FULL tunnel by default

infra/vpn-ingress/new-peer.sh:178-184 emits AllowedIPs = 0.0.0.0/0 by default; --split is the opt-in that narrows it to ${WG_LAN_CIDR}, ${WG_CLIENT_CIDR}. An earlier reading of this file had the two branches backwards. There is no full-tunnel option to add — it is already the default, and runbooks/vpn-remote-access.md already documented both modes.

The reported internet problem is NOT diagnosed

No client has connected since the power-cycle — wg show wg0 shows the peer with no endpoint and no handshake on all three nodes, and FORWARD -i wg0 plus the MASQUERADE rule both read 0 packets. So no measurement of the forwarded path is possible yet and nothing here should be read as explaining it.

Leading hypothesis, reasoned not measured: the profile is IPv4-only (0.0.0.0/0, no ::/0, and there is no ::/0 anywhere in new-peer.sh). A dual-stack client therefore tunnels all IPv4 but sends IPv6 outside the tunnel, while PiHole — reached over IPv4 through the tunnel — still answers with AAAA records. Address selection prefers IPv6, so public dual-stack sites fail while LAN and cluster targets (IPv4-only, no AAAA) keep working. That matches the reported symptom precisely. Untested.

Local gate: rc 3, zero failures, committed-first and verified (decisions-log counted 48 entries = main's 47 + this one). Only gitea-settings (CI-only token) and L2/L3 conformance (no cluster) did not run. Not green — rc 3.

🤖 Generated with Claude Code

## This PR previously claimed a bug that does not exist. The fix has been withdrawn. **Original claim:** a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself (the announcer does not answer its own ARP request), and because `vpn-ingress` runs on `hostNetwork` and MASQUERADEs clients to the node's own address, VPN clients inherit that — so all 16 Gateway hostnames were dead over the VPN. **That causal claim is falsified.** Re-measured on prod with all 8 `cilium-l2announce-*` leases held by `k3s-prod-1`: | from `k3s-prod-1` (holds the gateway lease) | result | |---|---| | `ip neigh show 192.168.2.20` | `INCOMPLETE` — unresolved, as the original claim predicts | | `nc -z 192.168.2.20 443` | **OK** | | `curl --resolve home.keithjackson.dev:443:192.168.2.20 https://home.keithjackson.dev/` | **200 in 25 ms** | The asserted **cause is present** and the asserted **effect is absent**. prod-2 and prod-3, holding no lease, return the same 200. That is sufficient to withdraw the claim. ### Why it was never going to bite `cilium-dbg status` on every node: ``` KubeProxyReplacement: True [eno1 192.168.2.10 192.168.2.9 ... (Direct Routing), wg0 10.90.0.1] Masquerading: BPF [eno1, wg0] ``` **Cilium's LB datapath is attached to `wg0`.** A client packet arriving on `wg0` destined for a VIP is DNAT'd in BPF *before any neighbour lookup happens*, so the VIP's ARP state is irrelevant to forwarded client traffic. `cilium-dbg service list` also programs `192.168.2.20:443 => 127.0.0.1:13606` — the Gateway backend is a node-local Envoy socket, so every node serves that VIP without putting a packet on the LAN at all. ### Why the original investigation went wrong — the methodological error Every probe behind the original diagnosis was run from the `wireguard` pod on the grounds that it "shares the host netns, so this is the path a MASQUERADEd client takes". **That reasoning is wrong.** A MASQUERADEd client is **forwarded** (netfilter FORWARD + routing); a probe started inside the pod is **socket-originated** (Cilium socket-LB). Different hooks. Sharing a network namespace is not sharing a datapath hook. Two specific bad probes: - **`ip neigh ... FAILED`** is permanent, harmless, and partly a `ping` artefact — VIPs answer no ICMP even when perfectly healthy. - **`curl --interface wg0`** was used as the control to rule out a host-netns artefact. It fails (`rc=28`) on **all three** nodes, including ones whose ARP resolves perfectly. It discriminates nothing, so it never supported the conclusion it was cited for. - A raw-IP `curl -k https://192.168.2.20/` returns `rc=35` on all three nodes — that is **missing SNI**, not unreachability. Envoy needs SNI to select a listener. ### What remains in this PR **Documentation only** — no behaviour change. The Cilium `nodeSelector` edit, the `node_baseline` label tasks, the `k3s_prod` var, both alerts, the promtool tests, the KSM allowlist entry and the new runbook have all been reverted or deleted. Net diff vs `main` is three files: - `platform/vpn-ingress/README.md` — records that Cilium is attached to `wg0`, why VIP ARP state is a red herring, and that forwarded ≠ socket-originated - `runbooks/vpn-remote-access.md` — the same, plus the pre-existing `192.168.2.21:8443` (unifi) fault, which fails from **all three** nodes including ones with resolving ARP - `decisions/2026-10-17-vpn-ingress-l2-lease-collision.md` — rewritten in place to record the falsification ### Also corrected: the client profile is a FULL tunnel by default `infra/vpn-ingress/new-peer.sh:178-184` emits `AllowedIPs = 0.0.0.0/0` by default; `--split` is the opt-in that narrows it to `${WG_LAN_CIDR}, ${WG_CLIENT_CIDR}`. An earlier reading of this file had the two branches backwards. There is no full-tunnel option to add — it is already the default, and `runbooks/vpn-remote-access.md` already documented both modes. ### The reported internet problem is NOT diagnosed No client has connected since the power-cycle — `wg show wg0` shows the peer with **no endpoint and no handshake** on all three nodes, and `FORWARD -i wg0` plus the `MASQUERADE` rule both read **0 packets**. So no measurement of the forwarded path is possible yet and nothing here should be read as explaining it. Leading hypothesis, **reasoned not measured**: the profile is IPv4-only (`0.0.0.0/0`, no `::/0`, and there is no `::/0` anywhere in `new-peer.sh`). A dual-stack client therefore tunnels all IPv4 but sends IPv6 **outside** the tunnel, while PiHole — reached over IPv4 through the tunnel — still answers with AAAA records. Address selection prefers IPv6, so public dual-stack sites fail while LAN and cluster targets (IPv4-only, no AAAA) keep working. That matches the reported symptom precisely. Untested. Local gate: **rc 3, zero failures**, committed-first and verified (`decisions-log` counted 48 entries = main's 47 + this one). Only `gitea-settings` (CI-only token) and L2/L3 `conformance` (no cluster) did not run. Not green — rc 3. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-18 03:42:57 +00:00

fix(vpn-ingress): the VPN ingress node must announce no L2 VIPs, or clients route nothing

gate / gate (pull_request) Successful in 2m22s

Details

gate / conformance (pull_request) Has been skipped

Details

f5353cbd30

A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
holds itself: the announcer does not answer its own ARP request, so on the
lease holder the neighbour entry settles at FAILED and packets to that VIP die
before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes
x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced.

platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and
MASQUERADEs remote clients to the node's own address, so a VPN client takes
the node's LAN path and inherits its ARP failures. The router's udp/51820
forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16
Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering
(its VIP has two Services and k3s-prod-2 held the other lease). The tunnel
handshook, names resolved, every app hung, and nothing reported it.

  on k3s-prod-1:  ip neigh show 192.168.2.20 -> FAILED      (its own VIP)
  on prod-2/3:    ip neigh show 192.168.2.20 -> lladdr ...  (prod-1), 443 OK

Mark the node and exclude it from announcing:

* node_baseline labels the forwarded node (node_baseline_vpn_ingress_node,
  k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes
  the label from any node that no longer matches so moving it MOVES the label.
* l2-announcement-policy.yaml excludes that label, so the node ARP-resolves
  every VIP. Fixes all VIPs for VPN clients, not just the one that collided.
  Cost: announcement candidacy drops from 3 nodes to 2.
* VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels
  (label added to the kube-state-metrics allowlist), with
  VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches,
  so a missing label would silently re-arm the whole failure AND leave the
  critical alert inert. Four promtool cases, including that inertness.

A node LABEL rather than a cluster-vars knob because the alert must read it
too, and extract_rules.py rightly refuses a ${...} placeholder in an alert
expr -- the rule unit-tested would not be the rule deployed. One source of
truth, and it touches no sensitive path.

Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and
"failover is this one line". Both false; the first caused this.

UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the
handoff entry's third section.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 added 1 commit 2026-10-18 04:16:00 +00:00

Merge branch 'main' into fix/vpn-ingress-l2-lease-collision

gate / gate (pull_request) Successful in 1m55s

Details

gate / conformance (pull_request) Has been skipped

Details

0913ea9450

gitea-admin referenced this issue from a commit

2026-10-18 05:01:00 +00:00

revert(vpn-ingress): withdraw the L2 lease-collision fix — a node CAN reach a VIP it announces

gitea-admin force-pushed fix/vpn-ingress-l2-lease-collision from 0913ea9450 to e82dab5df5

2026-10-18 05:01:00 +00:00

Compare

gitea-admin changed title from ~~fix(vpn-ingress): the VPN ingress node must announce no L2 VIPs, or clients route nothing~~ to docs(vpn-ingress): withdraw the L2-lease diagnosis — it was falsified by measurement

2026-10-18 05:06:48 +00:00

gitea-admin referenced this pull request

2026-10-18 07:40:25 +00:00

fix(conformance): one regex made the merge gate take 28 minutes instead of 2 #76

gitea-admin referenced this issue from a commit

2026-10-18 07:40:25 +00:00

fix(conformance): one regex made the merge gate take 28 minutes instead of 2

gitea-admin force-pushed fix/vpn-ingress-l2-lease-collision from e82dab5df5 to b258e47aa6

2026-10-18 08:14:30 +00:00

Compare

jakman85 commented

2026-10-31 15:26:04 +00:00

This merge request is just historical info of an effort that is now OBE and has no affect on the codebase. History of the work will be kept in the git history and PR history and, therefore, does not need to be merged.

jakman85 closed this pull request

2026-10-31 15:26:04 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#69