revert(vpn-ingress): withdraw the L2 lease-collision fix — a node CAN reach a VIP it announces
gate / gate (pull_request) Successful in 1m52s
gate / conformance (pull_request) Has been skipped

PR #69 claimed a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
lease it holds, and excluded the VPN ingress node from announcing on that basis.
The claim makes a sharp prediction; today's power-cycle put ALL EIGHT
cilium-l2announce-* leases on k3s-prod-1, and the prediction failed.

Measured 2026-10-17 from the wireguard pod's host netns on each node: the holder
reaches 192.168.2.20/.22/.23/.24 normally (curl 200 in 18ms on .20) while its
ip neigh for every one of them sits at FAILED. The asserted cause is present
without the asserted effect, which falsifies it. .21 (unifi) fails from all three
nodes and is a separate pre-existing fault.

It is inert because no path resolves a VIP by ARP: cilium-dbg reports
KubeProxyReplacement on [eno1, wg0] and programs 192.168.2.20:443 =>
127.0.0.1:13606, so VIPs are DNAT'd in BPF before any neighbour lookup, on the
forwarded path as well as the host one.

Both probes behind the original diagnosis were bad. `ip neigh` FAILED is
permanent and harmless, and a ping artefact. `curl --interface wg0` — the control
used to rule out a host-netns artefact — fails on all three nodes, including ones
whose ARP resolves, so it discriminates nothing. The deeper error: a MASQUERADEd
client is FORWARDED while every pod-runnable probe is socket-originated. Sharing
a netns is not sharing a hook.

Reverted the policy nodeSelector, the Ansible node label, both alerts, their
promtool tests and the kube-state-metrics allowlist entry. Kept documentation
only: the measurements, which probes lie, and the fact that the forwarded path
remains unmeasured without a connected client.

Also recorded, for the reported "LAN and cluster work, internet does not":
new-peer.sh has emitted a FULL tunnel (0.0.0.0/0) by default since f92707f, so
there is no full-tunnel option to add and no code change is made. The server side
is capable — ip_forward=1, the 10.90.0.0/24 MASQUERADE present with an empty
CILIUM_POST_nat ahead of it, FORWARD accepts, default route intact, nodes reach
1.1.1.1. A full tunnel egresses the HOUSE WAN, not PIA: the egress policy selects
pods by egress-class=vpn, this pod is labelled internet, and tun0 exists on
k3s-prod-3 only. Root cause NOT found; no client has connected since the
power-cycle (0 forwarded packets, no handshake), so it is not reproducible here.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-17 23:58:40 -05:00
1 parent 4656b07178
commit e82dab5df5
13 files changed
+338 -871

No files matched your search

+3 -33
View File
@@ -62,39 +62,9 @@
# as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to
# whichever node the scheduler picked and silently invalidate that forward on every reschedule. A
# DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of
# them, and if that node is lost every other node is already warm and answers the same keys, so no
# client profile ever changes. Each node binds udp/51820 in its own netns, so there is no conflict,
# and the two idle listeners cost ~15 MiB each.
#
# BUT THE NODES ARE NOT INTERCHANGEABLE, AND THIS COMMENT USED TO CLAIM THEY WERE.
# It said "FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work". That was wrong, and the
# wording cost two days of silently broken remote access on 2026-10-08. MEASURED that day: a node
# cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds ITSELF — the announcer
# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and packets to
# that VIP die before leaving the node:
#
# k3s-prod-1, holding the shared Gateway lease: ip neigh show 192.168.2.20 -> FAILED
# k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr …(prod-1)
#
# Because this pod MASQUERADEs clients to the node's own address (see PostUp below), a remote client
# inherits exactly that failure. With the forward on the Gateway's announcing node, all 16 Gateway
# hostnames were unreachable over the VPN while PiHole DNS still resolved — a tunnel that
# handshakes, resolves names, and then hangs on every single app.
#
# THE FIX IS DECLARATIVE AND LIVES OUTSIDE THIS FILE: the forwarded node is labelled
# homelab.keithjackson.dev/vpn-ingress=true by infra/ansible/roles/node_baseline (from
# node_baseline_vpn_ingress_node), and platform-config/cilium/l2-announcement-policy.yaml excludes
# that label from announcing ANY VIP, so the node can ARP-resolve all of them. A label rather than a
# cluster-vars knob because the alert that guards this has to read it too, and the alert unit-test
# harness refuses a postBuild placeholder in an expr. Failover is therefore TWO steps — move
# node_baseline_vpn_ingress_node and re-run the role, THEN re-point the router
# (runbooks/vpn-remote-access.md step 3).
# `VpnIngressNodeAnnouncesL2VIP` (platform/monitoring/alerts.yaml) fires if the two collide again.
#
# This pod deliberately does NOT probe that itself. A readinessProbe on VIP reachability would wedge
# a DaemonSet rollout behind an unrelated Gateway outage (maxUnavailable: 1 below), and a liveness
# probe would tear down a tunnel that is carrying LAN traffic perfectly well. The condition is a
# property of the CLUSTER's announcement layout, so it is watched there.
# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no
# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own
# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each.
#
# All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the
# DaemonSet to cover the cluster; add them if a node is ever tainted.