Measured on k3s-dev-3, not reasoned about. Three findings, one of which had the
dev gateway silently dead for ~80 minutes before this change:
1. Cilium caches (egress IP, egress ifindex) when it first resolves the CEGP's
`interface: tun0` and does NOT re-resolve when tun0 changes. Every tunnel
restart re-creates tun0 with a NEW ifindex, so vpn-class egress is blackholed
from then on. Observed: node had tun0 at ifindex 1987 while `cilium-dbg bpf
egress list` still said 1986; pod curl timed out. Forcing a re-apply moved the
map to 1987 and egress recovered instantly. `egress-gateway-reconciliation-
trigger-interval: 1s` does NOT cover netlink link events — dev has that same
1s setting and still did not self-heal, 20s+ after recovery.
Fix: vpn-egress-resteer.sh publishes "<tun0 addr>-<ifindex>" as a node label.
A node update makes the gateway re-process the policy (verified: cached
ifindex 1986 -> 1987 within 10s). A node label, not an annotation on the
policy, because the policy is Flux-managed GitOps and the node's labels are
already this role's territory. Fired by the OpenVPN up-script on every
connect, plus a 2-minute convergence timer for the boot case where the
kube-API is not up yet. Proof: `systemctl stop` then `start` of the tunnel now
recovers on its own (map 1989 -> 1990, PIA IP returns); before this commit the
identical sequence stayed broken indefinitely.
2. The nft kill-switch CANNOT see the traffic it claims to guard. A counters-only
diagnostic table on the gateway node recorded, while a vpn-class pod egressed:
0 packets in forward and 0 in postrouting for both the pod CIDR and the PIA
range, against 45166 packets on the same postrouting hook for WAN traffic and
16 packets for a node-originated curl out tun0. Cilium's eBPF datapath
bpf_redirect()s these skbs to the egress device without invoking netfilter.
So `ip saddr @set oifname != tun0 drop` can never fire for vpn-class pods.
What actually fails closed is Cilium's own drop when the cached egress
interface is unusable — tcpdump during a tunnel stop AND during a restart
showed 0 packets leaving either the WAN or tun0. Every rule now carries a
`counter` so the claim stays falsifiable, and every comment that called this
table the authoritative vpn-class backstop is corrected rather than left to
mislead the next reader.
Also removed the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable:
nothing stamps that mark (the only Cilium mark on the node is 0x200/0xf00, the
L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and
a live rule matching an arbitrary mark is a hazard to whatever picks it next.
3. The up-script flushed the nft source set on every connect, so a packet still
sourced from the previous tunnel address matched neither the ip-rule nor the
drop rule. The set is now additive (`add element` is idempotent; the
down-script already retained it): an element can only ever tighten the guard.
Finding 2 means that window could not have leaked POD traffic, but
node-originated tunnel traffic does traverse netfilter, and additive is free.
Kill-switch reloads no longer disarm the guard either: the .nft file uses
`table` + `flush table`, which clears rules but leaves the named set's elements
intact, so an Ansible run cannot empty the set mid-flight.
Co-Authored-By: Claude Opus 5 <[email protected]>
jakman85andClaude Opus 5
6f698f59af