Measured on k3s-dev-3, not reasoned about. Three findings, one of which had the
dev gateway silently dead for ~80 minutes before this change:
1. Cilium caches (egress IP, egress ifindex) when it first resolves the CEGP's
`interface: tun0` and does NOT re-resolve when tun0 changes. Every tunnel
restart re-creates tun0 with a NEW ifindex, so vpn-class egress is blackholed
from then on. Observed: node had tun0 at ifindex 1987 while `cilium-dbg bpf
egress list` still said 1986; pod curl timed out. Forcing a re-apply moved the
map to 1987 and egress recovered instantly. `egress-gateway-reconciliation-
trigger-interval: 1s` does NOT cover netlink link events — dev has that same
1s setting and still did not self-heal, 20s+ after recovery.
Fix: vpn-egress-resteer.sh publishes "<tun0 addr>-<ifindex>" as a node label.
A node update makes the gateway re-process the policy (verified: cached
ifindex 1986 -> 1987 within 10s). A node label, not an annotation on the
policy, because the policy is Flux-managed GitOps and the node's labels are
already this role's territory. Fired by the OpenVPN up-script on every
connect, plus a 2-minute convergence timer for the boot case where the
kube-API is not up yet. Proof: `systemctl stop` then `start` of the tunnel now
recovers on its own (map 1989 -> 1990, PIA IP returns); before this commit the
identical sequence stayed broken indefinitely.
2. The nft kill-switch CANNOT see the traffic it claims to guard. A counters-only
diagnostic table on the gateway node recorded, while a vpn-class pod egressed:
0 packets in forward and 0 in postrouting for both the pod CIDR and the PIA
range, against 45166 packets on the same postrouting hook for WAN traffic and
16 packets for a node-originated curl out tun0. Cilium's eBPF datapath
bpf_redirect()s these skbs to the egress device without invoking netfilter.
So `ip saddr @set oifname != tun0 drop` can never fire for vpn-class pods.
What actually fails closed is Cilium's own drop when the cached egress
interface is unusable — tcpdump during a tunnel stop AND during a restart
showed 0 packets leaving either the WAN or tun0. Every rule now carries a
`counter` so the claim stays falsifiable, and every comment that called this
table the authoritative vpn-class backstop is corrected rather than left to
mislead the next reader.
Also removed the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable:
nothing stamps that mark (the only Cilium mark on the node is 0x200/0xf00, the
L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and
a live rule matching an arbitrary mark is a hazard to whatever picks it next.
3. The up-script flushed the nft source set on every connect, so a packet still
sourced from the previous tunnel address matched neither the ip-rule nor the
drop rule. The set is now additive (`add element` is idempotent; the
down-script already retained it): an element can only ever tighten the guard.
Finding 2 means that window could not have leaked POD traffic, but
node-originated tunnel traffic does traverse netfilter, and additive is free.
Kill-switch reloads no longer disarm the guard either: the .nft file uses
`table` + `flush table`, which clears rules but leaves the named set's elements
intact, so an Ansible run cannot empty the set mid-flight.
Co-Authored-By: Claude Opus 5 <[email protected]>
The down-script previously flushed the kill-switch source set on tunnel-down, which
disarmed the 'ip saddr @vpn_egress_gateway_src oifname != tun0 drop' rule exactly when
the tunnel was down — leaving fail-closed to ride only on the stale SNAT source being
non-routable (could leak if the edge NATed it). Retain the set instead: the drop rule
stays armed so any vpn-class packet leaving the WAN while the tunnel is down is EXPLICITLY
dropped (edge-independent). Safe because the up-script flush+re-adds the source on every
reconnect. Validated behavior: WS-NET fault-injection on k3s-dev-3 (2026-08-24).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Author the vpn_egress_gateway role + playbook that stands up the NODE-level
PIA OpenVPN tunnel on egress-gateway node(s) — the host half of ADR-0015. The
CiliumEgressGatewayPolicy steering vpn-class pods is separate GitOps against the
same contract (group vpn_egress_gateway, node label
homelab.keithjackson.dev/egress-gateway=true, host iface tun0).
Honors all four ADR-0015 invariants:
- Split-tunnel (inv 1&2): OpenVPN route-noexec + pull-filter ignore
redirect-gateway => installs no routes; policy routing in a dedicated table
(51820) with `ip rule from <tun0-addr>` steers ONLY Cilium-SNAT'd vpn-class
traffic. Node default route stays on the WAN, tunnel up or down.
- Scoped nftables kill-switch (inv 3&4): own inet table, policy accept, drops
only tunnel-sourced (@vpn_egress_gateway_src set) packets not leaving tun0 =>
vpn-class fails closed (blackhole), node/internet-class fail open (WAN),
never matched.
PIA creds + the proven sweden-aes-128-cbc-udp-ip .ovpn come from Ansible vault
(host-side per ADR-0015/docs-07); vault keys documented, no secret values
committed. ansible-lint (production) + yamllint clean. Not deployed: green
conformance is gated on the live tunnel (operator host creds) — see role README.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 5