home-network-k3s-impl/infra/ansible/roles/vpn_egress_gateway/README.md

role: vpn_egress_gateway (ADR-0015)

Stands up the node-level VPN tunnel on the egress-gateway node(s): a host OpenVPN systemd service (PIA over OpenVPN-UDP), split-tunnel via policy routing, and a scoped nftables kill-switch. This is the host half of ADR-0015; the CiliumEgressGatewayPolicy that steers egress-class=vpn pods to this node's tun0 is separate GitOps authored against the shared interface contract below.

CONFORMANCE IS GATED ON A LIVE TUNNEL. This role cannot be proven green without the operator's PIA host credentials (see "What requires a live tunnel"). Authoring + lint pass in CI; the egress-class / vpn-killswitch conformance tests (R-NET-05/06/07) go green only once the tunnel is live in dev/test.

Interface contract (shared with the Cilium-policy agent)

• Inventory groups — vpn_egress_gateway (dev: k3s-dev-3) and vpn_egress_gateway_prod (prod: k3s-prod-3). Each holds exactly one host, and prod is a separate group so the flagless playbook run cannot reach it. Cilium OSS resolves a multi-match nodeSelector to "the first node in lexical ordering", so a second labelled node does not add failover — it silently steals the role.
• k8s node label homelab.keithjackson.dev/egress-gateway=true — set by this role; the CiliumEgressGatewayPolicy selects the gateway node by it. This label is what arms the datapath, which is why the role applies it LAST.
• Host tunnel iface tun0 — the Cilium policy SNATs vpn-class pods out of it.
• k8s node label homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex> — set by this role's re-steer script. Its VALUE is the uplink Cilium was last told about; its CHANGE is what makes Cilium re-resolve interface: tun0. Cilium caches that resolution and never revisits it on its own, so without this the gateway is dead after its first reconnect. See "Re-steering".

How the four ADR-0015 invariants are met

1 & 2 — split-tunnel; tun0 is never a default route

templates/vpn-egress.conf.j2 includes the operator's proven .ovpn and layers on route-noexec + pull-filter ignore "redirect-gateway", so OpenVPN installs no routes and never touches the main routing table. The node's own default route stays on the WAN whether the tunnel is up or down — k3s/etcd/kubelet are unaffected.

Steering is done with policy routing in a dedicated table (51820, named vpn_egress):

• The OpenVPN up-script adds default dev tun0 to table 51820 and an ip-rule from <tun0-addr> lookup 51820 (priority 5000). Only packets whose source is the tunnel address — i.e. the vpn-class pods Cilium has SNAT'd to tun0 — consult that table. Nothing else does, so nothing else is re-routed.
• The down-script removes the ip-rule and the tunnel default (leaving the blackhole), and retains the nft source set as the explicit fail-closed backstop (see kill-switch section).

3 & 4 — kill-switch scoped to vpn-class, opposite fail directions

Two independent layers, both scoped so they can only ever affect vpn-class traffic:

1. Routing blackhole (fail-closed fallback). Table 51820 always carries a permanent blackhole default (installed by vpn-egress-killswitch.service, metric 4000). When the tunnel default is withdrawn (flap / partial teardown) while the ip-rule still steers, packets hit the blackhole instead of falling through to the WAN.

2. nftables backstop (templates/vpn-egress-killswitch.nft.j2). Own inet vpn_egress_gateway_killswitch table, base chain policy accept (so it can only drop the narrow matches, never break anything). In postrouting:

   • oifname tun0 ip saddr @vpn_egress_gateway_src counter accept — tunnel up: let tunnel-sourced traffic out.
   • ip saddr @vpn_egress_gateway_src oifname != tun0 counter drop — fail closed: a tunnel-sourced packet trying to leave any other interface is dropped (no WAN leak). Armed even while the tunnel is down, because the source set is retained on tunnel-down (below).

   Both rules carry a counter, and both were exercised on k3s-prod-3 (2026-09-12): a source-bound ping -I 10.72.192.7 with the ip-rule present got 3/3 replies at 144 ms and moved killswitch_accept; with the ip-rule deleted, ip route get confirmed the packet would go via 192.168.2.1 dev eno1 and killswitch_drop moved 0 -> 3 packets / 252 bytes with 100% loss. The kill-switch is real, not aspirational — but read the next paragraph for what it can and cannot reach.

   SCOPE, HONESTLY — it does NOT cover vpn-class POD traffic. A counters-only diagnostic table on the dev gateway recorded, while a vpn-class pod egressed: 0 packets in forward and 0 in postrouting for the pod CIDR and for the PIA range, against 45166 packets on that same postrouting hook for WAN traffic and 16 for a node-originated curl out tun0. Cilium's eBPF egress-gateway datapath (bpf.masquerade: true) bpf_redirect()s those skbs straight to the egress device, so netfilter POSTROUTING is never invoked for them. What makes vpn-class fail closed is Cilium's own drop when the cached egress interface is unusable — tcpdump during a tunnel stop and during a tunnel restart showed 0 packets leaving either the WAN or tun0. This table's real, narrow guarantee is NODE-originated traffic sourced from a tunnel address.

@vpn_egress_gateway_src holds every tun0 address this node has had. The up-script adds the current tunnel source on every connect and never flushes (it used to flush first, which left a brief moment where a packet still sourced from the previous address matched neither the ip-rule nor the drop rule); the down-script intentionally retains it. Node traffic and internet-class traffic are never in the set and are never targeted — so they always hit policy accept and fail open toward the WAN (invariant 4), tunnel up or down. The opposite directions are structural, not incidental.

A kill-switch reload no longer disarms the guard: the .nft file uses table + flush table, which clears rules but leaves a named set's elements intact, so an Ansible run cannot empty the set mid-flight.

Explicit, edge-independent fail-closed backstop (tunnel-down)

Earlier the down-script flushed @vpn_egress_gateway_src, which disarmed the ip saddr @vpn_egress_gateway_src oifname != tun0 drop rule at exactly the moment the tunnel was down. Fail-closed then rode only on the stale SNAT source being non-routable — an implicit guarantee that could leak if the edge router NATed that stale source. The down-script now does NOT flush the set: retaining the last tun0 address keeps the drop rule armed, so any vpn-class packet that would leave the WAN while the tunnel is down is explicitly dropped in the host nft table — defense-in-depth, independent of edge-router behavior.

The set is now ADDITIVE (2026-09-12): the up-script only ever nft add elements the live source. It used to nft flush set first, and that flush was itself a hole — between a reconnect and Cilium refreshing its cached egress IP, a packet still sourced from the PREVIOUS address matched neither the ip-rule (deleted by the down-script) nor the drop rule (the address had just been flushed out of the set). Additive closes it, add element is idempotent, and an extra element can only ever tighten the rule. The set stays scoped to tunnel sources; node and internet-class traffic are never members, so no invariant is weakened (invariants 3 & 4 upheld). The routing blackhole in table 51820 remains as the second, independent fail-closed layer.

Why the node default route is provably unaffected (design-level verification)

• OpenVPN runs with route-noexec and pull-filter ignore "redirect-gateway" → it adds zero routes and cannot install a 0.0.0.0/0 via tun0. (This is exactly the failure mode that made hostNetwork gluetun non-viable in ADR-0015; here OpenVPN is explicitly told not to route.)
• All tunnel routing lives in table 51820 only; the up/down scripts touch that table and ip-rules, never table main / table default.
• The ip-rule matches from <tun0-addr> — an address the node itself never sources traffic from — so node-originated and internet-class packets never enter the tunnel table.
• The nft table is policy accept and matches only @vpn_egress_gateway_src.
• Live check (post-deploy): ip route show table main default still points at the WAN NIC, and curl ipinfo from the node shows the WAN IP, with the tunnel both up and forced down. Confirmed on k3s-prod-3 2026-09-12: default via 192.168.2.1 dev eno1 with the tunnel up, node curl = the WAN IP, k3s /readyz = ok.

Re-steering — Cilium does not follow tun0, and that is the whole ball game

Cilium resolves the policy's interface: tun0 once, caching an (egress IP, egress ifindex) pair in its BPF egress map, and does not re-resolve when tun0 changes. Upstream says so: "it does not automatically respond to a change in the gateway node's network configuration... You can force a fresh selection by re-applying the Egress Gateway policy." Measured on k3s-dev-3:

what	observed
after systemctl restart vpn-egress-tunnel	node tun0 at ifindex 1987, cilium-dbg bpf egress list still 1986
vpn-class pod egress in that state	times out; tcpdump shows 0 packets out the WAN and 0 out tun0 (no leak, no service)
egress-gateway-reconciliation-trigger-interval: 1s	does not help — dev has that setting; still stale 20s+ later. It coalesces reconciliations that k8s object events already triggered; netlink link events are not among them
stale egress IP with a live ifindex	does NOT leak either — the redirect follows the cached ifindex, so packets leave tun0 with the previous source (0 packets on the WAN, verified)
touching any node label on the gateway	cached ifindex 1986 -> 1987 within 10s, egress recovers immediately

So the failure mode is a silent outage on every tunnel restart or reboot, not a leak. The fix is templates/vpn-egress-resteer.sh.j2, which publishes <tun0 addr>-<ifindex> as the egress-uplink node label:

• a node label, not an annotation on the policy — the policy is Flux-managed GitOps and writing to it from a node invites drift/ownership churn; node labels are already this role's territory, and the label value doubles as the diagnostic (kubectl get nodes -L homelab.keithjackson.dev/egress-uplink);
• fired by the OpenVPN up-script on every connect, via systemctl start --no-block so it can never block or fail the tunnel and does not need the kube-API to be up at that instant;
• plus vpn-egress-resteer.timer (2 min) as the convergence backstop for the boot case where the API is not up yet and every in-script retry is exhausted;
• idempotent — if the label already records the live uplink it exits without writing, so the timer is free to run often.

Proof it works: systemctl stop then start of the tunnel now recovers on its own (map 1989 -> 1990, PIA IP returns). Before the fix the identical sequence stayed broken indefinitely — which also means the recovery half of the vpn-killswitch conformance test could not have passed on the pre-fix implementation.

What requires a LIVE tunnel to validate (cannot be tested without operator creds)

1. The PIA .ovpn + username/password actually authenticate and tun0 comes up as a host iface.
2. egress-class: a vpn-class pod's public IP is the PIA Stockholm IP; internet-class shows the WAN IP.
3. vpn-killswitch: forcing tun0/the tunnel service down blackholes vpn-class egress (no leak) while internet-class is unaffected — the R-NET-06 fault-injection gate.
4. That Cilium Egress Gateway SNAT to tun0 traverses host postrouting so the nft set-scoped rules actually see the packets (eBPF datapath interaction). ANSWERED 2026-09-12: it does NOT. Counters proved the steered pod packets never enter netfilter at all. See "SCOPE, HONESTLY" above. The question is closed; the guard's claim is narrowed rather than pretended.
5. MTU/PMTU across pod → tun0 → PIA (in-pod gluetun needed 1300 for Cilium VXLAN; on the node the tunnel rides the physical WAN, so vpn_egress_gateway_tun_mtu defaults empty — verify live).
6. Port forwarding (PIA NAT-PMP) for qBittorrent — ADR-0015 Phase-6 follow-up, not in this role.

Files

• defaults/main.yml — all vpn_egress_gateway_* vars; the three vault-backed credential vars.
• tasks/main.yml — packages, rt_tables, config/scripts/units, enable+start, re-steer, node label (kill-switch -> tunnel -> re-steer -> label last, because the label is what arms steering).
• handlers/main.yml — daemon-reload, kill-switch reload, tunnel restart.
• templates/ — wrapper .conf, credentials, up/down scripts, nft ruleset, the kill-switch and tunnel units, and vpn-egress-resteer.{sh,service,timer}.
• Playbook: playbooks/vpn.egress.gateway.yml. Vault keys: inventory/group_vars/vpn_egress_gateway/README.md (dev) and inventory/group_vars/vpn_egress_gateway_prod/README.md (prod).

Running it against prod

cd infra/ansible
ansible-playbook playbooks/vpn.egress.gateway.yml \
  -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts
# confirm it says exactly `k3s-prod-3`, then drop --list-hosts

The target var is vpn_egress_gateway_hosts, not the cluster-wide k3s_hosts. Never apply the egress-gateway label by hand, and never run this role with tags that skip the tunnel or kill-switch tasks — the label is the last step precisely so nothing is steered at a node that is not ready to carry it.