home-network-k3s-impl/infra/ansible/roles/vpn_egress_gateway

role: vpn_egress_gateway (ADR-0015)

Stands up the node-level VPN tunnel on the egress-gateway node(s): a host OpenVPN systemd service (PIA over OpenVPN-UDP), split-tunnel via policy routing, and a scoped nftables kill-switch. This is the host half of ADR-0015; the CiliumEgressGatewayPolicy that steers egress-class=vpn pods to this node's tun0 is separate GitOps authored against the shared interface contract below.

CONFORMANCE IS GATED ON A LIVE TUNNEL. This role cannot be proven green without the operator's PIA host credentials (see "What requires a live tunnel"). Authoring + lint pass in CI; the egress-class / vpn-killswitch conformance tests (R-NET-05/06/07) go green only once the tunnel is live in dev/test.

Interface contract (shared with the Cilium-policy agent)

• Inventory group vpn_egress_gateway — the egress-gateway node(s) (initially k3s-dev-3).
• k8s node label homelab.keithjackson.dev/egress-gateway=true — set by this role; the CiliumEgressGatewayPolicy selects the gateway node by it.
• Host tunnel iface tun0 — the Cilium policy SNATs vpn-class pods out of it.
• Optional fwmark 0x1c8 — a contract hook (see kill-switch layering) the Cilium side MAY stamp on vpn-class packets for a tunnel-down-independent host backstop.

How the four ADR-0015 invariants are met

1 & 2 — split-tunnel; tun0 is never a default route

templates/vpn-egress.conf.j2 includes the operator's proven .ovpn and layers on route-noexec + pull-filter ignore "redirect-gateway", so OpenVPN installs no routes and never touches the main routing table. The node's own default route stays on the WAN whether the tunnel is up or down — k3s/etcd/kubelet are unaffected.

Steering is done with policy routing in a dedicated table (51820, named vpn_egress):

• The OpenVPN up-script adds default dev tun0 to table 51820 and an ip-rule from <tun0-addr> lookup 51820 (priority 5000). Only packets whose source is the tunnel address — i.e. the vpn-class pods Cilium has SNAT'd to tun0 — consult that table. Nothing else does, so nothing else is re-routed.
• The down-script removes the ip-rule and the tunnel default (leaving the blackhole), and retains the nft source set as the explicit fail-closed backstop (see kill-switch section).

3 & 4 — kill-switch scoped to vpn-class, opposite fail directions

Two independent layers, both scoped so they can only ever affect vpn-class traffic:

1. Routing blackhole (fail-closed fallback). Table 51820 always carries a permanent blackhole default (installed by vpn-egress-killswitch.service, metric 4000). When the tunnel default is withdrawn (flap / partial teardown) while the ip-rule still steers, packets hit the blackhole instead of falling through to the WAN.
2. nftables backstop (templates/vpn-egress-killswitch.nft.j2). Own inet vpn_egress_gateway_killswitch table, base chain policy accept (so it can only drop the narrow matches, never break anything). In postrouting:
   • oifname tun0 ip saddr @vpn_egress_gateway_src accept — tunnel up: let steered traffic out.
   • ip saddr @vpn_egress_gateway_src oifname != tun0 drop — fail closed: a tunnel-sourced packet trying to leave any other interface is dropped (no WAN leak). This rule stays armed even while the tunnel is down because the source set is retained on tunnel-down (see below).
   • meta mark 0x1c8 oifname != tun0 drop — the optional fwmark backstop.

@vpn_egress_gateway_src holds the last live tun0 address. The up-script flushes then re-adds the current tunnel source on every connect; the down-script intentionally retains it (see "Explicit fail-closed backstop" below). Node traffic and internet-class traffic are never in the set, never carry the mark, and are never targeted — so they always hit policy accept and fail open toward the WAN (invariant 4), tunnel up or down. The opposite directions are structural, not incidental.

Explicit, edge-independent fail-closed backstop (tunnel-down)

Earlier the down-script flushed @vpn_egress_gateway_src, which disarmed the ip saddr @vpn_egress_gateway_src oifname != tun0 drop rule at exactly the moment the tunnel was down. Fail-closed then rode only on the stale SNAT source being non-routable — an implicit guarantee that could leak if the edge router NATed that stale source. The down-script now does NOT flush the set: retaining the last tun0 address keeps the drop rule armed, so any vpn-class packet that would leave the WAN while the tunnel is down is explicitly dropped in the host nft table — defense-in-depth, independent of edge-router behavior.

This is safe precisely because the up-script flushes the set and re-adds the (possibly new) tunnel source on every reconnect (nft flush set … then nft add element …), so a retained stale address self-corrects the instant the tunnel comes back. The set remains scoped to vpn-class only; node and internet-class traffic are never members, so no invariant is weakened — only vpn-class is ever affected (invariants 3 & 4 upheld). The routing blackhole in table 51820 remains as the second, independent fail-closed layer.

Why the node default route is provably unaffected (design-level verification)

• OpenVPN runs with route-noexec and pull-filter ignore "redirect-gateway" → it adds zero routes and cannot install a 0.0.0.0/0 via tun0. (This is exactly the failure mode that made hostNetwork gluetun non-viable in ADR-0015; here OpenVPN is explicitly told not to route.)
• All tunnel routing lives in table 51820 only; the up/down scripts touch that table and ip-rules, never table main / table default.
• The ip-rule matches from <tun0-addr> — an address the node itself never sources traffic from — so node-originated and internet-class packets never enter the tunnel table.
• The nft table is policy accept and matches only @vpn_egress_gateway_src / the fwmark.
• Live check (post-deploy): ip route show table main default still points at the WAN NIC, and curl ipinfo from the node shows the WAN IP, with the tunnel both up and forced down.

Kill-switch layering & ESCALATION (honest limit)

• The nft @vpn_egress_gateway_src rule and the routing blackhole robustly cover the case where the tunnel is up but a vpn-class packet is (mis)routed toward the WAN, and flaps.
• The tunnel fully-down case relies on Cilium Egress Gateway itself failing closed when its egress interface tun0 is absent (Cilium 1.16 drops egress-gateway-marked traffic with no valid egress path rather than falling back to the node's default route) plus the routing blackhole. That is the ADR's expected behavior.
• For a host backstop that is fully independent of Cilium even in a fail-open regression when the tunnel is down (no tun0 address to key on), the host needs a stable marker on vpn-class packets. The only contract-available marker is the 0x1c8 fwmark hook — which the Cilium-policy side must stamp. ESCALATION for the PM: either (a) confirm the Cilium Egress Gateway 1.16 fail-closed + routing-blackhole guarantee is accepted as sufficient for R-NET-06, or (b) add the shared fwmark to the interface contract so the CiliumEgressGatewayPolicy marks vpn-class traffic and this role's fwmark rule becomes the tunnel-down-independent backstop. This is flagged rather than worked around — no spec was weakened to reach done.

What requires a LIVE tunnel to validate (cannot be tested without operator creds)

1. The PIA .ovpn + username/password actually authenticate and tun0 comes up as a host iface.
2. egress-class: a vpn-class pod's public IP is the PIA Stockholm IP; internet-class shows the WAN IP.
3. vpn-killswitch: forcing tun0/the tunnel service down blackholes vpn-class egress (no leak) while internet-class is unaffected — the R-NET-06 fault-injection gate.
4. That Cilium Egress Gateway SNAT to tun0 traverses host postrouting so the nft set-scoped rules actually see the packets (eBPF datapath interaction — see escalation).
5. MTU/PMTU across pod → tun0 → PIA (in-pod gluetun needed 1300 for Cilium VXLAN; on the node the tunnel rides the physical WAN, so vpn_egress_gateway_tun_mtu defaults empty — verify live).
6. Port forwarding (PIA NAT-PMP) for qBittorrent — ADR-0015 Phase-6 follow-up, not in this role.

Files

• defaults/main.yml — all vpn_egress_gateway_* vars; the three vault-backed credential vars.
• tasks/main.yml — packages, rt_tables, config/scripts/units, enable+start, node label.
• handlers/main.yml — daemon-reload, kill-switch reload, tunnel restart.
• templates/ — wrapper .conf, credentials, up/down scripts, nft ruleset, both systemd units.
• Playbook: playbooks/vpn.egress.gateway.yml. Vault keys: inventory/group_vars/vpn_egress_gateway/README.md.