Files
home-network-k3s/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md
T
jakman85andClaude Opus 4.8 c68349f4ab ADR-0015: add non-negotiable egress invariants (default=WAN, scoped kill-switch)
Make the operator's requirement explicit: default (every pod + the nodes) egresses
the local/private WAN, never the VPN; VPN is strictly opt-in via egress-class=vpn;
the kill-switch blackholes ONLY vpn-class. The two classes fail in opposite
directions — internet fails open (WAN), vpn fails closed (blackhole). Added as ADR-0015
Invariants + a default-WAN principle in docs/03, gated by the egress-class/vpn-killswitch
conformance tests.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 00:52:07 -05:00

7.5 KiB

ADR-0015 — VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch)

  • Status: Accepted
  • Date: 2026-08-23
  • Serves: R-NET-05, R-NET-06, R-NET-07
  • Supersedes (egress-steering + tunnel placement of): ADR-0013 — which now stands only for the provider/protocol (PIA over OpenVPN-UDP + port forwarding). By extension this also closes the steering question left open since ADR-0008.

Context

ADR-0013 placed the VPN in a single shared in-cluster Gluetun pod and deferred the egress-steering mechanism (how egress-class=vpn pods actually route through it), naming "Cilium Egress Gateway → the gateway's tunnel interface" as the recommended-but-unvalidated option.

Implementation (2026-08) proved that option non-viable, and validated the tunnel itself:

  • The in-cluster Gluetun tunnel works — PIA Stockholm egress confirmed, stable.
  • But CiliumEgressGatewayPolicy SNATs matched pods out of a node interface; Gluetun-in-a-pod's tun0 lives in the pod netns, invisible to the node datapath. Running Gluetun hostNetwork to put tun0 on the node breaks the node: Gluetun owns its netns routing and on connect redirects 0.0.0.0/0 via the tunnel — in host netns that hijacks the node's own default route, cutting k3s/etcd/kubelet off. Gluetun has no "create tun0 but don't touch the default route" mode.
  • The accepted community alternative — the pod-gateway pattern (VXLAN overlay + admission-webhook-injected client routing + gateway NAT, e.g. angelnu/pod-gateway) — works but is heavy, mutates client pods, and side-steps Cilium's datapath.

The clean fix is to move the tunnel to where routing is fully controllable — the node — and use Cilium's native egress gateway to steer.

Decision

The egress: vpn contract is realized by a node-level VPN tunnel steered by Cilium Egress Gateway, with a host-level kill-switch:

  • Tunnel — host service, not a pod. One or two labeled egress-gateway nodes run a host OpenVPN systemd service (PIA over OpenVPN-UDP — provider/protocol per ADR-0013, same .ovpn). It is Ansible- managed host infrastructure (like k3s itself), not a Flux workload. It is configured split-tunnel: it creates tun0 but does not set the node default route (route-noexec + policy routing), so the node's own traffic (control plane, image pulls, other pods) stays on the WAN. Only traffic Cilium points at tun0 uses the VPN.
  • Steering — Cilium Egress Gateway. A CiliumEgressGatewayPolicy selects pods labeled homelab.keithjackson.dev/egress-class=vpn and egresses them via the gateway node's tun0 (SNAT to the tunnel address). Native Cilium eBPF — no client-pod changes, no sidecar, no overlay. (egressGateway, bpf.masquerade, kubeProxyReplacement are already enabled per ADR-0002.)
  • Kill-switch (fail-closed, R-NET-06) — host nftables. An nftables rule on the gateway node DROPS the steered (egress-class=vpn) traffic whenever tun0 is absent/down — a host backstop independent of Cilium egress-gateway fail-open edge cases. This is a stronger guarantee than either pod-based option.
  • Resilience. Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover) provide the active/standby the earlier ADRs asked for.

The user-facing contract is unchanged: services are labeled egress-class=vpn (Kyverno-enforced), have no independent egress path, and the rules are enforced outside the service — now via a Cilium policy + host nft rather than an in-cluster gateway pod.

Invariants (NON-NEGOTIABLE — implementation must satisfy all four)

The two egress classes have opposite fail-safe directions; getting this wrong either leaks VPN traffic or breaks the cluster. These are hard requirements, not preferences:

  1. Default is the local/private WAN path — for everything. Every pod (regardless of namespace) AND the nodes themselves egress via the normal private-network/WAN stack by default. Traffic is on the VPN only if a service explicitly opts in with egress-class=vpn.
  2. The VPN tunnel is never a default route. Not for the node, not for any pod. tun0 carries only the flows a CiliumEgressGatewayPolicy explicitly selects (egress-class=vpn). The host tunnel is split-tunnel (route-noexec/policy routing) precisely to guarantee this.
  3. The kill-switch blackhole is scoped to vpn-class only. When tun0 is down, the nftables rule drops only the egress-class=vpn (Cilium-egress-gateway-marked) traffic. It MUST NOT match or drop default/ internet-class traffic or node traffic. internet-class egress is unaffected whether the tunnel is up or down.
  4. Opposite fail directions. Default/internet traffic fails open (toward the WAN — never blackholed). egress-class=vpn traffic fails closed (toward blackhole — never leaks to the WAN). A misconfiguration or partial failure must degrade in these directions, never the reverse.

These are gated by the egress-class (default/internet shows WAN IP; vpn shows VPN IP) and vpn-killswitch (tunnel down ⇒ vpn-class blackholed, internet-class unaffected) conformance tests.

Consequences

  • Steering is Cilium-native and declarative (a CiliumEgressGatewayPolicy in Git) and touches no app pods — flipping a service's egress-class label re-routes it with no workload rearchitecting.
  • Two-layer ownership: the tunnel is host infra (Ansible role on the gateway nodes); the steering policy is Flux/GitOps. Consistent with how k3s is provisioned. This is the intended split, not a leak.
  • Secrets move host-side: PIA credentials + .ovpn live on the gateway node(s) via Ansible vault, not a cluster SOPS Secret — see docs/07. (SOPS still governs in-cluster secrets.)
  • Gluetun is dropped as the gateway. Its validated custom-mode config + the working PIA .ovpn transfer directly to the host OpenVPN service, so the connectivity work is retained; only the packaging changes.
  • Port forwarding (PIA NAT-PMP for qBittorrent) must be re-solved host-side (a small pf/natpmp helper that publishes the assigned port to the app) — Phase 6, as before.
  • SPOF on a single gateway node unless run HA (two nodes + Cilium failover).
  • Validated end-to-end by the egress-class (R-NET-05/07) and vpn-killswitch (R-NET-06) conformance tests: vpn-class egress shows the PIA IP, internet-class shows the WAN IP, a git label flip re-routes, and forcing tun0 down drops vpn-class egress (no leak) while internet-class is unaffected.

Alternatives considered

  • In-cluster Gluetun pod + Cilium Egress Gateway → its tun0 (ADR-0013's deferred option 1) — NON-VIABLE: hostNetwork Gluetun hijacks the node default route.
  • pod-gateway pattern (VXLAN overlay + admission webhook + gateway NAT) — the accepted in-cluster alternative; works but heavier, mutates client pods, bypasses Cilium's datapath. Rejected for a Cilium cluster in favor of the native node-level egress gateway.
  • Per-pod Gluetun sidecar (ADR-0008 Pattern A) — rejected earlier: privilege colocation with the workload and no declarative east-west policy.
  • Keeping the working in-cluster Gluetun pod for its OWN egress only — retained as the fallback if the node-level tunnel proves troublesome, but it cannot serve other pods, which is the whole egress: vpn requirement.