Files
home-network-k3s/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md
jakman85andClaude Opus 5.5 250fb54154 docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 22:46:00 -06:00

7.7 KiB

ADR-0015 — VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch)

  • Status: Accepted — tunnel placement, steering and the four invariants stand (Kill-switch and Resilience bullets, and the mechanism named in invariant 3, superseded by ADR-0020)
  • Date: 2026-08-23
  • Serves: R-NET-05, R-NET-06, R-NET-07
  • Supersedes (egress-steering + tunnel placement of): ADR-0013 — which now stands only for the provider/protocol (PIA over OpenVPN-UDP + port forwarding). By extension this also closes the steering question left open since ADR-0008.

Context

ADR-0013 placed the VPN in a single shared in-cluster Gluetun pod and deferred the egress-steering mechanism (how egress-class=vpn pods actually route through it), naming "Cilium Egress Gateway → the gateway's tunnel interface" as the recommended-but-unvalidated option.

Implementation (2026-08) proved that option non-viable, and validated the tunnel itself:

  • The in-cluster Gluetun tunnel works — PIA Stockholm egress confirmed, stable.
  • But CiliumEgressGatewayPolicy SNATs matched pods out of a node interface; Gluetun-in-a-pod's tun0 lives in the pod netns, invisible to the node datapath. Running Gluetun hostNetwork to put tun0 on the node breaks the node: Gluetun owns its netns routing and on connect redirects 0.0.0.0/0 via the tunnel — in host netns that hijacks the node's own default route, cutting k3s/etcd/kubelet off. Gluetun has no "create tun0 but don't touch the default route" mode.
  • The accepted community alternative — the pod-gateway pattern (VXLAN overlay + admission-webhook-injected client routing + gateway NAT, e.g. angelnu/pod-gateway) — works but is heavy, mutates client pods, and side-steps Cilium's datapath.

The clean fix is to move the tunnel to where routing is fully controllable — the node — and use Cilium's native egress gateway to steer.

Decision

The egress: vpn contract is realized by a node-level VPN tunnel steered by Cilium Egress Gateway, with a host-level kill-switch:

  • Tunnel — host service, not a pod. One or two labeled egress-gateway nodes run a host OpenVPN systemd service (PIA over OpenVPN-UDP — provider/protocol per ADR-0013, same .ovpn). It is Ansible- managed host infrastructure (like k3s itself), not a Flux workload. It is configured split-tunnel: it creates tun0 but does not set the node default route (route-noexec + policy routing), so the node's own traffic (control plane, image pulls, other pods) stays on the WAN. Only traffic Cilium points at tun0 uses the VPN.
  • Steering — Cilium Egress Gateway. A CiliumEgressGatewayPolicy selects pods labeled homelab.keithjackson.dev/egress-class=vpn and egresses them via the gateway node's tun0 (SNAT to the tunnel address). Native Cilium eBPF — no client-pod changes, no sidecar, no overlay. (egressGateway, bpf.masquerade, kubeProxyReplacement are already enabled per ADR-0002.)
  • Kill-switch (fail-closed, R-NET-06) — host nftables. An nftables rule on the gateway node DROPS the steered (egress-class=vpn) traffic whenever tun0 is absent/down — a host backstop independent of Cilium egress-gateway fail-open edge cases. This is a stronger guarantee than either pod-based option.
  • Resilience. Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover) provide the active/standby the earlier ADRs asked for.

The user-facing contract is unchanged: services are labeled egress-class=vpn (Kyverno-enforced), have no independent egress path, and the rules are enforced outside the service — now via a Cilium policy + host nft rather than an in-cluster gateway pod.

Invariants (NON-NEGOTIABLE — implementation must satisfy all four)

The two egress classes have opposite fail-safe directions; getting this wrong either leaks VPN traffic or breaks the cluster. These are hard requirements, not preferences:

  1. Default is the local/private WAN path — for everything. Every pod (regardless of namespace) AND the nodes themselves egress via the normal private-network/WAN stack by default. Traffic is on the VPN only if a service explicitly opts in with egress-class=vpn.
  2. The VPN tunnel is never a default route. Not for the node, not for any pod. tun0 carries only the flows a CiliumEgressGatewayPolicy explicitly selects (egress-class=vpn). The host tunnel is split-tunnel (route-noexec/policy routing) precisely to guarantee this.
  3. The kill-switch blackhole is scoped to vpn-class only. When tun0 is down, the nftables rule drops only the egress-class=vpn (Cilium-egress-gateway-marked) traffic. It MUST NOT match or drop default/ internet-class traffic or node traffic. internet-class egress is unaffected whether the tunnel is up or down.
  4. Opposite fail directions. Default/internet traffic fails open (toward the WAN — never blackholed). egress-class=vpn traffic fails closed (toward blackhole — never leaks to the WAN). A misconfiguration or partial failure must degrade in these directions, never the reverse.

These are gated by the egress-class (default/internet shows WAN IP; vpn shows VPN IP) and vpn-killswitch (tunnel down ⇒ vpn-class blackholed, internet-class unaffected) conformance tests.

Consequences

  • Steering is Cilium-native and declarative (a CiliumEgressGatewayPolicy in Git) and touches no app pods — flipping a service's egress-class label re-routes it with no workload rearchitecting.
  • Two-layer ownership: the tunnel is host infra (Ansible role on the gateway nodes); the steering policy is Flux/GitOps. Consistent with how k3s is provisioned. This is the intended split, not a leak.
  • Secrets move host-side: PIA credentials + .ovpn live on the gateway node(s) via Ansible vault, not a cluster SOPS Secret — see docs/07. (SOPS still governs in-cluster secrets.)
  • Gluetun is dropped as the gateway. Its validated custom-mode config + the working PIA .ovpn transfer directly to the host OpenVPN service, so the connectivity work is retained; only the packaging changes.
  • Port forwarding (PIA NAT-PMP for qBittorrent) must be re-solved host-side (a small pf/natpmp helper that publishes the assigned port to the app) — Phase 6, as before.
  • SPOF on a single gateway node unless run HA (two nodes + Cilium failover).
  • Validated end-to-end by the egress-class (R-NET-05/07) and vpn-killswitch (R-NET-06) conformance tests: vpn-class egress shows the PIA IP, internet-class shows the WAN IP, a git label flip re-routes, and forcing tun0 down drops vpn-class egress (no leak) while internet-class is unaffected.

Alternatives considered

  • In-cluster Gluetun pod + Cilium Egress Gateway → its tun0 (ADR-0013's deferred option 1) — NON-VIABLE: hostNetwork Gluetun hijacks the node default route.
  • pod-gateway pattern (VXLAN overlay + admission webhook + gateway NAT) — the accepted in-cluster alternative; works but heavier, mutates client pods, bypasses Cilium's datapath. Rejected for a Cilium cluster in favor of the native node-level egress gateway.
  • Per-pod Gluetun sidecar (ADR-0008 Pattern A) — rejected earlier: privilege colocation with the workload and no declarative east-west policy.
  • Keeping the working in-cluster Gluetun pod for its OWN egress only — retained as the fallback if the node-level tunnel proves troublesome, but it cannot serve other pods, which is the whole egress: vpn requirement.