The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
7.7 KiB
ADR-0015 — VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch)
- Status: Accepted — tunnel placement, steering and the four invariants stand (Kill-switch and Resilience bullets, and the mechanism named in invariant 3, superseded by ADR-0020)
- Date: 2026-08-23
- Serves: R-NET-05, R-NET-06, R-NET-07
- Supersedes (egress-steering + tunnel placement of): ADR-0013 — which now stands only for the provider/protocol (PIA over OpenVPN-UDP + port forwarding). By extension this also closes the steering question left open since ADR-0008.
Context
ADR-0013 placed the VPN in a single shared in-cluster Gluetun pod and deferred the egress-steering
mechanism (how egress-class=vpn pods actually route through it), naming "Cilium Egress Gateway → the
gateway's tunnel interface" as the recommended-but-unvalidated option.
Implementation (2026-08) proved that option non-viable, and validated the tunnel itself:
- The in-cluster Gluetun tunnel works — PIA Stockholm egress confirmed, stable.
- But
CiliumEgressGatewayPolicySNATs matched pods out of a node interface; Gluetun-in-a-pod'stun0lives in the pod netns, invisible to the node datapath. Running GluetunhostNetworkto puttun0on the node breaks the node: Gluetun owns its netns routing and on connect redirects0.0.0.0/0via the tunnel — in host netns that hijacks the node's own default route, cutting k3s/etcd/kubelet off. Gluetun has no "createtun0but don't touch the default route" mode. - The accepted community alternative — the pod-gateway pattern (VXLAN overlay + admission-webhook-injected
client routing + gateway NAT, e.g.
angelnu/pod-gateway) — works but is heavy, mutates client pods, and side-steps Cilium's datapath.
The clean fix is to move the tunnel to where routing is fully controllable — the node — and use Cilium's native egress gateway to steer.
Decision
The egress: vpn contract is realized by a node-level VPN tunnel steered by Cilium Egress Gateway, with a
host-level kill-switch:
- Tunnel — host service, not a pod. One or two labeled egress-gateway nodes run a host OpenVPN
systemd service (PIA over OpenVPN-UDP — provider/protocol per ADR-0013, same
.ovpn). It is Ansible- managed host infrastructure (like k3s itself), not a Flux workload. It is configured split-tunnel: it createstun0but does not set the node default route (route-noexec+ policy routing), so the node's own traffic (control plane, image pulls, other pods) stays on the WAN. Only traffic Cilium points attun0uses the VPN. - Steering — Cilium Egress Gateway. A
CiliumEgressGatewayPolicyselects pods labeledhomelab.keithjackson.dev/egress-class=vpnand egresses them via the gateway node'stun0(SNAT to the tunnel address). Native Cilium eBPF — no client-pod changes, no sidecar, no overlay. (egressGateway,bpf.masquerade,kubeProxyReplacementare already enabled per ADR-0002.) - Kill-switch (fail-closed, R-NET-06) — host nftables. An nftables rule on the gateway node DROPS the
steered (
egress-class=vpn) traffic whenevertun0is absent/down — a host backstop independent of Cilium egress-gateway fail-open edge cases. This is a stronger guarantee than either pod-based option. - Resilience. Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover) provide the active/standby the earlier ADRs asked for.
The user-facing contract is unchanged: services are labeled egress-class=vpn (Kyverno-enforced), have no
independent egress path, and the rules are enforced outside the service — now via a Cilium policy + host
nft rather than an in-cluster gateway pod.
Invariants (NON-NEGOTIABLE — implementation must satisfy all four)
The two egress classes have opposite fail-safe directions; getting this wrong either leaks VPN traffic or breaks the cluster. These are hard requirements, not preferences:
- Default is the local/private WAN path — for everything. Every pod (regardless of namespace) AND the
nodes themselves egress via the normal private-network/WAN stack by default. Traffic is on the VPN only
if a service explicitly opts in with
egress-class=vpn. - The VPN tunnel is never a default route. Not for the node, not for any pod.
tun0carries only the flows aCiliumEgressGatewayPolicyexplicitly selects (egress-class=vpn). The host tunnel is split-tunnel (route-noexec/policy routing) precisely to guarantee this. - The kill-switch blackhole is scoped to vpn-class only. When
tun0is down, the nftables rule drops only theegress-class=vpn(Cilium-egress-gateway-marked) traffic. It MUST NOT match or drop default/internet-class traffic or node traffic.internet-class egress is unaffected whether the tunnel is up or down. - Opposite fail directions. Default/
internettraffic fails open (toward the WAN — never blackholed).egress-class=vpntraffic fails closed (toward blackhole — never leaks to the WAN). A misconfiguration or partial failure must degrade in these directions, never the reverse.
These are gated by the egress-class (default/internet shows WAN IP; vpn shows VPN IP) and vpn-killswitch
(tunnel down ⇒ vpn-class blackholed, internet-class unaffected) conformance tests.
Consequences
- Steering is Cilium-native and declarative (a
CiliumEgressGatewayPolicyin Git) and touches no app pods — flipping a service'segress-classlabel re-routes it with no workload rearchitecting. - Two-layer ownership: the tunnel is host infra (Ansible role on the gateway nodes); the steering policy is Flux/GitOps. Consistent with how k3s is provisioned. This is the intended split, not a leak.
- Secrets move host-side: PIA credentials +
.ovpnlive on the gateway node(s) via Ansible vault, not a cluster SOPS Secret — see docs/07. (SOPS still governs in-cluster secrets.) - Gluetun is dropped as the gateway. Its validated custom-mode config + the working PIA
.ovpntransfer directly to the host OpenVPN service, so the connectivity work is retained; only the packaging changes. - Port forwarding (PIA NAT-PMP for qBittorrent) must be re-solved host-side (a small pf/natpmp helper that publishes the assigned port to the app) — Phase 6, as before.
- SPOF on a single gateway node unless run HA (two nodes + Cilium failover).
- Validated end-to-end by the
egress-class(R-NET-05/07) andvpn-killswitch(R-NET-06) conformance tests: vpn-class egress shows the PIA IP, internet-class shows the WAN IP, a git label flip re-routes, and forcingtun0down drops vpn-class egress (no leak) while internet-class is unaffected.
Alternatives considered
- In-cluster Gluetun pod + Cilium Egress Gateway → its
tun0(ADR-0013's deferred option 1) — NON-VIABLE:hostNetworkGluetun hijacks the node default route. - pod-gateway pattern (VXLAN overlay + admission webhook + gateway NAT) — the accepted in-cluster alternative; works but heavier, mutates client pods, bypasses Cilium's datapath. Rejected for a Cilium cluster in favor of the native node-level egress gateway.
- Per-pod Gluetun sidecar (ADR-0008 Pattern A) — rejected earlier: privilege colocation with the workload and no declarative east-west policy.
- Keeping the working in-cluster Gluetun pod for its OWN egress only — retained as the fallback if the
node-level tunnel proves troublesome, but it cannot serve other pods, which is the whole
egress: vpnrequirement.