home-network-k3s-impl/platform/vpn-egress

platform/vpn-egress — shared VPN egress gateway (Phase 3, ADR-0013)

Single shared Gluetun gateway: PIA over OpenVPN-UDP (ADR-0013 amendment — Gluetun does not support PIA over WireGuard), built-in kill-switch, native port forwarding. vpn-class services (sonarr, radarr, jackett, qbittorrent) egress only through this tunnel; internet-class services go out the normal WAN. Serves R-NET-05/06/07.

STATUS: enabled in platform/kustomization.yaml (Phase 3). pia-credentials SOPS secret is provisioned. Remaining work: validate the tunnel comes up + correct egress IP, then the egress-steering mechanism below (the hard part) and the kill-switch conformance test.

Pieces

• gluetun-gateway.yaml — the Gluetun Deployment (replicas: 1, Recreate) + the gluetun-control Service (control API :8000 for forwarded-port lookup + tunnel-state health). Kill-switch = FIREWALL=on.
• pia-credentials (SOPS) — PIA username/password only; Gluetun ships PIA's OpenVPN configs.

Operator hand-off (when deploying Phase 3)

1. Provision pia-credentials (see bootstrap/sops/secret-templates/pia-credentials.yaml.example, namespace vpn-egress, keys username/password) → sops -e -i → uncomment it in kustomization.yaml.
2. cluster-vars (already defined in clusters/base/cluster-vars.yaml; a ConfigMap — values are plain strings under data:, injected into the ${...} tokens by Flux postBuild.substituteFrom). Confirm or override:
   • PIA_REGION — a port-forwarding-capable PIA region (default Sweden; e.g. Netherlands, CA Toronto, Switzerland; NOT a US region — those lack PF). Gluetun SERVER_REGIONS.
   • CLUSTER_POD_CIDR — the Cilium cluster-pool pod CIDR (default 10.0.0.0/8; pods get 10.x, confirmed live — NOT the k3s 10.42.0.0/16). Gluetun FIREWALL_OUTBOUND_SUBNETS. Per-env overrides go in clusters/<env>/kustomization.yaml as a cluster-vars JSON patch (- op: replace, path: /data/PIA_REGION, value: "…"). With the self-managing root, a git push auto-applies (no manual kubectl apply -k).
3. Enable - vpn-egress in platform/kustomization.yaml (Phase 3).

Egress steering — the DESIGN DECISION (needs live validation)

The hard part: routing vpn-class pods through this ONE shared tunnel. Cilium Egress Gateway (CiliumEgressGatewayPolicy) SNATs matched pods out of a gateway node's interface — it does not natively route through a pod's tunnel. Options considered:

1. Cilium Egress Gateway → the Gluetun node's tunnel interface (recommended). Pin Gluetun to a node; a CiliumEgressGatewayPolicy selects pods labeled homelab.keithjackson.dev/egress-class=vpn and egresses them via that node's tunnel interface (tun0, OpenVPN). If the tunnel drops, tun0 is gone → traffic has no egress path → fail-closed (the kill-switch behavior R-NET-06 needs). This is the most Cilium-native fit but requires Gluetun's tunnel interface to be visible to the node datapath (hostNetwork Gluetun, or a shared netns) — the detail to VALIDATE.
2. Gateway-pod routing: client pods default-route to the Gluetun pod IP (init-container + NET_ADMIN, or a Multus secondary interface); Gluetun forwards+NATs via the tunnel. Works without Cilium egress gateway but is more moving parts and pod-IP-fragile.
3. Per-pod Gluetun sidecar — explicitly REJECTED by ADR-0013 (single shared gateway preferred).

FINDING (2026-08-23): option 1 is NOT viable with Gluetun — needs an architect decision

With the tunnel now live (custom-mode Gluetun, PIA Stockholm egress), option 1 fails on inspection: CiliumEgressGatewayPolicy interface: tun0 requires tun0 on the node, which means running Gluetun hostNetwork: true. But Gluetun owns its netns routing — on connect its [routing] logic adds 0.0.0.0/0 via the tunnel. In hostNetwork that would redirect the NODE's own default route through the VPN, breaking k3s/etcd/kubelet on that node. Gluetun has no supported "create tun0 but don't touch the default route" mode, so option 1 can't expose tun0 on the node safely. (The external-router-offload pattern some use — MikroTik/RouterOS policy routing per pod-subnet — is out of scope: ADR-0013 mandates an in-cluster gateway on a dumb SOHO edge.)

→ Escalated. The remaining viable path is option 2 (gateway-pod routing): egress-class=vpn pods default-route to the Gluetun pod via an init-container (+NET_ADMIN), and Gluetun forwards+NATs out tun0. More moving parts, and it requires Gluetun to forward for foreign pods (not its native same-netns model), so it needs validation. Kill-switch (R-NET-06) then relies on Gluetun's FIREWALL=on dropping non-tunnel egress. Fixtures for the egress-class/vpn-killswitch conformance tests are authored; the CiliumEgressGatewayPolicy / steering datapath is intentionally NOT committed until the mechanism is chosen

• validated, to avoid shipping an untested (and potentially node-breaking) datapath.

Port forwarding (qBittorrent) — follow-up

PIA assigns a forwarded port that can change on reconnect. A small port-sync helper (qbittorrent-natmap) reads the current port from the gluetun-control API (:8000 /v1/openvpn/portforwarded) and writes it into qBittorrent, keeping it hands-off (R-UPD). Authored alongside the qBittorrent app (Phase 6 apps / or here when qBittorrent lands).