# platform/vpn-egress — VPN egress steering (Phase 3, ADR-0015) `vpn`-class services (sonarr, radarr, jackett, qbittorrent) egress **only** through a PIA VPN tunnel; `internet`-class services go out the normal WAN. Serves **R-NET-05/06/07**. Egress is realized per **[ADR-0015](../../../home-network-k3s/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md)** (spec `docs/03`): a **node-level host OpenVPN split-tunnel** on a labeled egress-gateway node, steered by a **Cilium Egress Gateway** policy, with a **host nftables kill-switch**. There is **no in-cluster gateway pod** — the tunnel is host infrastructure, not a Flux workload. > **STATUS: live on BOTH clusters** — `k3s-dev-3` and, since 2026-09-12, `k3s-prod-3`. > The `egress-class` (R-NET-05/07) and `vpn-killswitch` (R-NET-06) tests pass on dev. > **Caveat on that "conformance-green":** the `vpn-killswitch` test stops and then restarts the > gateway tunnel, and until 2026-09-12 a restarted tunnel never recovered (Cilium kept a stale > egress ifindex — see below), so the test's recovery step could not have passed on the > implementation as it stood. Whatever produced the earlier green result, re-run it. ## How it works (ADR-0015) Three layers, two owners: - **Tunnel — host OpenVPN (Ansible).** A labeled egress-gateway node runs a **host OpenVPN systemd service** (PIA over OpenVPN-UDP — provider/protocol still per ADR-0013). It is **split-tunnel** (`route-noexec` + policy routing): it creates `tun0` but never becomes the node default route, so the node's own traffic (control plane, image pulls, other pods) stays on the WAN. Managed by the Ansible role **`infra/ansible/roles/vpn_egress_gateway/`**, like k3s itself — not a cluster workload. - **Steering — Cilium Egress Gateway (GitOps).** `egress-steering.yaml` holds a `CiliumEgressGatewayPolicy` that selects pods labeled `homelab.keithjackson.dev/egress-class=vpn` (Kyverno-enforced) and SNATs them out the gateway node's `tun0`. Native Cilium eBPF — no client-pod changes, no sidecar, no overlay. `excludedCIDRs` keeps RFC1918 / cluster / LAN traffic on the local datapath; only public destinations leave via the tunnel. - **Kill-switch — Cilium's own drop, NOT host nftables (corrected 2026-09-12).** vpn-class fails closed because Cilium drops the traffic when it cannot use its cached egress interface. The host nftables table is retained for a narrower job. See "The kill-switch does not do what this file used to claim" below — that section is the measurement, not an opinion. ## LIVE ON PROD since 2026-09-12 — `k3s-prod-3` | | dev | prod | |---|---|---| | gateway node | `k3s-dev-3` | `k3s-prod-3` | | inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) | | vpn-class egress IP | `158.173.241.36` (PIA Stockholm) | `158.173.241.36` (PIA Stockholm) | | internet-class egress IP | home WAN | `136.51.115.252` (home WAN) | Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three nodes — all egress the PIA address; plex and nextcloud (internet-class) still egress the home WAN. **Before the gateway existed, prod's vpn-class pods had no internet at all**: the CEGP was reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress list` showed `Gateway IP: Not Found` for their `0.0.0.0/0` entry and the traffic was dropped. Arming the gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also means nothing had been leaking in the meantime. Both environments currently share one PIA account and, as it happens, one public egress IP. That is inside the account's connection limit, but it means the two environments are not distinguishable from the outside; a separate prod PIA account is the clean fix if that ever matters (see `infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`). ## The kill-switch does not do what this file used to claim (measured 2026-09-12) This README, `egress-steering.yaml`, and the role all used to say the **host nftables kill-switch** is the authoritative fail-closed backstop for `vpn`-class traffic. **It cannot be.** With `bpf.masquerade: true` — which Egress Gateway requires — the SNAT and the egress happen in eBPF, which `bpf_redirect()`s the skb straight to the egress device. netfilter POSTROUTING is never invoked for those packets. A counters-only nft table installed alongside the real one on the gateway node, while a vpn-class pod egressed: | counter | packets | |---|---| | `forward`, saddr = pod CIDR | **0** | | `forward`, saddr = PIA tunnel range | **0** | | `postrouting`, oif tun0, saddr = tunnel range | **0** | | `postrouting`, oif WAN, saddr = pod CIDR | **0** | | `postrouting`, oif WAN, any (control: the hook IS live) | 45166 | | `postrouting`, oif tun0, saddr = tunnel (control: a node-originated curl) | 16 | **What actually fails closed is Cilium itself.** When it cannot use its cached egress interface/IP it drops the packet. Verified with tcpdump on *both* the WAN and tun0, during a tunnel stop and during a tunnel restart: **0 packets left either device.** There is no leak — the guarantee is real — but the mechanism is Cilium's, and the existing `vpn-killswitch` conformance test passes because of it, not because of the nft rules. The nft table is **kept**, with a `counter` on every rule, for the narrower thing it genuinely does cover: **node-originated** traffic sourced from a tunnel address. That was proven on `k3s-prod-3` in both directions — a source-bound ping with the tunnel ip-rule present got 3/3 replies and moved `killswitch_accept`; with the ip-rule removed (so the routing table sent it out `eno1`) `killswitch_drop` moved **0 -> 3 packets / 252 bytes** and the ping got nothing back. If `killswitch_drop` ever starts counting in normal operation, the datapath has changed — read that as news, not reassurance. Also removed: the `meta mark 0x1c8 oifname != tun0 drop` "contract hook". Nothing stamps that mark (the only Cilium mark on the gateway node is `0x200/0xf00`, the L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is a hazard to whatever component picks that value next. The role's `vpn_egress_gateway_fwmark` variable is gone with it, and the "add the fwmark to the interface contract" escalation in the role README is **closed as not-implementable** rather than left open. ## Cilium never re-resolves `tun0` — the gateway dies on every reconnect without help Cilium resolves `interface: tun0` once into a cached (egress IP, egress ifindex) and does not revisit it. Every tunnel restart re-creates tun0 with a **new ifindex**, so the cached one goes stale and vpn-class egress is silently blackholed. Found the hard way: the dev gateway had been dead for ~80 minutes when this work started, with no alert and nothing in the ledger. `egress-gateway-reconciliation-trigger-interval: 1s` does **not** help; both clusters carry that setting and dev still did not self-heal 20s+ after the tunnel was back. It coalesces reconciliations that k8s object events already triggered — netlink link events are not among them. The Ansible role now publishes `homelab.keithjackson.dev/egress-uplink=-` on the gateway node on every connect, plus a 2-minute convergence timer. That node update makes Cilium re-process the policy and re-resolve the interface (verified: cached ifindex 1986 -> 1987 within 10s, egress recovered immediately). A node label rather than an annotation on this Flux-managed policy, so GitOps state is never written to from a node. **Debugging a dead gateway starts here:** ```bash kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink kubectl -n kube-system exec -c cilium-agent -- \ cilium-dbg bpf egress list # Egress Ifindex must match the node's live `ip link show tun0` ``` If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the gateway node is the next stop. ## Resilience — there is NO gateway failover (spec correction, escalated) **ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover) provide the active/standby the earlier ADRs asked for", and spec `docs/03` repeats it. That feature does not exist in Cilium OSS.** The supported topology is **exactly one gateway node**. Verified against the CRD actually deployed on this cluster (`kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`, apiVersion `cilium.io/v2`, Cilium **v1.20.1**) — the field's own description: > **`egressGateway`** — "EgressGateway is the gateway node responsible for SNATing traffic. > **In case multiple nodes are a match for the given set of labels, the first node in lexical > ordering based on their name will be selected.**" Consequences, stated plainly: - **One gateway node is the supported topology.** `infra/ansible/inventory/hosts.ini`'s `[vpn_egress_gateway]` group is singleton by design. - **A second labelled node does NOT provide failover.** It is not a standby; it carries no traffic and takes over nothing when the first node dies. Gateway-node loss = `vpn`-class egress down (fail-closed, per R-NET-06 — no leak, but no service). - **Lexical ordering decides who wins if two are ever labelled.** Labelling `k3s-prod-1` while `k3s-prod-3` is the live gateway would **silently** move the role to `k3s-prod-1` — because `1` sorts before `3` — taking the tunnel and the public egress IP with it, with no event and no alert. - **The plural `egressGateways` field (Cilium 1.20) is not a workaround.** It *shards* endpoints across entries by CiliumEndpoint UID rather than running active/standby: losing one entry's node blackholes that entry's share and re-shards the remainder (breaking live connections), and each shard leaves from a *different* public IP. Upstream confirmation: [cilium/cilium#30157](https://github.com/cilium/cilium/issues/30157) ("egress gateway doesn't support failover between nodes") and [cilium/cilium#18230](https://github.com/cilium/cilium/issues/18230) ("egress gateway does not tolerate single node failure"). **Governance:** ADR-0015 and `docs/03` are specifications and are immutable to agents (spec `AGENTS.md` §11, R-TEST-06, enforced by `ci/spec-immutability.sh`) — they are **not** edited here. The correction is raised as an **escalation with a drafted ADR-0016 supersede** in `DECISIONS-LOG.md` (entry of 2026-09-06) for the operator/architect to accept. Until then this file and `egress-steering.yaml` are the accurate statement of what the implementation can actually do. ## Secrets — host-side now PIA credentials + the `.ovpn` live **on the gateway node via Ansible vault** (see spec `docs/07`), **not** a cluster SOPS Secret. SOPS still governs in-cluster secrets; VPN egress simply no longer needs one. The `pia-credentials.sops.yaml` and `pia-custom-ovpn.sops.yaml` secrets in this directory are **now UNUSED** — leftovers from the retired Gluetun pod, **retained pending a separate cleanup change**. ## Pieces in this directory - `namespace.yaml` — the `vpn-egress` namespace (now holds only the Cilium policy + the retained, unused secrets; no privileged gateway pod under ADR-0015). - `egress-steering.yaml` — the `CiliumEgressGatewayPolicy` (the steering datapath). - `pia-credentials.sops.yaml`, `pia-custom-ovpn.sops.yaml` — **unused**, retained pending cleanup. ## History (brief) ADR-0013 originally placed the VPN in a single shared **Gluetun pod** and deferred the steering mechanism. Implementation proved that a Gluetun-in-a-pod tunnel **cannot be a Cilium Egress Gateway target**: the policy SNATs out a *node* interface, but the pod's `tun0` lives in the pod netns, and running Gluetun `hostNetwork` to expose `tun0` on the node hijacks the node's default route (breaking k3s/etcd/kubelet). ADR-0015 resolved this by moving the tunnel to the **node** (host OpenVPN split-tunnel) and steering with Cilium's native egress gateway. See ADR-0015 for the full rationale. ## Port forwarding (qBittorrent) — follow-up PIA assigns a forwarded port that can change on reconnect. Under ADR-0015 this is re-solved **host-side** (a small pf/natpmp helper on the gateway node that publishes the assigned port to the app) — Phase 6, alongside the qBittorrent app.