# platform/vpn-egress — VPN egress steering (Phase 3, ADR-0015) `vpn`-class services (sonarr, radarr, jackett, qbittorrent) egress **only** through a PIA VPN tunnel; `internet`-class services go out the normal WAN. Serves **R-NET-05/06/07**. Egress is realized per **[ADR-0015](../../../home-network-k3s/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md)** (spec `docs/03`): a **node-level host OpenVPN split-tunnel** on a labeled egress-gateway node, steered by a **Cilium Egress Gateway** policy, with a **host nftables kill-switch**. There is **no in-cluster gateway pod** — the tunnel is host infrastructure, not a Flux workload. > **STATUS: live on BOTH clusters** — `k3s-dev-3` and, since 2026-09-12, `k3s-prod-3`. > The `egress-class` (R-NET-05/07) and `vpn-killswitch` (R-NET-06) tests pass on dev. > **Caveat on that "conformance-green":** the `vpn-killswitch` test stops and then restarts the > gateway tunnel, and until 2026-09-12 a restarted tunnel never recovered (Cilium kept a stale > egress ifindex — see below), so the test's recovery step could not have passed on the > implementation as it stood. Whatever produced the earlier green result, re-run it. ## How it works (ADR-0015) Three layers, two owners: - **Tunnel — host OpenVPN (Ansible).** A labeled egress-gateway node runs a **host OpenVPN systemd service** (PIA over OpenVPN-UDP — provider/protocol still per ADR-0013). It is **split-tunnel** (`route-noexec` + policy routing): it creates `tun0` but never becomes the node default route, so the node's own traffic (control plane, image pulls, other pods) stays on the WAN. Managed by the Ansible role **`infra/ansible/roles/vpn_egress_gateway/`**, like k3s itself — not a cluster workload. - **Steering — Cilium Egress Gateway (GitOps).** `egress-steering.yaml` holds a `CiliumEgressGatewayPolicy` that selects pods labeled `homelab.keithjackson.dev/egress-class=vpn` (Kyverno-enforced) and SNATs them out the gateway node's `tun0`. Native Cilium eBPF — no client-pod changes, no sidecar, no overlay. `excludedCIDRs` keeps RFC1918 / cluster / LAN traffic on the local datapath; only public destinations leave via the tunnel. - **Kill-switch — Cilium's own drop, NOT host nftables (corrected 2026-09-12).** vpn-class fails closed because Cilium drops the traffic when it cannot use its cached egress interface. The host nftables table is retained for a narrower job. See "The kill-switch does not do what this file used to claim" below — that section is the measurement, not an opinion. ## LIVE ON PROD since 2026-09-12 — `k3s-prod-3` | | dev | prod | |---|---|---| | gateway node | `k3s-dev-3` | `k3s-prod-3` | | inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) | | PIA server | `158.173.241.112` (the .ovpn's pin) | `158.173.241.135` (group_vars override — see below) | | vpn-class egress IP | `158.173.241.36` | `158.173.241.11` | | internet-class egress IP | home WAN | `136.51.115.252` (home WAN) | Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three nodes — all egress the PIA address; plex and nextcloud (internet-class) still egress the home WAN. **Before the gateway existed, prod's vpn-class pods had no internet at all**: the CEGP was reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress list` showed `Gateway IP: Not Found` for their `0.0.0.0/0` entry and the traffic was dropped. Arming the gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also means nothing had been leaking in the meantime. ### One PIA account is fine. One pinned PIA *server* is not. Standing prod up broke dev, and the reason is not the connection limit. The vaulted `.ovpn` pins **one** server, so both gateways dialled it — and PIA handed **both sessions the same tunnel address**. Verbatim from both journals: ``` [Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198 PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ... ``` Each reconnect steals `10.72.192.7` on that server; the loser receives nothing, trips `--ping-restart` after 60 s, reconnects and steals it back. Measured cycle: dev restarting at 22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC with prod interleaved about a minute off — every ~2 minutes, indefinitely. Neither environment had reliable vpn-class egress; prod's first probes looked green only because they landed inside an up-window. Dev recovered the minute prod left `.112`: its last `Inactivity timeout` is 23:05:06 UTC, and at 23:12:03 that same session was still up — seven minutes unbroken against a two-minute cycle — with a source-bound ping returning 4/4 at 146 ms and vpn-class pod egress back to `158.173.241.36`. **Fix:** `vpn_egress_gateway_pia_remote` in `group_vars/vpn_egress_gateway_prod/vars.yml` rewrites just the `remote` line, pointing prod at `158.173.241.135` — another member of the same PIA Stockholm LB pool (`sweden.privacy.network` resolves to `158.173.241.{34,55,112,135,239}`; every server in it takes the same credentials and CA). Dev keeps `.112`. **So: one gateway per PIA server.** A third gateway, or a rebuild that re-pins a server, must pick an unused address from the pool. A separate prod PIA account would also solve it — see `infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`. That switch also produced the cleanest possible live proof of the re-steer path, for free: prod's tunnel address changed from `10.72.192.7` (ifindex 128) to `10.150.192.86` (ifindex 129) — a real new PIA address on a real reconnect, the exact case Cilium does not follow. The `egress-uplink` label moved to `10.150.192.86-129` on its own and all four vpn-class workloads kept egressing, now via `158.173.241.11`. No manual step, no CEGP edit. ## The kill-switch does not do what this file used to claim (measured 2026-09-12) This README, `egress-steering.yaml`, and the role all used to say the **host nftables kill-switch** is the authoritative fail-closed backstop for `vpn`-class traffic. **It cannot be.** With `bpf.masquerade: true` — which Egress Gateway requires — the SNAT and the egress happen in eBPF, which `bpf_redirect()`s the skb straight to the egress device. netfilter POSTROUTING is never invoked for those packets. A counters-only nft table installed alongside the real one on the gateway node, while a vpn-class pod egressed: | counter | packets | |---|---| | `forward`, saddr = pod CIDR | **0** | | `forward`, saddr = PIA tunnel range | **0** | | `postrouting`, oif tun0, saddr = tunnel range | **0** | | `postrouting`, oif WAN, saddr = pod CIDR | **0** | | `postrouting`, oif WAN, any (control: the hook IS live) | 45166 | | `postrouting`, oif tun0, saddr = tunnel (control: a node-originated curl) | 16 | **What actually fails closed is Cilium itself.** When it cannot use its cached egress interface/IP it drops the packet. Verified with tcpdump on *both* the WAN and tun0, during a tunnel stop and during a tunnel restart: **0 packets left either device.** There is no leak — the guarantee is real — but the mechanism is Cilium's, and the existing `vpn-killswitch` conformance test passes because of it, not because of the nft rules. The nft table is **kept**, with a `counter` on every rule, for the narrower thing it genuinely does cover: **node-originated** traffic sourced from a tunnel address. That was proven on `k3s-prod-3` in both directions — a source-bound ping with the tunnel ip-rule present got 3/3 replies and moved `killswitch_accept`; with the ip-rule removed (so the routing table sent it out `eno1`) `killswitch_drop` moved **0 -> 3 packets / 252 bytes** and the ping got nothing back. If `killswitch_drop` ever starts counting in normal operation, the datapath has changed — read that as news, not reassurance. Also removed: the `meta mark 0x1c8 oifname != tun0 drop` "contract hook". Nothing stamps that mark (the only Cilium mark on the gateway node is `0x200/0xf00`, the L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is a hazard to whatever component picks that value next. The role's `vpn_egress_gateway_fwmark` variable is gone with it, and the "add the fwmark to the interface contract" escalation in the role README is **closed as not-implementable** rather than left open. ## Cilium never re-resolves `tun0` — the gateway dies on every reconnect without help Cilium resolves `interface: tun0` once into a cached (egress IP, egress ifindex) and does not revisit it. Every tunnel restart re-creates tun0 with a **new ifindex**, so the cached one goes stale and vpn-class egress is silently blackholed. Found the hard way: the dev gateway had been dead for ~80 minutes when this work started, with no alert and nothing in the ledger. `egress-gateway-reconciliation-trigger-interval: 1s` does **not** help; both clusters carry that setting and dev still did not self-heal 20s+ after the tunnel was back. It coalesces reconciliations that k8s object events already triggered — netlink link events are not among them. The Ansible role now publishes `homelab.keithjackson.dev/egress-uplink=-` on the gateway node on every connect, plus a 2-minute convergence timer. That node update makes Cilium re-process the policy and re-resolve the interface (verified: cached ifindex 1986 -> 1987 within 10s, egress recovered immediately). A node label rather than an annotation on this Flux-managed policy, so GitOps state is never written to from a node. **Debugging a dead gateway starts here:** ```bash kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink kubectl -n kube-system exec -c cilium-agent -- \ cilium-dbg bpf egress list # Egress Ifindex must match the node's live `ip link show tun0` ``` If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the gateway node is the next stop. ### And now something watches it (2026-09-12) The self-heal closed the datapath hole; it did not close the monitoring one, and a mechanism that silently stops working is the same outage again. Two rules in `platform/monitoring/alerts.yaml`: - **`VPNEgressSteeringStale`** (critical, `for: 15m`) — fires when the live tun0 was re-created more times in the last hour than the `egress-uplink` label changed, i.e. Cilium was never told about a reconnect. This is the case `VPNTunnelDown` cannot see: tun0 present, bytes flowing, steering pointed at a destroyed ifindex. - **`VPNEgressUplinkLabelMissing`** (warning, `for: 30m`) — fires when a node carries `egress-gateway=true` but no `egress-uplink` label reaches kube-state-metrics, which is the state in which the rule above would be inert. The watchdog for the watchdog. Both are gated on a gateway node existing (the same `kube_node_labels` guard `VPNTunnelDown` uses), so a cluster with no gateway cannot be made to fire them. `egress-uplink` had to be added to `kube-state-metrics.metricLabelsAllowlist` in `platform/monitoring/kube-prometheus-stack.yaml` for the label to exist as a metric at all — the two changes must ship together. **What still is not watched:** "the label was written and Cilium still ignored it." Cilium OSS exposes no metric carrying the egress map's contents (the map is readable only through `cilium-dbg bpf egress list`), and the cilium-agent is not scraped on either cluster in the first place — no `cilium_*` or `hubble_*` series exist. Closing that last gap means either exporting the map's ifindex from the re-steer script through a node-exporter textfile collector, or a black-box probe from a vpn-class pod. Neither exists yet. ## Resilience — there is NO gateway failover (spec correction, escalated) **ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover) provide the active/standby the earlier ADRs asked for", and spec `docs/03` repeats it. That feature does not exist in Cilium OSS.** The supported topology is **exactly one gateway node**. Verified against the CRD actually deployed on this cluster (`kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`, apiVersion `cilium.io/v2`, Cilium **v1.20.1**) — the field's own description: > **`egressGateway`** — "EgressGateway is the gateway node responsible for SNATing traffic. > **In case multiple nodes are a match for the given set of labels, the first node in lexical > ordering based on their name will be selected.**" Consequences, stated plainly: - **One gateway node is the supported topology.** `infra/ansible/inventory/hosts.ini`'s `[vpn_egress_gateway]` group is singleton by design. - **A second labelled node does NOT provide failover.** It is not a standby; it carries no traffic and takes over nothing when the first node dies. Gateway-node loss = `vpn`-class egress down (fail-closed, per R-NET-06 — no leak, but no service). - **Lexical ordering decides who wins if two are ever labelled.** Labelling `k3s-prod-1` while `k3s-prod-3` is the live gateway would **silently** move the role to `k3s-prod-1` — because `1` sorts before `3` — taking the tunnel and the public egress IP with it, with no event and no alert. - **The plural `egressGateways` field (Cilium 1.20) is not a workaround.** It *shards* endpoints across entries by CiliumEndpoint UID rather than running active/standby: losing one entry's node blackholes that entry's share and re-shards the remainder (breaking live connections), and each shard leaves from a *different* public IP. Upstream confirmation: [cilium/cilium#30157](https://github.com/cilium/cilium/issues/30157) ("egress gateway doesn't support failover between nodes") and [cilium/cilium#18230](https://github.com/cilium/cilium/issues/18230) ("egress gateway does not tolerate single node failure"). **Governance:** ADR-0015 and `docs/03` are specifications and are immutable to agents (spec `AGENTS.md` §11, R-TEST-06, enforced by `ci/spec-immutability.sh`) — they are **not** edited here. The correction is raised as an **escalation with a drafted ADR-0016 supersede** in `DECISIONS-LOG.md` (entry of 2026-09-06) for the operator/architect to accept. Until then this file and `egress-steering.yaml` are the accurate statement of what the implementation can actually do. ## Secrets — host-side now PIA credentials + the `.ovpn` live **on the gateway node via Ansible vault** (see spec `docs/07`), **not** a cluster SOPS Secret. SOPS still governs in-cluster secrets; VPN egress simply no longer needs one. The `pia-credentials.sops.yaml` and `pia-custom-ovpn.sops.yaml` secrets in this directory are **now UNUSED** — leftovers from the retired Gluetun pod, **retained pending a separate cleanup change**. ## Pieces in this directory - `namespace.yaml` — the `vpn-egress` namespace (now holds only the Cilium policy + the retained, unused secrets; no privileged gateway pod under ADR-0015). - `egress-steering.yaml` — the `CiliumEgressGatewayPolicy` (the steering datapath). - `pia-credentials.sops.yaml`, `pia-custom-ovpn.sops.yaml` — **unused**, retained pending cleanup. ## History (brief) ADR-0013 originally placed the VPN in a single shared **Gluetun pod** and deferred the steering mechanism. Implementation proved that a Gluetun-in-a-pod tunnel **cannot be a Cilium Egress Gateway target**: the policy SNATs out a *node* interface, but the pod's `tun0` lives in the pod netns, and running Gluetun `hostNetwork` to expose `tun0` on the node hijacks the node's default route (breaking k3s/etcd/kubelet). ADR-0015 resolved this by moving the tunnel to the **node** (host OpenVPN split-tunnel) and steering with Cilium's native egress gateway. See ADR-0015 for the full rationale. ## Port forwarding (qBittorrent) — follow-up PIA assigns a forwarded port that can change on reconnect. Under ADR-0015 this is re-solved **host-side** (a small pf/natpmp helper on the gateway node that publishes the assigned port to the app) — Phase 6, alongside the qBittorrent app.