home-network-k3s-impl/platform/vpn-egress/README.md

platform/vpn-egress — VPN egress steering (Phase 3, ADR-0015)

vpn-class services (sonarr, radarr, jackett, qbittorrent) egress only through a PIA VPN tunnel; internet-class services go out the normal WAN. Serves R-NET-05/06/07.

Egress is realized per ADR-0015 (spec docs/03): a node-level host OpenVPN split-tunnel on a labeled egress-gateway node, steered by a Cilium Egress Gateway policy, with a host nftables kill-switch. There is no in-cluster gateway pod — the tunnel is host infrastructure, not a Flux workload.

STATUS: live on BOTH clusters — k3s-dev-3 and, since 2026-09-12, k3s-prod-3. The egress-class (R-NET-05/07) and vpn-killswitch (R-NET-06) tests pass on dev. Caveat on that "conformance-green": the vpn-killswitch test stops and then restarts the gateway tunnel, and until 2026-09-12 a restarted tunnel never recovered (Cilium kept a stale egress ifindex — see below), so the test's recovery step could not have passed on the implementation as it stood. Whatever produced the earlier green result, re-run it.

How it works (ADR-0015)

Three layers, two owners:

• Tunnel — host OpenVPN (Ansible). A labeled egress-gateway node runs a host OpenVPN systemd service (PIA over OpenVPN-UDP — provider/protocol still per ADR-0013). It is split-tunnel (route-noexec + policy routing): it creates tun0 but never becomes the node default route, so the node's own traffic (control plane, image pulls, other pods) stays on the WAN. Managed by the Ansible role infra/ansible/roles/vpn_egress_gateway/, like k3s itself — not a cluster workload.
• Steering — Cilium Egress Gateway (GitOps). egress-steering.yaml holds a CiliumEgressGatewayPolicy that selects pods labeled homelab.keithjackson.dev/egress-class=vpn (Kyverno-enforced) and SNATs them out the gateway node's tun0. Native Cilium eBPF — no client-pod changes, no sidecar, no overlay. excludedCIDRs keeps RFC1918 / cluster / LAN traffic on the local datapath; only public destinations leave via the tunnel.
• Kill-switch — Cilium's own drop, NOT host nftables (corrected 2026-09-12). vpn-class fails closed because Cilium drops the traffic when it cannot use its cached egress interface. The host nftables table is retained for a narrower job. See "The kill-switch does not do what this file used to claim" below — that section is the measurement, not an opinion.

LIVE ON PROD since 2026-09-12 — k3s-prod-3

	dev	prod
gateway node	k3s-dev-3	k3s-prod-3
inventory group	vpn_egress_gateway	vpn_egress_gateway_prod (separate on purpose)
PIA server	158.173.241.112 (the .ovpn's pin)	158.173.241.135 (group_vars override — see below)
vpn-class egress IP	158.173.241.36	158.173.241.11
internet-class egress IP	home WAN	136.51.115.252 (home WAN)

Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three nodes — all egress the PIA address; plex and nextcloud (internet-class) still egress the home WAN. Before the gateway existed, prod's vpn-class pods had no internet at all: the CEGP was reconciled with no node matching its nodeSelector, so cilium-dbg bpf egress list showed Gateway IP: Not Found for their 0.0.0.0/0 entry and the traffic was dropped. Arming the gateway restored their egress rather than moving it from the WAN to the tunnel — which also means nothing had been leaking in the meantime.

One PIA account is fine. One pinned PIA server is not.

Standing prod up broke dev, and the reason is not the connection limit. The vaulted .ovpn pins one server, so both gateways dialled it — and PIA handed both sessions the same tunnel address. Verbatim from both journals:

[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198
PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ...

Each reconnect steals 10.72.192.7 on that server; the loser receives nothing, trips --ping-restart after 60 s, reconnects and steals it back. Measured cycle: dev restarting at 22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC with prod interleaved about a minute off — every ~2 minutes, indefinitely. Neither environment had reliable vpn-class egress; prod's first probes looked green only because they landed inside an up-window.

Dev recovered the minute prod left .112: its last Inactivity timeout is 23:05:06 UTC, and at 23:12:03 that same session was still up — seven minutes unbroken against a two-minute cycle — with a source-bound ping returning 4/4 at 146 ms and vpn-class pod egress back to 158.173.241.36.

Fix: vpn_egress_gateway_pia_remote in group_vars/vpn_egress_gateway_prod/vars.yml rewrites just the remote line, pointing prod at 158.173.241.135 — another member of the same PIA Stockholm LB pool (sweden.privacy.network resolves to 158.173.241.{34,55,112,135,239}; every server in it takes the same credentials and CA). Dev keeps .112.

So: one gateway per PIA server. A third gateway, or a rebuild that re-pins a server, must pick an unused address from the pool. A separate prod PIA account would also solve it — see infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md.

That switch also produced the cleanest possible live proof of the re-steer path, for free: prod's tunnel address changed from 10.72.192.7 (ifindex 128) to 10.150.192.86 (ifindex 129) — a real new PIA address on a real reconnect, the exact case Cilium does not follow. The egress-uplink label moved to 10.150.192.86-129 on its own and all four vpn-class workloads kept egressing, now via 158.173.241.11. No manual step, no CEGP edit.

The kill-switch does not do what this file used to claim (measured 2026-09-12)

This README, egress-steering.yaml, and the role all used to say the host nftables kill-switch is the authoritative fail-closed backstop for vpn-class traffic. It cannot be. With bpf.masquerade: true — which Egress Gateway requires — the SNAT and the egress happen in eBPF, which bpf_redirect()s the skb straight to the egress device. netfilter POSTROUTING is never invoked for those packets.

A counters-only nft table installed alongside the real one on the gateway node, while a vpn-class pod egressed:

counter	packets
forward, saddr = pod CIDR	0
forward, saddr = PIA tunnel range	0
postrouting, oif tun0, saddr = tunnel range	0
postrouting, oif WAN, saddr = pod CIDR	0
postrouting, oif WAN, any (control: the hook IS live)	45166
postrouting, oif tun0, saddr = tunnel (control: a node-originated curl)	16

What actually fails closed is Cilium itself. When it cannot use its cached egress interface/IP it drops the packet. Verified with tcpdump on both the WAN and tun0, during a tunnel stop and during a tunnel restart: 0 packets left either device. There is no leak — the guarantee is real — but the mechanism is Cilium's, and the existing vpn-killswitch conformance test passes because of it, not because of the nft rules.

The nft table is kept, with a counter on every rule, for the narrower thing it genuinely does cover: node-originated traffic sourced from a tunnel address. That was proven on k3s-prod-3 in both directions — a source-bound ping with the tunnel ip-rule present got 3/3 replies and moved killswitch_accept; with the ip-rule removed (so the routing table sent it out eno1) killswitch_drop moved 0 -> 3 packets / 252 bytes and the ping got nothing back. If killswitch_drop ever starts counting in normal operation, the datapath has changed — read that as news, not reassurance.

Also removed: the meta mark 0x1c8 oifname != tun0 drop "contract hook". Nothing stamps that mark (the only Cilium mark on the gateway node is 0x200/0xf00, the L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is a hazard to whatever component picks that value next. The role's vpn_egress_gateway_fwmark variable is gone with it, and the "add the fwmark to the interface contract" escalation in the role README is closed as not-implementable rather than left open.

Cilium never re-resolves tun0 — the gateway dies on every reconnect without help

Cilium resolves interface: tun0 once into a cached (egress IP, egress ifindex) and does not revisit it. Every tunnel restart re-creates tun0 with a new ifindex, so the cached one goes stale and vpn-class egress is silently blackholed. Found the hard way: the dev gateway had been dead for ~80 minutes when this work started, with no alert and nothing in the ledger.

egress-gateway-reconciliation-trigger-interval: 1s does not help; both clusters carry that setting and dev still did not self-heal 20s+ after the tunnel was back. It coalesces reconciliations that k8s object events already triggered — netlink link events are not among them.

The Ansible role now publishes homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex> on the gateway node on every connect, plus a 2-minute convergence timer. That node update makes Cilium re-process the policy and re-resolve the interface (verified: cached ifindex 1986 -> 1987 within 10s, egress recovered immediately). A node label rather than an annotation on this Flux-managed policy, so GitOps state is never written to from a node.

Debugging a dead gateway starts here:

kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink
kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
  cilium-dbg bpf egress list      # Egress Ifindex must match the node's live `ip link show tun0`

If they disagree, the re-steer path is broken — journalctl -u vpn-egress-resteer on the gateway node is the next stop.

And now something watches it (2026-09-12)

The self-heal closed the datapath hole; it did not close the monitoring one, and a mechanism that silently stops working is the same outage again. Two rules in platform/monitoring/alerts.yaml:

• VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was re-created more times in the last hour than the egress-uplink label changed, i.e. Cilium was never told about a reconnect. This is the case VPNTunnelDown cannot see: tun0 present, bytes flowing, steering pointed at a destroyed ifindex.
• VPNEgressUplinkLabelMissing (warning, for: 30m) — fires when a node carries egress-gateway=true but no egress-uplink label reaches kube-state-metrics, which is the state in which the rule above would be inert. The watchdog for the watchdog.

Both are gated on a gateway node existing (the same kube_node_labels guard VPNTunnelDown uses), so a cluster with no gateway cannot be made to fire them. egress-uplink had to be added to kube-state-metrics.metricLabelsAllowlist in platform/monitoring/kube-prometheus-stack.yaml for the label to exist as a metric at all — the two changes must ship together.

What still is not watched: "the label was written and Cilium still ignored it." Cilium OSS exposes no metric carrying the egress map's contents (the map is readable only through cilium-dbg bpf egress list), and the cilium-agent is not scraped on either cluster in the first place — no cilium_* or hubble_* series exist. Closing that last gap means either exporting the map's ifindex from the re-steer script through a node-exporter textfile collector, or a black-box probe from a vpn-class pod. Neither exists yet.

Resilience — there is NO gateway failover (spec correction, escalated)

ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover) provide the active/standby the earlier ADRs asked for", and spec docs/03 repeats it. That feature does not exist in Cilium OSS. The supported topology is exactly one gateway node.

Verified against the CRD actually deployed on this cluster (kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json, apiVersion cilium.io/v2, Cilium v1.20.1) — the field's own description:

egressGateway — "EgressGateway is the gateway node responsible for SNATing traffic. In case multiple nodes are a match for the given set of labels, the first node in lexical ordering based on their name will be selected."

Consequences, stated plainly:

• One gateway node is the supported topology. infra/ansible/inventory/hosts.ini's [vpn_egress_gateway] group is singleton by design.
• A second labelled node does NOT provide failover. It is not a standby; it carries no traffic and takes over nothing when the first node dies. Gateway-node loss = vpn-class egress down (fail-closed, per R-NET-06 — no leak, but no service).
• Lexical ordering decides who wins if two are ever labelled. Labelling k3s-prod-1 while k3s-prod-3 is the live gateway would silently move the role to k3s-prod-1 — because 1 sorts before 3 — taking the tunnel and the public egress IP with it, with no event and no alert.
• The plural egressGateways field (Cilium 1.20) is not a workaround. It shards endpoints across entries by CiliumEndpoint UID rather than running active/standby: losing one entry's node blackholes that entry's share and re-shards the remainder (breaking live connections), and each shard leaves from a different public IP. Upstream confirmation: cilium/cilium#30157 ("egress gateway doesn't support failover between nodes") and cilium/cilium#18230 ("egress gateway does not tolerate single node failure").

Governance: ADR-0015 and docs/03 are specifications and are immutable to agents (spec AGENTS.md §11, R-TEST-06, enforced by ci/spec-immutability.sh) — they are not edited here. The correction is raised as an escalation with a drafted ADR-0016 supersede in DECISIONS-LOG.md (entry of 2026-09-06) for the operator/architect to accept. Until then this file and egress-steering.yaml are the accurate statement of what the implementation can actually do.

Secrets — host-side now

PIA credentials + the .ovpn live on the gateway node via Ansible vault (see spec docs/07), not a cluster SOPS Secret. SOPS still governs in-cluster secrets; VPN egress simply no longer needs one.

The pia-credentials.sops.yaml and pia-custom-ovpn.sops.yaml secrets in this directory are now UNUSED — leftovers from the retired Gluetun pod, retained pending a separate cleanup change.

Pieces in this directory

• namespace.yaml — the vpn-egress namespace (now holds only the Cilium policy + the retained, unused secrets; no privileged gateway pod under ADR-0015).
• egress-steering.yaml — the CiliumEgressGatewayPolicy (the steering datapath).
• pia-credentials.sops.yaml, pia-custom-ovpn.sops.yaml — unused, retained pending cleanup.

History (brief)

ADR-0013 originally placed the VPN in a single shared Gluetun pod and deferred the steering mechanism. Implementation proved that a Gluetun-in-a-pod tunnel cannot be a Cilium Egress Gateway target: the policy SNATs out a node interface, but the pod's tun0 lives in the pod netns, and running Gluetun hostNetwork to expose tun0 on the node hijacks the node's default route (breaking k3s/etcd/kubelet). ADR-0015 resolved this by moving the tunnel to the node (host OpenVPN split-tunnel) and steering with Cilium's native egress gateway. See ADR-0015 for the full rationale.

Port forwarding (qBittorrent) — follow-up

PIA assigns a forwarded port that can change on reconnect. Under ADR-0015 this is re-solved host-side (a small pf/natpmp helper on the gateway node that publishes the assigned port to the app) — Phase 6, alongside the qBittorrent app.