From 34f41a0fbae5185d904212b41673381bd1956ca6 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 24 Aug 2026 00:26:26 -0500 Subject: [PATCH] phase3(vpn): egress-class conformance fixtures + steering design finding Author the internet/vpn egress-class fixture pods for the egress-class + vpn-killswitch conformance tests. Record the datapath finding: ADR-0013's preferred steering (Cilium egress gateway -> gluetun tun0) is NOT viable because gluetun must run hostNetwork to put tun0 on the node, and gluetun then redirects the NODE's default route through the VPN (breaks k3s/etcd). Escalated: remaining viable path is option 2 (gateway-pod routing). No steering datapath committed until the mechanism is chosen + validated. Co-Authored-By: Claude Opus 4.8 --- .../fixtures/internet-and-vpn-pods.yaml | 36 +++++++++++++++++++ platform/vpn-egress/README.md | 22 +++++++++--- 2 files changed, 53 insertions(+), 5 deletions(-) create mode 100644 conformance/tests/l2-behavioral/egress-class/fixtures/internet-and-vpn-pods.yaml diff --git a/conformance/tests/l2-behavioral/egress-class/fixtures/internet-and-vpn-pods.yaml b/conformance/tests/l2-behavioral/egress-class/fixtures/internet-and-vpn-pods.yaml new file mode 100644 index 0000000..d6738a1 --- /dev/null +++ b/conformance/tests/l2-behavioral/egress-class/fixtures/internet-and-vpn-pods.yaml @@ -0,0 +1,36 @@ +# Two egress-class workloads for the egress-class + vpn-killswitch conformance tests. +# `internet` egresses the normal WAN; `vpn` is steered through the gluetun tunnel (CiliumEgressGatewayPolicy +# in platform/vpn-egress/egress-steering.yaml). Both just need curl to hit an echo-IP service. +apiVersion: v1 +kind: Pod +metadata: + name: egress-internet + labels: + app: egress-test + homelab.keithjackson.dev/egress-class: internet +spec: + containers: + - name: curl + image: curlimages/curl:8.11.1 + command: ["sleep", "3600"] + resources: + requests: { cpu: 10m, memory: 16Mi } + limits: { memory: 32Mi } + restartPolicy: Never +--- +apiVersion: v1 +kind: Pod +metadata: + name: egress-vpn + labels: + app: egress-test + homelab.keithjackson.dev/egress-class: vpn +spec: + containers: + - name: curl + image: curlimages/curl:8.11.1 + command: ["sleep", "3600"] + resources: + requests: { cpu: 10m, memory: 16Mi } + limits: { memory: 32Mi } + restartPolicy: Never diff --git a/platform/vpn-egress/README.md b/platform/vpn-egress/README.md index ab3cc9a..936cd1b 100644 --- a/platform/vpn-egress/README.md +++ b/platform/vpn-egress/README.md @@ -45,11 +45,23 @@ natively route through a *pod's* tunnel. Options considered: Cilium egress gateway but is more moving parts and pod-IP-fragile. 3. **Per-pod Gluetun sidecar** — explicitly REJECTED by ADR-0013 (single shared gateway preferred). -**Plan:** validate option 1 against a live PIA tunnel (confirm `tun0` on the Gluetun node + a -`CiliumEgressGatewayPolicy` steering `egress-class=vpn` pods, then run the `egress-class` (R-NET-05/07) -and `vpn-killswitch` (R-NET-06) conformance tests). Fall back to option 2 if the interface isn't -addressable. The `CiliumEgressGatewayPolicy` is intentionally NOT committed until validated, to avoid -shipping an untested datapath policy. +### FINDING (2026-08-23): option 1 is NOT viable with Gluetun — needs an architect decision +With the tunnel now live (custom-mode Gluetun, PIA Stockholm egress), option 1 fails on inspection: +`CiliumEgressGatewayPolicy interface: tun0` requires `tun0` on the **node**, which means running Gluetun +`hostNetwork: true`. But Gluetun *owns its netns routing* — on connect its `[routing]` logic adds +`0.0.0.0/0` via the tunnel. In hostNetwork that would **redirect the NODE's own default route through the +VPN**, breaking k3s/etcd/kubelet on that node. Gluetun has no supported "create tun0 but don't touch the +default route" mode, so option 1 can't expose `tun0` on the node safely. (The external-router-offload +pattern some use — MikroTik/RouterOS policy routing per pod-subnet — is out of scope: ADR-0013 mandates an +*in-cluster* gateway on a dumb SOHO edge.) + +**→ Escalated.** The remaining viable path is **option 2 (gateway-pod routing)**: `egress-class=vpn` pods +default-route to the Gluetun pod via an init-container (+NET_ADMIN), and Gluetun forwards+NATs out `tun0`. +More moving parts, and it requires Gluetun to forward for *foreign* pods (not its native same-netns model), +so it needs validation. Kill-switch (R-NET-06) then relies on Gluetun's `FIREWALL=on` dropping non-tunnel +egress. Fixtures for the `egress-class`/`vpn-killswitch` conformance tests are authored; the +`CiliumEgressGatewayPolicy` / steering datapath is intentionally NOT committed until the mechanism is chosen ++ validated, to avoid shipping an untested (and potentially node-breaking) datapath. ## Port forwarding (qBittorrent) — follow-up PIA assigns a forwarded port that can change on reconnect. A small **port-sync helper**