Merge feature/phase3-egress-steering-groundwork into main

conformance/tests/l2-behavioral/egress-class/fixtures/internet-and-vpn-pods.yaml

@@ -0,0 +1,36 @@
+# Two egress-class workloads for the egress-class + vpn-killswitch conformance tests.
+# `internet` egresses the normal WAN; `vpn` is steered through the gluetun tunnel (CiliumEgressGatewayPolicy
+# in platform/vpn-egress/egress-steering.yaml). Both just need curl to hit an echo-IP service.
+apiVersion: v1
+kind: Pod
+metadata:
+  name: egress-internet
+  labels:
+    app: egress-test
+    homelab.keithjackson.dev/egress-class: internet
+spec:
+  containers:
+    - name: curl
+      image: curlimages/curl:8.11.1
+      command: ["sleep", "3600"]
+      resources:
+        requests: { cpu: 10m, memory: 16Mi }
+        limits: { memory: 32Mi }
+  restartPolicy: Never
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: egress-vpn
+  labels:
+    app: egress-test
+    homelab.keithjackson.dev/egress-class: vpn
+spec:
+  containers:
+    - name: curl
+      image: curlimages/curl:8.11.1
+      command: ["sleep", "3600"]
+      resources:
+        requests: { cpu: 10m, memory: 16Mi }
+        limits: { memory: 32Mi }
+  restartPolicy: Never

platform/vpn-egress/README.md

@@ -45,11 +45,23 @@ natively route through a *pod's* tunnel. Options considered:
    Cilium egress gateway but is more moving parts and pod-IP-fragile.
 3. **Per-pod Gluetun sidecar** — explicitly REJECTED by ADR-0013 (single shared gateway preferred).
 
-**Plan:** validate option 1 against a live PIA tunnel (confirm `tun0` on the Gluetun node + a
+### FINDING (2026-08-23): option 1 is NOT viable with Gluetun — needs an architect decision
-`CiliumEgressGatewayPolicy` steering `egress-class=vpn` pods, then run the `egress-class` (R-NET-05/07)
+With the tunnel now live (custom-mode Gluetun, PIA Stockholm egress), option 1 fails on inspection:
-and `vpn-killswitch` (R-NET-06) conformance tests). Fall back to option 2 if the interface isn't
+`CiliumEgressGatewayPolicy interface: tun0` requires `tun0` on the **node**, which means running Gluetun
-addressable. The `CiliumEgressGatewayPolicy` is intentionally NOT committed until validated, to avoid
+`hostNetwork: true`. But Gluetun *owns its netns routing* — on connect its `[routing]` logic adds
-shipping an untested datapath policy.
+`0.0.0.0/0` via the tunnel. In hostNetwork that would **redirect the NODE's own default route through the
+VPN**, breaking k3s/etcd/kubelet on that node. Gluetun has no supported "create tun0 but don't touch the
+default route" mode, so option 1 can't expose `tun0` on the node safely. (The external-router-offload
+pattern some use — MikroTik/RouterOS policy routing per pod-subnet — is out of scope: ADR-0013 mandates an
+*in-cluster* gateway on a dumb SOHO edge.)
+
+**→ Escalated.** The remaining viable path is **option 2 (gateway-pod routing)**: `egress-class=vpn` pods
+default-route to the Gluetun pod via an init-container (+NET_ADMIN), and Gluetun forwards+NATs out `tun0`.
+More moving parts, and it requires Gluetun to forward for *foreign* pods (not its native same-netns model),
+so it needs validation. Kill-switch (R-NET-06) then relies on Gluetun's `FIREWALL=on` dropping non-tunnel
+egress. Fixtures for the `egress-class`/`vpn-killswitch` conformance tests are authored; the
+`CiliumEgressGatewayPolicy` / steering datapath is intentionally NOT committed until the mechanism is chosen
++ validated, to avoid shipping an untested (and potentially node-breaking) datapath.
 
 ## Port forwarding (qBittorrent) — follow-up
 PIA assigns a forwarded port that can change on reconnect. A small **port-sync helper**