ADR-0015: VPN egress steering — node-level tunnel + Cilium Egress Gateway

The in-cluster Gluetun-pod gateway (ADR-0013) cannot be steered: a pod's tun0 is invisible to the node datapath, and hostNetwork Gluetun hijacks the node's own default route (breaks k3s/etcd). The tunnel itself works (PIA egress validated), but not the steering. New decision (ADR-0015): move the tunnel to a HOST OpenVPN systemd service on egress-gateway node(s) (Ansible-managed, split-tunnel so it doesn't touch the node default route), steer egress-class=vpn pods with a CiliumEgressGatewayPolicy (native eBPF, no client-pod changes), and enforce the kill-switch with a host nftables rule (fail-closed independent of Cilium). PIA/OpenVPN-UDP provider decision (ADR-0013) stands; the .ovpn transfers over; PF re-solved host-side (Phase 6). Amended for consistency: ADR-0013 (client+steering superseded), decisions/README, docs/00, docs/01, docs/03, docs/07, agents/task-board. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 00:46:18 -05:00
1 parent 0ba06ec25b
commit 7598cff945
@@ -54,5 +54,6 @@ serves. **Do not edit an accepted decision in place** — supersede it with a ne
 | [0010](adr-0010-conformance-testing.md) | Conformance testing: automated, reproducible, behavior-based | Accepted | R-TEST-01..06 |
 | [0011](adr-0011-test-tooling-kyverno-chainsaw.md) | Conformance test tooling: Kyverno + Chainsaw | Accepted | R-TEST-01..05 |
 | [0012](adr-0012-vpn-egress-single-expressvpn-gateway.md) | VPN egress: single shared in-cluster ExpressVPN gateway | ~~Superseded by 0013~~ | R-NET-05/06/07 |
-| [0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md) | VPN provider: PIA via Gluetun (OpenVPN-UDP + port forwarding) | Accepted (amended 2026-08-22) | R-NET-05/06/07 |
+| [0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md) | VPN provider: PIA over OpenVPN-UDP + port forwarding | Accepted — provider/protocol only (client+steering superseded by 0015) | R-NET-05/06/07 |
 | [0014](adr-0014-idp-keycloak.md) | Identity Provider: Keycloak (SSO via OIDC + forward-auth) | Accepted | R-IDP-01..06 |
+| [0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md) | VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch) | Accepted | R-NET-05/06/07 |
@@ -1,5 +1,6 @@
 # ADR-0013 — VPN provider: PIA via Gluetun (OpenVPN-UDP + native port forwarding)
- Status: Accepted (amended 2026-08-22 — protocol WireGuard → OpenVPN-UDP)
+- Status: Accepted — provider/protocol only (amended 2026-08-22 protocol→OpenVPN-UDP; Gluetun client + egress
+  steering superseded by [ADR-0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md) 2026-08-23)
 - Date: 2026-08-15
 - Serves: R-NET-05, R-NET-06, R-NET-07
 - Supersedes: [ADR-0012](adr-0012-vpn-egress-single-expressvpn-gateway.md)
@@ -19,6 +20,15 @@
 > contained swap, since the egress-steering design is protocol-agnostic. The filename retains the
 > `-wireguard` slug as a stable historical identifier.

+> **Superseded in part — 2026-08-23 by [ADR-0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md).** The
+> **provider + protocol** here (PIA over OpenVPN-UDP, native port forwarding) STANDS. What is superseded: the
+> **client + placement** (a single in-cluster **Gluetun** pod) and the deferred **steering** mechanism. Live
+> validation confirmed the in-cluster Gluetun tunnel works (PIA egress) but **cannot be steered by Cilium
+> Egress Gateway** — Gluetun's `tun0` is in the pod netns, and `hostNetwork` Gluetun hijacks the node's own
+> default route (breaks k3s/etcd). ADR-0015 moves the tunnel to a **host OpenVPN service on egress-gateway
+> node(s)**, steered by **Cilium Egress Gateway** with a **host nftables kill-switch**. The same PIA `.ovpn`
+> transfers over.
+
 ## Context
 ADR-0012 kept the operator's required model — a single shared in-cluster VPN gateway with rules
 enforced outside the service — but named **ExpressVPN** as the provider, which forced a custom
@@ -0,0 +1,78 @@
+# ADR-0015 — VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch)
+- Status: Accepted
+- Date: 2026-08-23
+- Serves: R-NET-05, R-NET-06, R-NET-07
+- Supersedes (egress-steering + tunnel placement of): [ADR-0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md)
+  — which now stands only for the **provider/protocol** (PIA over OpenVPN-UDP + port forwarding). By
+  extension this also closes the steering question left open since [ADR-0008](adr-0008-vpn-egress-model.md).
+
+## Context
+ADR-0013 placed the VPN in a **single shared in-cluster Gluetun pod** and *deferred* the egress-steering
+mechanism (how `egress-class=vpn` pods actually route through it), naming "Cilium Egress Gateway → the
+gateway's tunnel interface" as the recommended-but-unvalidated option.
+
+Implementation (2026-08) proved that option **non-viable**, and validated the tunnel itself:
+- The in-cluster Gluetun tunnel **works** — PIA Stockholm egress confirmed, stable.
+- But `CiliumEgressGatewayPolicy` SNATs matched pods out of a **node** interface; Gluetun-in-a-pod's `tun0`
+  lives in the *pod* netns, invisible to the node datapath. Running Gluetun `hostNetwork` to put `tun0` on
+  the node **breaks the node**: Gluetun owns its netns routing and on connect redirects `0.0.0.0/0` via the
+  tunnel — in host netns that hijacks the **node's own default route**, cutting k3s/etcd/kubelet off. Gluetun
+  has no "create `tun0` but don't touch the default route" mode.
+- The accepted community alternative — the **pod-gateway** pattern (VXLAN overlay + admission-webhook-injected
+  client routing + gateway NAT, e.g. `angelnu/pod-gateway`) — works but is heavy, mutates client pods, and
+  side-steps Cilium's datapath.
+
+The clean fix is to move the tunnel to where routing is fully controllable — the **node** — and use Cilium's
+native egress gateway to steer.
+
+## Decision
+The `egress: vpn` contract is realized by a **node-level VPN tunnel steered by Cilium Egress Gateway, with a
+host-level kill-switch**:
+
+- **Tunnel — host service, not a pod.** One or two labeled **egress-gateway nodes** run a **host OpenVPN
+  systemd service** (PIA over OpenVPN-UDP — provider/protocol per ADR-0013, same `.ovpn`). It is **Ansible-
+  managed host infrastructure** (like k3s itself), not a Flux workload. It is configured **split-tunnel**:
+  it creates `tun0` but does **not** set the node default route (`route-noexec` + policy routing), so the
+  node's own traffic (control plane, image pulls, other pods) stays on the WAN. Only traffic Cilium points
+  at `tun0` uses the VPN.
+- **Steering — Cilium Egress Gateway.** A `CiliumEgressGatewayPolicy` selects pods labeled
+  `homelab.keithjackson.dev/egress-class=vpn` and egresses them via the gateway node's `tun0` (SNAT to the
+  tunnel address). Native Cilium eBPF — **no client-pod changes, no sidecar, no overlay**. (`egressGateway`,
+  `bpf.masquerade`, `kubeProxyReplacement` are already enabled per ADR-0002.)
+- **Kill-switch (fail-closed, R-NET-06) — host nftables.** An **nftables rule on the gateway node** DROPS the
+  steered (`egress-class=vpn`) traffic whenever `tun0` is absent/down — a host backstop **independent of**
+  Cilium egress-gateway fail-open edge cases. This is a *stronger* guarantee than either pod-based option.
+- **Resilience.** Two egress-gateway nodes + **Cilium Egress Gateway HA** (gateway failover) provide the
+  active/standby the earlier ADRs asked for.
+
+The user-facing contract is unchanged: services are labeled `egress-class=vpn` (Kyverno-enforced), have **no
+independent egress path**, and the rules are **enforced outside the service** — now via a Cilium policy + host
+nft rather than an in-cluster gateway pod.
+
+## Consequences
+- Steering is **Cilium-native and declarative** (a `CiliumEgressGatewayPolicy` in Git) and touches no app pods
+  — flipping a service's `egress-class` label re-routes it with no workload rearchitecting.
+- **Two-layer ownership:** the tunnel is host infra (**Ansible** role on the gateway nodes); the steering
+  policy is **Flux/GitOps**. Consistent with how k3s is provisioned. This is the intended split, not a leak.
+- **Secrets move host-side:** PIA credentials + `.ovpn` live on the gateway node(s) via **Ansible vault**, not
+  a cluster SOPS Secret — see docs/07. (SOPS still governs in-cluster secrets.)
+- **Gluetun is dropped** as the gateway. Its validated custom-mode config + the working PIA `.ovpn` transfer
+  directly to the host OpenVPN service, so the connectivity work is retained; only the packaging changes.
+- **Port forwarding** (PIA NAT-PMP for qBittorrent) must be re-solved host-side (a small pf/natpmp helper that
+  publishes the assigned port to the app) — Phase 6, as before.
+- **SPOF** on a single gateway node unless run HA (two nodes + Cilium failover).
+- Validated end-to-end by the **`egress-class`** (R-NET-05/07) and **`vpn-killswitch`** (R-NET-06) conformance
+  tests: vpn-class egress shows the PIA IP, internet-class shows the WAN IP, a git label flip re-routes, and
+  forcing `tun0` down drops vpn-class egress (no leak) while internet-class is unaffected.
+
+## Alternatives considered
+- **In-cluster Gluetun pod + Cilium Egress Gateway → its `tun0`** (ADR-0013's deferred option 1) — NON-VIABLE:
+  `hostNetwork` Gluetun hijacks the node default route.
+- **pod-gateway pattern** (VXLAN overlay + admission webhook + gateway NAT) — the accepted *in-cluster*
+  alternative; works but heavier, mutates client pods, bypasses Cilium's datapath. Rejected for a Cilium
+  cluster in favor of the native node-level egress gateway.
+- **Per-pod Gluetun sidecar** (ADR-0008 Pattern A) — rejected earlier: privilege colocation with the workload
+  and no declarative east-west policy.
+- **Keeping the working in-cluster Gluetun pod for its OWN egress only** — retained as the fallback if the
+  node-level tunnel proves troublesome, but it cannot serve *other* pods, which is the whole `egress: vpn`
+  requirement.