From c68349f4ab062c2aed5be2901666aba801cd4da0 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 24 Aug 2026 00:52:07 -0500 Subject: [PATCH] ADR-0015: add non-negotiable egress invariants (default=WAN, scoped kill-switch) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Make the operator's requirement explicit: default (every pod + the nodes) egresses the local/private WAN, never the VPN; VPN is strictly opt-in via egress-class=vpn; the kill-switch blackholes ONLY vpn-class. The two classes fail in opposite directions — internet fails open (WAN), vpn fails closed (blackhole). Added as ADR-0015 Invariants + a default-WAN principle in docs/03, gated by the egress-class/vpn-killswitch conformance tests. Co-Authored-By: Claude Opus 4.8 --- ...-vpn-egress-steering-node-tunnel-cilium.md | 21 +++++++++++++++++++ docs/03-networking-isolation-and-egress.md | 11 +++++++++- 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md b/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md index 615eec8..42f6930 100644 --- a/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md +++ b/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md @@ -49,6 +49,27 @@ The user-facing contract is unchanged: services are labeled `egress-class=vpn` ( independent egress path**, and the rules are **enforced outside the service** — now via a Cilium policy + host nft rather than an in-cluster gateway pod. +### Invariants (NON-NEGOTIABLE — implementation must satisfy all four) +The two egress classes have **opposite fail-safe directions**; getting this wrong either leaks VPN traffic or +breaks the cluster. These are hard requirements, not preferences: + +1. **Default is the local/private WAN path — for everything.** Every pod (regardless of namespace) AND the + nodes themselves egress via the normal private-network/WAN stack by default. Traffic is on the VPN **only** + if a service explicitly opts in with `egress-class=vpn`. +2. **The VPN tunnel is never a default route.** Not for the node, not for any pod. `tun0` carries **only** the + flows a `CiliumEgressGatewayPolicy` explicitly selects (`egress-class=vpn`). The host tunnel is split-tunnel + (`route-noexec`/policy routing) precisely to guarantee this. +3. **The kill-switch blackhole is scoped to vpn-class only.** When `tun0` is down, the nftables rule drops + **only** the `egress-class=vpn` (Cilium-egress-gateway-marked) traffic. It MUST NOT match or drop default/ + `internet`-class traffic or node traffic. `internet`-class egress is unaffected whether the tunnel is up + or down. +4. **Opposite fail directions.** Default/`internet` traffic **fails open** (toward the WAN — never blackholed). + `egress-class=vpn` traffic **fails closed** (toward blackhole — never leaks to the WAN). A misconfiguration + or partial failure must degrade in these directions, never the reverse. + +These are gated by the `egress-class` (default/internet shows WAN IP; vpn shows VPN IP) and `vpn-killswitch` +(tunnel down ⇒ vpn-class blackholed, internet-class unaffected) conformance tests. + ## Consequences - Steering is **Cilium-native and declarative** (a `CiliumEgressGatewayPolicy` in Git) and touches no app pods — flipping a service's `egress-class` label re-routes it with no workload rearchitecting. diff --git a/docs/03-networking-isolation-and-egress.md b/docs/03-networking-isolation-and-egress.md index bdd33eb..d194d2e 100644 --- a/docs/03-networking-isolation-and-egress.md +++ b/docs/03-networking-isolation-and-egress.md @@ -92,7 +92,16 @@ VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the desig shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside the service**. The steering mechanism was settled by **ADR-0015** after the original in-cluster-Gluetun approach proved un-steerable (a Gluetun *pod*'s `tun0` is invisible to the node datapath, and -`hostNetwork` Gluetun hijacks the node's own default route). The model: +`hostNetwork` Gluetun hijacks the node's own default route). + +**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes themselves — egress the +local/private WAN by default and **never** the VPN. Only services labeled `egress-class=vpn` are steered to +the tunnel, and the kill-switch blackholes **only** that class. The two classes fail in **opposite** +directions: `internet` fails **open** (WAN, never blackholed); `vpn` fails **closed** (blackhole, never leaks +to WAN). See the ADR-0015 invariants — implementation must not route the default through the VPN or blackhole +the wrong class. + +The model: - **Tunnel on the node, not in a pod.** One or two labeled **egress-gateway nodes** run a **host OpenVPN systemd service** — **PIA over OpenVPN-UDP** (provider/protocol per ADR-0013; "non-TCP or better"),