ADR-0015: add non-negotiable egress invariants (default=WAN, scoped kill-switch)

Make the operator's requirement explicit: default (every pod + the nodes) egresses
the local/private WAN, never the VPN; VPN is strictly opt-in via egress-class=vpn;
the kill-switch blackholes ONLY vpn-class. The two classes fail in opposite
directions — internet fails open (WAN), vpn fails closed (blackhole). Added as ADR-0015
Invariants + a default-WAN principle in docs/03, gated by the egress-class/vpn-killswitch
conformance tests.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-24 00:52:07 -05:00
1 parent 6770087d68
commit c68349f4ab
2 files changed
+31 -1

No files matched your search

@@ -49,6 +49,27 @@ The user-facing contract is unchanged: services are labeled `egress-class=vpn` (
independent egress path**, and the rules are **enforced outside the service** — now via a Cilium policy + host
nft rather than an in-cluster gateway pod.
### Invariants (NON-NEGOTIABLE — implementation must satisfy all four)
The two egress classes have **opposite fail-safe directions**; getting this wrong either leaks VPN traffic or
breaks the cluster. These are hard requirements, not preferences:
1. **Default is the local/private WAN path — for everything.** Every pod (regardless of namespace) AND the
nodes themselves egress via the normal private-network/WAN stack by default. Traffic is on the VPN **only**
if a service explicitly opts in with `egress-class=vpn`.
2. **The VPN tunnel is never a default route.** Not for the node, not for any pod. `tun0` carries **only** the
flows a `CiliumEgressGatewayPolicy` explicitly selects (`egress-class=vpn`). The host tunnel is split-tunnel
(`route-noexec`/policy routing) precisely to guarantee this.
3. **The kill-switch blackhole is scoped to vpn-class only.** When `tun0` is down, the nftables rule drops
**only** the `egress-class=vpn` (Cilium-egress-gateway-marked) traffic. It MUST NOT match or drop default/
`internet`-class traffic or node traffic. `internet`-class egress is unaffected whether the tunnel is up
or down.
4. **Opposite fail directions.** Default/`internet` traffic **fails open** (toward the WAN — never blackholed).
`egress-class=vpn` traffic **fails closed** (toward blackhole — never leaks to the WAN). A misconfiguration
or partial failure must degrade in these directions, never the reverse.
These are gated by the `egress-class` (default/internet shows WAN IP; vpn shows VPN IP) and `vpn-killswitch`
(tunnel down ⇒ vpn-class blackholed, internet-class unaffected) conformance tests.
## Consequences
- Steering is **Cilium-native and declarative** (a `CiliumEgressGatewayPolicy` in Git) and touches no app pods
— flipping a service's `egress-class` label re-routes it with no workload rearchitecting.
+10 -1
View File
@@ -92,7 +92,16 @@ VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the desig
shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside
the service**. The steering mechanism was settled by **ADR-0015** after the original in-cluster-Gluetun
approach proved un-steerable (a Gluetun *pod*'s `tun0` is invisible to the node datapath, and
`hostNetwork` Gluetun hijacks the node's own default route). The model:
`hostNetwork` Gluetun hijacks the node's own default route).
**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes themselves — egress the
local/private WAN by default and **never** the VPN. Only services labeled `egress-class=vpn` are steered to
the tunnel, and the kill-switch blackholes **only** that class. The two classes fail in **opposite**
directions: `internet` fails **open** (WAN, never blackholed); `vpn` fails **closed** (blackhole, never leaks
to WAN). See the ADR-0015 invariants — implementation must not route the default through the VPN or blackhole
the wrong class.
The model:
- **Tunnel on the node, not in a pod.** One or two labeled **egress-gateway nodes** run a **host OpenVPN
systemd service** — **PIA over OpenVPN-UDP** (provider/protocol per ADR-0013; "non-TCP or better"),