ADR-0015: add non-negotiable egress invariants (default=WAN, scoped kill-switch)
Make the operator's requirement explicit: default (every pod + the nodes) egresses the local/private WAN, never the VPN; VPN is strictly opt-in via egress-class=vpn; the kill-switch blackholes ONLY vpn-class. The two classes fail in opposite directions — internet fails open (WAN), vpn fails closed (blackhole). Added as ADR-0015 Invariants + a default-WAN principle in docs/03, gated by the egress-class/vpn-killswitch conformance tests. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
1 parent
6770087d68
commit
c68349f4ab
2 files changed
+31
-1
No files matched your search
@@ -49,6 +49,27 @@ The user-facing contract is unchanged: services are labeled `egress-class=vpn` (
|
|||||||
independent egress path**, and the rules are **enforced outside the service** — now via a Cilium policy + host
|
independent egress path**, and the rules are **enforced outside the service** — now via a Cilium policy + host
|
||||||
nft rather than an in-cluster gateway pod.
|
nft rather than an in-cluster gateway pod.
|
||||||
|
|
||||||
|
### Invariants (NON-NEGOTIABLE — implementation must satisfy all four)
|
||||||
|
The two egress classes have **opposite fail-safe directions**; getting this wrong either leaks VPN traffic or
|
||||||
|
breaks the cluster. These are hard requirements, not preferences:
|
||||||
|
|
||||||
|
1. **Default is the local/private WAN path — for everything.** Every pod (regardless of namespace) AND the
|
||||||
|
nodes themselves egress via the normal private-network/WAN stack by default. Traffic is on the VPN **only**
|
||||||
|
if a service explicitly opts in with `egress-class=vpn`.
|
||||||
|
2. **The VPN tunnel is never a default route.** Not for the node, not for any pod. `tun0` carries **only** the
|
||||||
|
flows a `CiliumEgressGatewayPolicy` explicitly selects (`egress-class=vpn`). The host tunnel is split-tunnel
|
||||||
|
(`route-noexec`/policy routing) precisely to guarantee this.
|
||||||
|
3. **The kill-switch blackhole is scoped to vpn-class only.** When `tun0` is down, the nftables rule drops
|
||||||
|
**only** the `egress-class=vpn` (Cilium-egress-gateway-marked) traffic. It MUST NOT match or drop default/
|
||||||
|
`internet`-class traffic or node traffic. `internet`-class egress is unaffected whether the tunnel is up
|
||||||
|
or down.
|
||||||
|
4. **Opposite fail directions.** Default/`internet` traffic **fails open** (toward the WAN — never blackholed).
|
||||||
|
`egress-class=vpn` traffic **fails closed** (toward blackhole — never leaks to the WAN). A misconfiguration
|
||||||
|
or partial failure must degrade in these directions, never the reverse.
|
||||||
|
|
||||||
|
These are gated by the `egress-class` (default/internet shows WAN IP; vpn shows VPN IP) and `vpn-killswitch`
|
||||||
|
(tunnel down ⇒ vpn-class blackholed, internet-class unaffected) conformance tests.
|
||||||
|
|
||||||
## Consequences
|
## Consequences
|
||||||
- Steering is **Cilium-native and declarative** (a `CiliumEgressGatewayPolicy` in Git) and touches no app pods
|
- Steering is **Cilium-native and declarative** (a `CiliumEgressGatewayPolicy` in Git) and touches no app pods
|
||||||
— flipping a service's `egress-class` label re-routes it with no workload rearchitecting.
|
— flipping a service's `egress-class` label re-routes it with no workload rearchitecting.
|
||||||
|
|||||||
@@ -92,7 +92,16 @@ VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the desig
|
|||||||
shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside
|
shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside
|
||||||
the service**. The steering mechanism was settled by **ADR-0015** after the original in-cluster-Gluetun
|
the service**. The steering mechanism was settled by **ADR-0015** after the original in-cluster-Gluetun
|
||||||
approach proved un-steerable (a Gluetun *pod*'s `tun0` is invisible to the node datapath, and
|
approach proved un-steerable (a Gluetun *pod*'s `tun0` is invisible to the node datapath, and
|
||||||
`hostNetwork` Gluetun hijacks the node's own default route). The model:
|
`hostNetwork` Gluetun hijacks the node's own default route).
|
||||||
|
|
||||||
|
**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes themselves — egress the
|
||||||
|
local/private WAN by default and **never** the VPN. Only services labeled `egress-class=vpn` are steered to
|
||||||
|
the tunnel, and the kill-switch blackholes **only** that class. The two classes fail in **opposite**
|
||||||
|
directions: `internet` fails **open** (WAN, never blackholed); `vpn` fails **closed** (blackhole, never leaks
|
||||||
|
to WAN). See the ADR-0015 invariants — implementation must not route the default through the VPN or blackhole
|
||||||
|
the wrong class.
|
||||||
|
|
||||||
|
The model:
|
||||||
|
|
||||||
- **Tunnel on the node, not in a pod.** One or two labeled **egress-gateway nodes** run a **host OpenVPN
|
- **Tunnel on the node, not in a pod.** One or two labeled **egress-gateway nodes** run a **host OpenVPN
|
||||||
systemd service** — **PIA over OpenVPN-UDP** (provider/protocol per ADR-0013; "non-TCP or better"),
|
systemd service** — **PIA over OpenVPN-UDP** (provider/protocol per ADR-0013; "non-TCP or better"),
|
||||||
|
|||||||
Reference in new issue
Block a user