diff --git a/infra/ansible/roles/vpn_egress_gateway/README.md b/infra/ansible/roles/vpn_egress_gateway/README.md index 2d3aece..59992c8 100644 --- a/infra/ansible/roles/vpn_egress_gateway/README.md +++ b/infra/ansible/roles/vpn_egress_gateway/README.md @@ -32,7 +32,8 @@ Steering is done with **policy routing** in a dedicated table (`51820`, named `v **`from lookup 51820`** (priority 5000). Only packets whose source is the tunnel address — i.e. the vpn-class pods Cilium has SNAT'd to `tun0` — consult that table. Nothing else does, so nothing else is re-routed. -- The **down-script** removes the ip-rule and the tunnel default (leaving the blackhole). +- The **down-script** removes the ip-rule and the tunnel default (leaving the blackhole), and + **retains** the nft source set as the explicit fail-closed backstop (see kill-switch section). ### 3 & 4 — kill-switch scoped to vpn-class, opposite fail directions Two independent layers, both scoped so they can only ever affect vpn-class traffic: @@ -46,13 +47,32 @@ Two independent layers, both scoped so they can only ever affect vpn-class traff narrow matches, never break anything). In `postrouting`: - `oifname tun0 ip saddr @vpn_egress_gateway_src accept` — tunnel up: let steered traffic out. - `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` — **fail closed:** a tunnel-sourced packet - trying to leave any other interface is dropped (no WAN leak). + trying to leave any other interface is dropped (no WAN leak). This rule stays **armed even + while the tunnel is down** because the source set is retained on tunnel-down (see below). - `meta mark 0x1c8 oifname != tun0 drop` — the optional fwmark backstop. -`@vpn_egress_gateway_src` holds the **live tun0 address** (added by the up-script, flushed by the -down-script). Node traffic and `internet`-class traffic are never in the set, never carry the -mark, and are never targeted — so they always hit `policy accept` and **fail open toward the -WAN** (invariant 4), tunnel up or down. The opposite directions are structural, not incidental. +`@vpn_egress_gateway_src` holds the **last live tun0 address**. The up-script **flushes then +re-adds** the current tunnel source on every connect; the down-script **intentionally retains +it** (see "Explicit fail-closed backstop" below). Node traffic and `internet`-class traffic are +never in the set, never carry the mark, and are never targeted — so they always hit +`policy accept` and **fail open toward the WAN** (invariant 4), tunnel up or down. The opposite +directions are structural, not incidental. + +### Explicit, edge-independent fail-closed backstop (tunnel-down) +Earlier the down-script flushed `@vpn_egress_gateway_src`, which **disarmed** the +`ip saddr @vpn_egress_gateway_src oifname != tun0 drop` rule at exactly the moment the tunnel was +down. Fail-closed then rode only on the stale SNAT source being non-routable — an implicit +guarantee that could leak if the edge router NATed that stale source. The down-script now +**does NOT flush the set**: retaining the last tun0 address keeps the drop rule armed, so any +vpn-class packet that would leave the WAN while the tunnel is down is **explicitly dropped** in +the host nft table — defense-in-depth, independent of edge-router behavior. + +This is safe precisely because the **up-script flushes the set and re-adds the (possibly new) +tunnel source on every reconnect** (`nft flush set …` then `nft add element …`), so a retained +stale address self-corrects the instant the tunnel comes back. The set remains scoped to +vpn-class only; node and internet-class traffic are never members, so **no invariant is +weakened — only vpn-class is ever affected** (invariants 3 & 4 upheld). The routing blackhole in +table `51820` remains as the second, independent fail-closed layer. ## Why the node default route is provably unaffected (design-level verification) - OpenVPN runs with `route-noexec` and `pull-filter ignore "redirect-gateway"` → it adds **zero** diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 index 5701318..21ec2ba 100644 --- a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 @@ -4,21 +4,34 @@ # split-tunnel steering so vpn-class traffic FAILS CLOSED, never leaks to the WAN # (ADR-0015 invariants 3 & 4). # -# Order matters: (1) empty the kill-switch source set and drop the ip-rule so no packet -# can be routed toward a vanishing tun0, then (2) remove the tunnel default from the -# dedicated table. The permanent `blackhole default` in that table is LEFT in place as -# the fail-closed fallback. The node's main-table WAN default is never touched here. +# Order matters: (1) drop the ip-rule so no packet can be routed toward a vanishing tun0, +# then (2) remove the tunnel default from the dedicated table. The permanent +# `blackhole default` in that table is LEFT in place as one fail-closed fallback. +# +# EXPLICIT FAIL-CLOSED BACKSTOP: the kill-switch source set is INTENTIONALLY RETAINED +# here (we do NOT flush it). Keeping the last live tun0 address in +# `{{ vpn_egress_gateway_nft_src_set }}` keeps the nft rule +# `ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != tun0 drop` ARMED while the +# tunnel is down, so any vpn-class packet that would leave the WAN is EXPLICITLY dropped +# — defense-in-depth that does not depend on the stale SNAT source being non-routable or +# on the edge router refusing to NAT it. This is safe because the up-script FLUSHES then +# re-adds the (possibly new) tunnel source on every reconnect, so the set self-corrects. +# The set is scoped to vpn-class only; node and internet-class traffic are never in it. +# The node's main-table WAN default is never touched here. set -uo pipefail TABLE_ID="{{ vpn_egress_gateway_route_table_id }}" RULE_PRIO="{{ vpn_egress_gateway_rule_priority }}" -NFT_TABLE="{{ vpn_egress_gateway_nft_table }}" -NFT_SET="{{ vpn_egress_gateway_nft_src_set }}" +# NFT_TABLE/NFT_SET are intentionally NOT referenced here: the nft source set is retained +# on tunnel-down (see header) as the explicit fail-closed backstop, so this script no +# longer mutates the nft ruleset. IFACE="${dev:-{{ vpn_egress_gateway_tun_iface }}}" SRC="${ifconfig_local:-}" -# 1) Stop scoping/steering to the (now dead) tunnel source. -nft flush set inet "${NFT_TABLE}" "${NFT_SET}" 2>/dev/null || true +# 1) Stop steering to the (now dead) tunnel source by dropping the ip-rule. +# NOTE: the nft source set is deliberately NOT flushed here — see header. Retaining the +# last tun0 address keeps the `oifname != tun0 drop` rule armed as the explicit, +# edge-independent fail-closed backstop until the up-script re-registers on reconnect. if [ -n "${SRC}" ]; then while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done fi @@ -26,4 +39,4 @@ fi # 2) Remove the tunnel default from the dedicated table; keep the blackhole fallback. ip route del default dev "${IFACE}" table "${TABLE_ID}" 2>/dev/null || true -echo "vpn-egress-down: steering removed; vpn-class fails closed (blackhole retained)" +echo "vpn-egress-down: steering removed; nft src set retained as explicit fail-closed backstop (blackhole retained)"