vpn_egress_gateway: retain nft src set on tunnel-down (explicit fail-closed backstop)

The down-script previously flushed the kill-switch source set on tunnel-down, which
disarmed the 'ip saddr @vpn_egress_gateway_src oifname != tun0 drop' rule exactly when
the tunnel was down — leaving fail-closed to ride only on the stale SNAT source being
non-routable (could leak if the edge NATed it). Retain the set instead: the drop rule
stays armed so any vpn-class packet leaving the WAN while the tunnel is down is EXPLICITLY
dropped (edge-independent). Safe because the up-script flush+re-adds the source on every
reconnect. Validated behavior: WS-NET fault-injection on k3s-dev-3 (2026-08-24).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

infra/ansible/roles/vpn_egress_gateway/README.md

@@ -32,7 +32,8 @@ Steering is done with **policy routing** in a dedicated table (`51820`, named `v
   **`from <tun0-addr> lookup 51820`** (priority 5000). Only packets whose source is the tunnel
   address — i.e. the vpn-class pods Cilium has SNAT'd to `tun0` — consult that table. Nothing
   else does, so nothing else is re-routed.
-- The **down-script** removes the ip-rule and the tunnel default (leaving the blackhole).
+- The **down-script** removes the ip-rule and the tunnel default (leaving the blackhole), and
+  **retains** the nft source set as the explicit fail-closed backstop (see kill-switch section).
 
 ### 3 & 4 — kill-switch scoped to vpn-class, opposite fail directions
 Two independent layers, both scoped so they can only ever affect vpn-class traffic:
@@ -46,13 +47,32 @@ Two independent layers, both scoped so they can only ever affect vpn-class traff
    narrow matches, never break anything). In `postrouting`:
    - `oifname tun0 ip saddr @vpn_egress_gateway_src accept` — tunnel up: let steered traffic out.
    - `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` — **fail closed:** a tunnel-sourced packet
-     trying to leave any other interface is dropped (no WAN leak).
+     trying to leave any other interface is dropped (no WAN leak). This rule stays **armed even
+     while the tunnel is down** because the source set is retained on tunnel-down (see below).
    - `meta mark 0x1c8 oifname != tun0 drop` — the optional fwmark backstop.
 
-`@vpn_egress_gateway_src` holds the **live tun0 address** (added by the up-script, flushed by the
-down-script). Node traffic and `internet`-class traffic are never in the set, never carry the
-mark, and are never targeted — so they always hit `policy accept` and **fail open toward the
-WAN** (invariant 4), tunnel up or down. The opposite directions are structural, not incidental.
+`@vpn_egress_gateway_src` holds the **last live tun0 address**. The up-script **flushes then
+re-adds** the current tunnel source on every connect; the down-script **intentionally retains
+it** (see "Explicit fail-closed backstop" below). Node traffic and `internet`-class traffic are
+never in the set, never carry the mark, and are never targeted — so they always hit
+`policy accept` and **fail open toward the WAN** (invariant 4), tunnel up or down. The opposite
+directions are structural, not incidental.
+
+### Explicit, edge-independent fail-closed backstop (tunnel-down)
+Earlier the down-script flushed `@vpn_egress_gateway_src`, which **disarmed** the
+`ip saddr @vpn_egress_gateway_src oifname != tun0 drop` rule at exactly the moment the tunnel was
+down. Fail-closed then rode only on the stale SNAT source being non-routable — an implicit
+guarantee that could leak if the edge router NATed that stale source. The down-script now
+**does NOT flush the set**: retaining the last tun0 address keeps the drop rule armed, so any
+vpn-class packet that would leave the WAN while the tunnel is down is **explicitly dropped** in
+the host nft table — defense-in-depth, independent of edge-router behavior.
+
+This is safe precisely because the **up-script flushes the set and re-adds the (possibly new)
+tunnel source on every reconnect** (`nft flush set …` then `nft add element …`), so a retained
+stale address self-corrects the instant the tunnel comes back. The set remains scoped to
+vpn-class only; node and internet-class traffic are never members, so **no invariant is
+weakened — only vpn-class is ever affected** (invariants 3 & 4 upheld). The routing blackhole in
+table `51820` remains as the second, independent fail-closed layer.
 
 ## Why the node default route is provably unaffected (design-level verification)
 - OpenVPN runs with `route-noexec` and `pull-filter ignore "redirect-gateway"` → it adds **zero**

infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2

@@ -4,21 +4,34 @@
 # split-tunnel steering so vpn-class traffic FAILS CLOSED, never leaks to the WAN
 # (ADR-0015 invariants 3 & 4).
 #
-# Order matters: (1) empty the kill-switch source set and drop the ip-rule so no packet
-# can be routed toward a vanishing tun0, then (2) remove the tunnel default from the
-# dedicated table. The permanent `blackhole default` in that table is LEFT in place as
-# the fail-closed fallback. The node's main-table WAN default is never touched here.
+# Order matters: (1) drop the ip-rule so no packet can be routed toward a vanishing tun0,
+# then (2) remove the tunnel default from the dedicated table. The permanent
+# `blackhole default` in that table is LEFT in place as one fail-closed fallback.
+#
+# EXPLICIT FAIL-CLOSED BACKSTOP: the kill-switch source set is INTENTIONALLY RETAINED
+# here (we do NOT flush it). Keeping the last live tun0 address in
+# `{{ vpn_egress_gateway_nft_src_set }}` keeps the nft rule
+# `ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != tun0 drop` ARMED while the
+# tunnel is down, so any vpn-class packet that would leave the WAN is EXPLICITLY dropped
+# — defense-in-depth that does not depend on the stale SNAT source being non-routable or
+# on the edge router refusing to NAT it. This is safe because the up-script FLUSHES then
+# re-adds the (possibly new) tunnel source on every reconnect, so the set self-corrects.
+# The set is scoped to vpn-class only; node and internet-class traffic are never in it.
+# The node's main-table WAN default is never touched here.
 set -uo pipefail
 
 TABLE_ID="{{ vpn_egress_gateway_route_table_id }}"
 RULE_PRIO="{{ vpn_egress_gateway_rule_priority }}"
-NFT_TABLE="{{ vpn_egress_gateway_nft_table }}"
-NFT_SET="{{ vpn_egress_gateway_nft_src_set }}"
+# NFT_TABLE/NFT_SET are intentionally NOT referenced here: the nft source set is retained
+# on tunnel-down (see header) as the explicit fail-closed backstop, so this script no
+# longer mutates the nft ruleset.
 IFACE="${dev:-{{ vpn_egress_gateway_tun_iface }}}"
 SRC="${ifconfig_local:-}"
 
-# 1) Stop scoping/steering to the (now dead) tunnel source.
-nft flush set inet "${NFT_TABLE}" "${NFT_SET}" 2>/dev/null || true
+# 1) Stop steering to the (now dead) tunnel source by dropping the ip-rule.
+#    NOTE: the nft source set is deliberately NOT flushed here — see header. Retaining the
+#    last tun0 address keeps the `oifname != tun0 drop` rule armed as the explicit,
+#    edge-independent fail-closed backstop until the up-script re-registers on reconnect.
 if [ -n "${SRC}" ]; then
   while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done
 fi
@@ -26,4 +39,4 @@ fi
 # 2) Remove the tunnel default from the dedicated table; keep the blackhole fallback.
 ip route del default dev "${IFACE}" table "${TABLE_ID}" 2>/dev/null || true
 
-echo "vpn-egress-down: steering removed; vpn-class fails closed (blackhole retained)"
+echo "vpn-egress-down: steering removed; nft src set retained as explicit fail-closed backstop (blackhole retained)"