docs(vpn): stop crediting the nftables kill-switch for a drop it never makes

Four places in the tree told the reader that the host nftables table is the
authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress
README, egress-steering.yaml, the role README, and the vpn-killswitch test's own
GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway
datapath bpf_redirect()s those skbs to the egress device without invoking
netfilter POSTROUTING. Corrected with the measurement in each place rather than
softened, because a guard the repo trusts and that does not fire is worse than no
guard.

What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops
the traffic when it cannot use its cached egress interface, verified by tcpdump on
both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out
either). The nft table keeps a narrower, now-proven job: node-originated traffic
sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on
k3s-prod-3 when such a packet was routed at the WAN).

Also recorded:

* egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the
  gateway node is what tells Cilium it changed (the egress-uplink node label). A
  future reader's instinct will be to have the node write to that object; the
  comment says not to, because it is Flux-managed, and points at the label and
  `cilium-dbg bpf egress list` as the debugging entry points instead.
* egress-steering.yaml — why the new NFS media mounts are not affected by
  pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns,
  so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server
  too, but that is the second reason, not the first.
* vpn-killswitch test — its recovery step depends on the re-steer path, and could
  not have passed before it existed. Its no-leak assertion was always right; only
  the explanation was wrong. Also states the prerequisite plainly: the test cannot
  run at all without a labelled gateway node.
* platform README — prod is live on k3s-prod-3 with the before/after egress IPs,
  and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no
  matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway
  restored egress rather than moving it off the WAN.
* role README — the standing "add the shared fwmark to the interface contract"
  escalation is closed as not implementable on Cilium OSS, rather than left open
  for someone to attempt.

DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 /
docs-03 amendment for the architect (specs are immutable to agents) and the five
things left for the operator — chief among them that a silent gateway outage still
has no alert, which is what let the dev gateway sit dead for 80 minutes.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -2089,3 +2089,200 @@ fail-closed branch fixes the "unmapped path contributes nothing" hole; it does n
 Both belong to `ci/`, so they are left to that branch rather than smuggled in here. The full L0
 layer was run unscoped on this branch to compensate: **12 offline checks, 11 PASS, 1 SKIP**
 (`placeholder-guard-coupling`, pre-existing), 0 FAIL.
+
+## 2026-09-12 — VPN egress on PROD (`k3s-prod-3`), and three things the repo believed that are not true (branch `feat/vpn-egress-prod`)
+
+The operator asked for VPN egress on prod. An earlier assessment this session had recommended
+holding off over three suspected defects; the operator decided, so the defects were fixed first
+and then the gateway was stood up. All three were settled by **measurement on the dev gateway
+node**, not by reading code — and two of the three turned out to be different from what was
+suspected, in one case much worse.
+
+### The headline: the DEV gateway was already dead, silently, and had been for ~80 minutes
+
+`cilium-dbg bpf egress list` on `k3s-dev-3` showed `Egress Ifindex 3`. There is no ifindex 3 on
+that node (the list goes 1, 2, 4, 5, …) — it was a **destroyed** tun0. The live tun0 was ifindex
+1780. vpn-class pods on both the gateway node and a remote node timed out. Nothing alerted, and
+the ledger still recorded the gateway as conformance-green.
+
+**Cilium resolves a `CiliumEgressGatewayPolicy`'s `interface: tun0` exactly once**, caching an
+(egress IP, egress ifindex) pair, and does not re-resolve when tun0 changes. Upstream is explicit:
+*"it does not automatically respond to a change in the gateway node's network configuration… You
+can force a fresh selection by re-applying the Egress Gateway policy."* Every tunnel restart
+re-creates tun0 with a new ifindex, so **the gateway is dead after its first reconnect or reboot.**
+`egress-gateway-reconciliation-trigger-interval: 1s` does not cover it — both clusters carry that
+setting and dev was still stale 20s+ after recovery; the interval coalesces reconciliations that
+k8s object events already triggered, and netlink link events are not among them.
+
+Measured, in order:
+
+| step | observation |
+|---|---|
+| baseline | map ifindex 3, live tun0 1780, vpn-class pods time out |
+| annotate the CEGP (the upstream remedy) | map -> 1780 within 8s; pods return the PIA IP `158.173.241.36` immediately |
+| `systemctl restart vpn-egress-tunnel` | tun0 1780 -> 1985, map STAYS 1780, pods break again |
+| `systemctl stop`, wait 30s, `start` (the exact `vpn-killswitch` conformance sequence) | tun0 -> 1990, map stays 1989, pods still broken 20s+ later |
+| set ANY unrelated label on the gateway NODE | map -> live ifindex within 10s, egress recovers |
+
+**Fix:** `roles/vpn_egress_gateway` now publishes
+`homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node — from the
+OpenVPN up-script on every connect (via `systemctl start --no-block`, so it can never block or
+fail the tunnel and does not need the kube-API up at that instant), plus a 2-minute convergence
+timer for the boot case. A **node label, not an annotation on the policy**: the policy is
+Flux-managed GitOps and a node writing to GitOps state is a drift fight, whereas node labels are
+already this role's territory — and the label's value doubles as the diagnostic. After the fix,
+stop->start recovers unaided (map 1989 -> 1990, PIA IP returns).
+
+**Consequence for the ledger:** the `vpn-killswitch` (R-NET-06) test stops *and restarts* the
+tunnel. Its recovery step **could not have passed** on the implementation as it stood — verified
+twice. Whatever produced the earlier green result needs re-running; the fail-closed half was and
+remains sound.
+
+### Defect as reported vs. as measured
+
+**Reported:** the up-script's `nft flush set` opens a leak window — on reconnect Cilium keeps
+SNAT-ing to the previous address A while only B is in the set, so packets sourced from A match
+neither the ip-rule nor the drop rule and **leave via the WAN**.
+
+**Measured: there is no WAN leak.** Cilium's redirect follows the cached *ifindex*, not a routing
+lookup, so a stale egress IP cannot reach the WAN. Tested directly by swapping tun0's address in
+place (keeping the ifindex, as an OpenVPN `persist-tun` soft reconnect does) while Cilium held the
+old one: the pod's packets left **tun0** carrying the stale source, PIA answered, and tcpdump on
+the WAN caught **0** packets. Same result in the stale-ifindex case: 0 packets out the WAN, 0 out
+tun0 — the redirect to a dead ifindex simply fails and the packet is dropped.
+
+The flush was still wrong, so the set is now **additive** (`add element` is idempotent; the
+down-script already retained it). It cannot leak pod traffic — see the next section for why — but
+node-originated tunnel traffic *does* traverse netfilter, and additive costs nothing. A kill-switch
+reload no longer disarms the guard either: the `.nft` file uses `table` + `flush table`, which
+clears rules but leaves a named set's elements intact.
+
+### The nftables kill-switch cannot see the traffic it was written to guard
+
+Suspected; now proven, with the counters the brief asked for. A counters-only diagnostic nft table
+on the gateway node, while a vpn-class pod egressed:
+
+| counter | packets |
+|---|---|
+| `forward`, saddr = pod CIDR | **0** |
+| `forward`, saddr = PIA tunnel range | **0** |
+| `postrouting`, oif tun0, saddr = tunnel range | **0** |
+| `postrouting`, oif WAN, saddr = pod CIDR | **0** |
+| `postrouting`, oif WAN, any — control, the hook is live | 45166 |
+| `postrouting`, oif tun0, saddr = tunnel — control, a node-originated curl | 16 |
+
+With `bpf.masquerade: true` (which Egress Gateway requires) the SNAT and egress happen in eBPF,
+which `bpf_redirect()`s the skb to the egress device without invoking netfilter POSTROUTING. The
+hook works and the matching works — the steered pod packets are simply never presented to it.
+
+**So what fails closed is Cilium's own drop**, confirmed by tcpdump on both devices during a
+tunnel stop and a tunnel restart: 0 packets left either. The existing `vpn-killswitch` test passes
+because of that, not because of the nft rules. Every comment in the tree that credited the nft
+table has been corrected rather than left to mislead the next reader.
+
+**The table is kept, not deleted**, because it does cover something real — node-originated traffic
+sourced from a tunnel address — and that was proven in **both** directions on `k3s-prod-3`:
+
+- source-bound `ping -I 10.72.192.7` with the tunnel ip-rule present: 3/3 replies at 144 ms,
+  `killswitch_accept` advances;
+- ip-rule removed, so `ip route get` says `via 192.168.2.1 dev eno1`: **`killswitch_drop`
+  0 -> 3 packets / 252 bytes**, 100% loss.
+
+Every rule now carries a `counter` so the claim stays falsifiable. If `killswitch_drop` ever
+counts in normal operation, the datapath has changed.
+
+Also **removed** the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable. Nothing stamps
+that mark — the only Cilium mark on the gateway node is `0x200/0xf00`, which is the L7-proxy rule
+to table 2004 — and the traffic never reaches netfilter anyway. A live rule matching an arbitrary
+mark is a hazard to whatever component picks that value next, not a backstop. This **closes** the
+role README's standing escalation ("add the shared fwmark to the interface contract so the
+CiliumEgressGatewayPolicy marks vpn-class traffic") as **not implementable on Cilium OSS**: there
+is no supported way to make the policy stamp a mark, and no host firewall hook to catch it in.
+
+### Prod stand-up
+
+`[vpn_egress_gateway_prod]` is a **separate** inventory group holding only `k3s-prod-3`, with its
+own `group_vars/vpn_egress_gateway_prod/vault.yml`. Prod is deliberately NOT a member of
+`[vpn_egress_gateway]`, which is the default target of `playbooks/vpn.egress.gateway.yml` — sharing
+the group would mean a flagless run silently reconfigures a prod node's kill-switch datapath. Run
+as `-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3`, with `--list-hosts`
+first (confirmed: exactly `k3s-prod-3`).
+
+Node 3 because node 1's SSD was replaced 2026-09-11 (least operational history) and node 2 carries
+the §0b memattr history. **Exactly one node, forever**: Cilium OSS picks "the first node in lexical
+ordering" among nodeSelector matches, so labelling `k3s-prod-1` would silently take the gateway
+role, and the public egress IP, away from `k3s-prod-3`.
+
+The vault file is a **verbatim copy of dev's ciphertext** — same PIA account per the operator, and
+copying the encrypted blob means the plaintext never touched disk, a shell, or a transcript. The
+two files are independent from here on; rotating means re-encrypting both, which is correct for two
+environments.
+
+**`wlp1s0` did not need bringing down — it does not exist.** The brief and `hosts.ini` both said
+node 3 has wlp1s0 up on 192.168.2.195. Re-verified before choosing it, because a second live uplink
+in a kill-switch datapath is not a footnote: there is no `wl*` netdev on the node, no `phy80211`
+device, the `wlp1s0` stanza in `/etc/network/interfaces` is commented out, 192.168.2.195 is
+assigned nowhere, and the only IPv4 addresses are `lo` and `eno1 192.168.2.12`. The inventory
+comment is corrected.
+
+**Result on prod:**
+
+| | before | after |
+|---|---|---|
+| jackett / qbittorrent / radarr / sonarr (vpn-class, spread across all 3 nodes) | no internet at all — CEGP reconciled with no matching node, `cilium-dbg bpf egress list` showed `Gateway IP: Not Found`, traffic dropped | all four egress `158.173.241.36` (PIA Stockholm) |
+| plex / nextcloud (internet-class) | `136.51.115.252` (home WAN) | `136.51.115.252`, unchanged |
+| node's own egress | WAN | WAN — `default via 192.168.2.1 dev eno1`, k3s `/readyz` ok |
+
+Worth stating plainly: arming the gateway **restored** vpn-class egress rather than moving it from
+the WAN to the tunnel. Nothing had been leaking in the meantime, because nothing had been going
+out.
+
+### The NFS media mounts are unaffected, verified rather than assumed
+
+sonarr, radarr, plex and nextcloud gained NFS mounts from `192.168.2.102:/mnt/Storage/plex` (and
+`/mnt/Storage/nextcloud`) earlier today, so the interaction with pod-scoped egress steering was
+checked before and after arming:
+
+- **The structural reason it cannot break:** an NFS PersistentVolume is mounted by the **node
+  kernel in the host netns**, not by the pod, so the CEGP's `selectors.podSelector` never matches
+  it. No `excludedCIDRs` entry is needed for that.
+- **And belt-and-braces:** the server is `192.168.2.102`, inside the existing
+  `192.168.0.0/16` exclusion anyway.
+- **Empirically, after arming:** sonarr (on `k3s-prod-1`) and radarr (on `k3s-prod-3`, the gateway
+  node itself) both still `ls` **and write** to `/media`; plex lists it; the node's own
+  `findmnt`-visible mounts are intact and readable. qBittorrent, incidentally, has **no** NFS
+  media mount in this repo — only plex, sonarr, radarr and nextcloud do.
+
+### Left for the operator / not settled here
+
+1. **Re-run `vpn-killswitch` (R-NET-06) against a labelled cluster** and refresh the ledger. It can
+   only run where a node carries the gateway label, and its recovery step is only passable with
+   this branch's re-steer fix. It is an L3 fault-injection test that stops the tunnel on a LIVE
+   cluster, so it was not fired at prod unasked.
+2. **Both clusters now exit from the same PIA public IP** (`158.173.241.36`) on one account. Inside
+   the connection limit, but the two environments are indistinguishable from outside. A separate
+   prod PIA account is the clean fix if that ever matters.
+3. **Gateway-node loss is still an outage** — Cilium OSS has no egress-gateway failover, and the
+   ADR-0016 supersede drafted on 2026-09-06 is still open. Nothing here changes that; the
+   single-gateway choice is now recorded in the inventory itself, not just in prose.
+4. **A silent gateway outage still has no alert.** The re-steer timer means it now self-heals, but
+   nothing watches the invariant "the gateway node's `egress-uplink` label matches
+   `cilium-dbg bpf egress list`'s Egress Ifindex". That is a monitoring gap, not a datapath one,
+   and it is the thing that let dev sit dead for 80 minutes. A Prometheus rule over a small
+   node-exporter textfile from the re-steer script would close it.
+5. **Hygiene, unrelated but found on the way:** `/etc/network/interfaces` on `k3s-prod-3` holds a
+   commented-out wireless stanza containing the home WiFi **PSK in plaintext**. It is inert
+   (wireless is gone) and not printed here. Worth scrubbing on all three nodes.
+6. **`ADR-0015` and `docs/03` still describe the nftables kill-switch as the vpn-class
+   fail-closed mechanism**, and specs are immutable to agents. Drafted amendment for the architect:
+   > *R-NET-06's fail-closed guarantee for `egress-class=vpn` is provided by the Cilium Egress
+   > Gateway datapath, which drops steered traffic when the gateway's egress interface cannot be
+   > used; this was verified with packet capture on both the WAN and tunnel devices. The host
+   > nftables table is a defence-in-depth guard for node-originated traffic sourced from the tunnel
+   > address only — with `bpf.masquerade` enabled, which Egress Gateway requires, steered pod
+   > traffic does not traverse netfilter and cannot be guarded there. The "optional fwmark
+   > contract hook" is withdrawn: Cilium OSS provides no way for the policy to stamp such a mark.*
+   > *Additionally, `interface:` on a CiliumEgressGatewayPolicy is resolved once and cached; a
+   > gateway whose tunnel interface is dynamic requires an explicit re-resolution trigger on every
+   > interface change. This is a standing operational requirement of the ADR-0015 design, not an
+   > implementation detail.*

conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml

@@ -10,8 +10,31 @@
 #       egress-gateway node is STOPPED (fault injection) —
 # THEN  the vpn-class workload reaches NOTHING on the plain internet: its curl FAILS
 #       and, critically, does NOT return the WAN IP (a WAN IP = LEAK = test FAILS).
-#       The host nftables kill-switch blackholes egress-class=vpn traffic while tun0
-#       is absent (ADR-0015 invariant 3/4: vpn fails CLOSED).
+#       vpn-class egress is blackholed while tun0 is absent (ADR-0015 invariant 3/4:
+#       vpn fails CLOSED).
+#
+#       MECHANISM CORRECTED 2026-09-12: this comment used to credit "the host nftables
+#       kill-switch". That table never sees this traffic — Cilium's eBPF egress-gateway
+#       datapath (bpf.masquerade) bpf_redirect()s the steered skbs to the egress device
+#       without invoking netfilter POSTROUTING, measured with counters on the gateway
+#       node (0 packets for the pod CIDR and the PIA range in forward and postrouting,
+#       against 45166 on the same hook for WAN traffic). What fails closed is CILIUM's
+#       own drop when it cannot use the cached egress interface; tcpdump on both the WAN
+#       and tun0 during the stop caught 0 packets. The ASSERTION below is unchanged and
+#       still the right one — only the explanation was wrong.
+#
+#       PREREQUISITE, AND IT IS NOT OPTIONAL: this test cannot run at all unless a node
+#       carries homelab.keithjackson.dev/egress-gateway=true (the fault Job is pinned to
+#       it by nodeSelector, and with no gateway there is no tunnel to stop).
+#
+#       RECOVERY DEPENDS ON THE RE-STEER PATH. Stopping and starting the tunnel gives
+#       tun0 a NEW ifindex, and Cilium does not re-resolve `interface: tun0` on its own,
+#       so before 2026-09-12 the recovery step could not pass: measured, the map stayed
+#       on the old ifindex and vpn-class egress never came back. roles/vpn_egress_gateway
+#       now publishes the live uplink as a node label on every connect, which makes
+#       Cilium re-resolve; with it, stop->start recovers unaided (map 1989 -> 1990, PIA IP
+#       returns). If this test's recovery step ever fails again, check
+#       `journalctl -u vpn-egress-resteer` on the gateway node first.
 # AND   the `internet`-class workload is UNAFFECTED (invariant 3: internet fails OPEN).
 # WHEN  the tunnel RECOVERS (service restarted), THEN vpn-class egress returns (PIA IP).
 #

infra/ansible/roles/vpn_egress_gateway/README.md

@@ -12,12 +12,19 @@ separate GitOps authored against the shared **interface contract** below.
 > once the tunnel is live in dev/test.
 
 ## Interface contract (shared with the Cilium-policy agent)
-- **Inventory group** `vpn_egress_gateway` — the egress-gateway node(s) (initially `k3s-dev-3`).
+- **Inventory groups** — `vpn_egress_gateway` (dev: `k3s-dev-3`) and `vpn_egress_gateway_prod`
+  (prod: `k3s-prod-3`). Each holds **exactly one** host, and prod is a separate group so the
+  flagless playbook run cannot reach it. Cilium OSS resolves a multi-match nodeSelector to "the
+  first node in lexical ordering", so a second labelled node does not add failover — it silently
+  steals the role.
 - **k8s node label** `homelab.keithjackson.dev/egress-gateway=true` — set by this role; the
-  `CiliumEgressGatewayPolicy` selects the gateway node by it.
+  `CiliumEgressGatewayPolicy` selects the gateway node by it. **This label is what arms the
+  datapath**, which is why the role applies it LAST.
 - **Host tunnel iface** `tun0` — the Cilium policy SNATs vpn-class pods out of it.
-- **Optional fwmark** `0x1c8` — a contract *hook* (see kill-switch layering) the Cilium side
-  MAY stamp on vpn-class packets for a tunnel-down-independent host backstop.
+- **k8s node label** `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` — set by this
+  role's re-steer script. Its VALUE is the uplink Cilium was last told about; its CHANGE is what
+  makes Cilium re-resolve `interface: tun0`. Cilium caches that resolution and never revisits it
+  on its own, so without this the gateway is dead after its first reconnect. See "Re-steering".
 
 ## How the four ADR-0015 invariants are met
 
@@ -45,19 +52,42 @@ Two independent layers, both scoped so they can only ever affect vpn-class traff
 2. **nftables backstop (`templates/vpn-egress-killswitch.nft.j2`).** Own `inet
    vpn_egress_gateway_killswitch` table, base chain **policy `accept`** (so it can only *drop* the
    narrow matches, never break anything). In `postrouting`:
-   - `oifname tun0 ip saddr @vpn_egress_gateway_src accept` — tunnel up: let steered traffic out.
-   - `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` — **fail closed:** a tunnel-sourced packet
-     trying to leave any other interface is dropped (no WAN leak). This rule stays **armed even
-     while the tunnel is down** because the source set is retained on tunnel-down (see below).
-   - `meta mark 0x1c8 oifname != tun0 drop` — the optional fwmark backstop.
+   - `oifname tun0 ip saddr @vpn_egress_gateway_src counter accept` — tunnel up: let
+     tunnel-sourced traffic out.
+   - `ip saddr @vpn_egress_gateway_src oifname != tun0 counter drop` — **fail closed:** a
+     tunnel-sourced packet trying to leave any other interface is dropped (no WAN leak). Armed
+     even while the tunnel is down, because the source set is retained on tunnel-down (below).
 
-`@vpn_egress_gateway_src` holds the **last live tun0 address**. The up-script **flushes then
-re-adds** the current tunnel source on every connect; the down-script **intentionally retains
-it** (see "Explicit fail-closed backstop" below). Node traffic and `internet`-class traffic are
-never in the set, never carry the mark, and are never targeted — so they always hit
+   **Both rules carry a `counter`, and both were exercised on k3s-prod-3 (2026-09-12):** a
+   source-bound `ping -I 10.72.192.7` with the ip-rule present got 3/3 replies at 144 ms and
+   moved `killswitch_accept`; with the ip-rule deleted, `ip route get` confirmed the packet
+   would go `via 192.168.2.1 dev eno1` and `killswitch_drop` moved **0 -> 3 packets / 252
+   bytes** with 100% loss. The kill-switch is real, not aspirational — but read the next
+   paragraph for what it can and cannot reach.
+
+   **SCOPE, HONESTLY — it does NOT cover vpn-class POD traffic.** A counters-only diagnostic
+   table on the dev gateway recorded, while a vpn-class pod egressed: 0 packets in `forward`
+   and 0 in `postrouting` for the pod CIDR *and* for the PIA range, against 45166 packets on
+   that same postrouting hook for WAN traffic and 16 for a node-originated curl out tun0.
+   Cilium's eBPF egress-gateway datapath (`bpf.masquerade: true`) `bpf_redirect()`s those skbs
+   straight to the egress device, so netfilter POSTROUTING is never invoked for them. What
+   makes vpn-class fail closed is **Cilium's own drop** when the cached egress interface is
+   unusable — tcpdump during a tunnel stop and during a tunnel restart showed 0 packets leaving
+   either the WAN or tun0. This table's real, narrow guarantee is NODE-originated traffic
+   sourced from a tunnel address.
+
+`@vpn_egress_gateway_src` holds **every tun0 address this node has had**. The up-script **adds**
+the current tunnel source on every connect and never flushes (it used to flush first, which left
+a brief moment where a packet still sourced from the previous address matched neither the
+ip-rule nor the drop rule); the down-script **intentionally retains it**. Node traffic and
+`internet`-class traffic are never in the set and are never targeted — so they always hit
 `policy accept` and **fail open toward the WAN** (invariant 4), tunnel up or down. The opposite
 directions are structural, not incidental.
 
+A kill-switch **reload no longer disarms the guard**: the `.nft` file uses `table` + `flush
+table`, which clears rules but leaves a named set's elements intact, so an Ansible run cannot
+empty the set mid-flight.
+
 ### Explicit, edge-independent fail-closed backstop (tunnel-down)
 Earlier the down-script flushed `@vpn_egress_gateway_src`, which **disarmed** the
 `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` rule at exactly the moment the tunnel was
@@ -67,12 +97,15 @@ guarantee that could leak if the edge router NATed that stale source. The down-s
 vpn-class packet that would leave the WAN while the tunnel is down is **explicitly dropped** in
 the host nft table — defense-in-depth, independent of edge-router behavior.
 
-This is safe precisely because the **up-script flushes the set and re-adds the (possibly new)
-tunnel source on every reconnect** (`nft flush set …` then `nft add element …`), so a retained
-stale address self-corrects the instant the tunnel comes back. The set remains scoped to
-vpn-class only; node and internet-class traffic are never members, so **no invariant is
-weakened — only vpn-class is ever affected** (invariants 3 & 4 upheld). The routing blackhole in
-table `51820` remains as the second, independent fail-closed layer.
+The set is now **ADDITIVE** (2026-09-12): the up-script only ever `nft add element`s the live
+source. It used to `nft flush set` first, and that flush was itself a hole — between a reconnect
+and Cilium refreshing its cached egress IP, a packet still sourced from the PREVIOUS address
+matched neither the ip-rule (deleted by the down-script) nor the drop rule (the address had just
+been flushed out of the set). Additive closes it, `add element` is idempotent, and an extra
+element can only ever *tighten* the rule. The set stays scoped to tunnel sources; node and
+internet-class traffic are never members, so **no invariant is weakened** (invariants 3 & 4
+upheld). The routing blackhole in table `51820` remains as the second, independent fail-closed
+layer.
 
 ## Why the node default route is provably unaffected (design-level verification)
 - OpenVPN runs with `route-noexec` and `pull-filter ignore "redirect-gateway"` → it adds **zero**
@@ -82,40 +115,79 @@ table `51820` remains as the second, independent fail-closed layer.
   ip-rules, never `table main` / `table default`.
 - The ip-rule matches **`from <tun0-addr>`** — an address the node itself never sources traffic
   from — so node-originated and internet-class packets never enter the tunnel table.
-- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src` / the fwmark.
+- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src`.
 - Live check (post-deploy): `ip route show table main default` still points at the WAN NIC, and
   `curl ipinfo` from the node shows the WAN IP, with the tunnel both up and forced down.
+  Confirmed on `k3s-prod-3` 2026-09-12: `default via 192.168.2.1 dev eno1` with the tunnel up,
+  node curl = the WAN IP, k3s `/readyz` = ok.
 
-## Kill-switch layering & ESCALATION (honest limit)
-- The nft `@vpn_egress_gateway_src` rule and the routing blackhole robustly cover the case where the
-  tunnel is up but a vpn-class packet is (mis)routed toward the WAN, and flaps.
-- The **tunnel fully-down** case relies on **Cilium Egress Gateway itself failing closed** when
-  its egress interface `tun0` is absent (Cilium 1.16 drops egress-gateway-marked traffic with no
-  valid egress path rather than falling back to the node's default route) **plus** the routing
-  blackhole. That is the ADR's expected behavior.
-- For a host backstop that is **fully independent of Cilium even in a fail-open regression** when
-  the tunnel is down (no tun0 address to key on), the host needs a stable marker on vpn-class
-  packets. The only contract-available marker is the **`0x1c8` fwmark hook** — which the
-  *Cilium-policy* side must stamp. **ESCALATION for the PM:** either (a) confirm the Cilium
-  Egress Gateway 1.16 fail-closed + routing-blackhole guarantee is accepted as sufficient for
-  R-NET-06, or (b) add the shared fwmark to the interface contract so the CiliumEgressGatewayPolicy
-  marks vpn-class traffic and this role's fwmark rule becomes the tunnel-down-independent backstop.
-  This is flagged rather than worked around — no spec was weakened to reach done.
+## Re-steering — Cilium does not follow tun0, and that is the whole ball game
+
+Cilium resolves the policy's `interface: tun0` **once**, caching an (egress IP, egress ifindex)
+pair in its BPF egress map, and does not re-resolve when tun0 changes. Upstream says so: *"it
+does not automatically respond to a change in the gateway node's network configuration... You can
+force a fresh selection by re-applying the Egress Gateway policy."* Measured on `k3s-dev-3`:
+
+| what | observed |
+|---|---|
+| after `systemctl restart vpn-egress-tunnel` | node tun0 at ifindex 1987, `cilium-dbg bpf egress list` still 1986 |
+| vpn-class pod egress in that state | times out; tcpdump shows **0** packets out the WAN and **0** out tun0 (no leak, no service) |
+| `egress-gateway-reconciliation-trigger-interval: 1s` | does **not** help — dev has that setting; still stale 20s+ later. It coalesces reconciliations that k8s object events already triggered; netlink link events are not among them |
+| stale egress **IP** with a live ifindex | does NOT leak either — the redirect follows the cached *ifindex*, so packets leave tun0 with the previous source (0 packets on the WAN, verified) |
+| touching any node label on the gateway | cached ifindex 1986 -> 1987 within 10s, egress recovers immediately |
+
+So the failure mode is a **silent outage on every tunnel restart or reboot**, not a leak. The fix
+is `templates/vpn-egress-resteer.sh.j2`, which publishes `<tun0 addr>-<ifindex>` as the
+`egress-uplink` node label:
+
+- **a node label, not an annotation on the policy** — the policy is Flux-managed GitOps and
+  writing to it from a node invites drift/ownership churn; node labels are already this role's
+  territory, and the label value doubles as the diagnostic (`kubectl get nodes -L
+  homelab.keithjackson.dev/egress-uplink`);
+- **fired by the OpenVPN up-script** on every connect, via `systemctl start --no-block` so it can
+  never block or fail the tunnel and does not need the kube-API to be up at that instant;
+- **plus `vpn-egress-resteer.timer`** (2 min) as the convergence backstop for the boot case where
+  the API is not up yet and every in-script retry is exhausted;
+- **idempotent** — if the label already records the live uplink it exits without writing, so the
+  timer is free to run often.
+
+Proof it works: `systemctl stop` then `start` of the tunnel now recovers on its own (map
+1989 -> 1990, PIA IP returns). Before the fix the identical sequence stayed broken indefinitely —
+which also means the **recovery half of the `vpn-killswitch` conformance test could not have
+passed** on the pre-fix implementation.
 
 ## What requires a LIVE tunnel to validate (cannot be tested without operator creds)
 1. The PIA `.ovpn` + username/password actually authenticate and `tun0` comes up as a host iface.
 2. `egress-class`: a vpn-class pod's public IP is the PIA Stockholm IP; internet-class shows the WAN IP.
 3. `vpn-killswitch`: forcing `tun0`/the tunnel service down blackholes vpn-class egress (no leak)
    while internet-class is unaffected — the R-NET-06 fault-injection gate.
-4. That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped
-   rules actually see the packets (eBPF datapath interaction — see escalation).
+4. ~~That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped
+   rules actually see the packets (eBPF datapath interaction).~~ **ANSWERED 2026-09-12: it does
+   NOT.** Counters proved the steered pod packets never enter netfilter at all. See "SCOPE,
+   HONESTLY" above. The question is closed; the guard's claim is narrowed rather than pretended.
 5. MTU/PMTU across `pod → tun0 → PIA` (in-pod gluetun needed 1300 for Cilium VXLAN; on the node
    the tunnel rides the physical WAN, so `vpn_egress_gateway_tun_mtu` defaults empty — verify live).
 6. Port forwarding (PIA NAT-PMP) for qBittorrent — ADR-0015 Phase-6 follow-up, not in this role.
 
 ## Files
 - `defaults/main.yml` — all `vpn_egress_gateway_*` vars; the three vault-backed credential vars.
-- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, node label.
+- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, re-steer, node label
+  (kill-switch -> tunnel -> re-steer -> **label last**, because the label is what arms steering).
 - `handlers/main.yml` — daemon-reload, kill-switch reload, tunnel restart.
-- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, both systemd units.
-- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys: `inventory/group_vars/vpn_egress_gateway/README.md`.
+- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, the kill-switch and
+  tunnel units, and `vpn-egress-resteer.{sh,service,timer}`.
+- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys:
+  `inventory/group_vars/vpn_egress_gateway/README.md` (dev) and
+  `inventory/group_vars/vpn_egress_gateway_prod/README.md` (prod).
+
+## Running it against prod
+```bash
+cd infra/ansible
+ansible-playbook playbooks/vpn.egress.gateway.yml \
+  -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts
+# confirm it says exactly `k3s-prod-3`, then drop --list-hosts
+```
+The target var is `vpn_egress_gateway_hosts`, **not** the cluster-wide `k3s_hosts`. Never apply
+the `egress-gateway` label by hand, and never run this role with tags that skip the tunnel or
+kill-switch tasks — the label is the last step precisely so nothing is steered at a node that is
+not ready to carry it.

platform/vpn-egress/README.md

@@ -8,8 +8,12 @@ Egress is realized per **[ADR-0015](../../../home-network-k3s/decisions/adr-0015
 by a **Cilium Egress Gateway** policy, with a **host nftables kill-switch**. There is **no in-cluster
 gateway pod** — the tunnel is host infrastructure, not a Flux workload.
 
-> **STATUS: live + conformance-green.** The node tunnel was validated end-to-end; the `egress-class`
-> (R-NET-05/07) and `vpn-killswitch` (R-NET-06) conformance tests pass against the live cluster.
+> **STATUS: live on BOTH clusters** — `k3s-dev-3` and, since 2026-09-12, `k3s-prod-3`.
+> The `egress-class` (R-NET-05/07) and `vpn-killswitch` (R-NET-06) tests pass on dev.
+> **Caveat on that "conformance-green":** the `vpn-killswitch` test stops and then restarts the
+> gateway tunnel, and until 2026-09-12 a restarted tunnel never recovered (Cilium kept a stale
+> egress ifindex — see below), so the test's recovery step could not have passed on the
+> implementation as it stood. Whatever produced the earlier green result, re-run it.
 
 ## How it works (ADR-0015)
 Three layers, two owners:
@@ -24,9 +28,101 @@ Three layers, two owners:
   (Kyverno-enforced) and SNATs them out the gateway node's `tun0`. Native Cilium eBPF — no client-pod
   changes, no sidecar, no overlay. `excludedCIDRs` keeps RFC1918 / cluster / LAN traffic on the local
   datapath; only public destinations leave via the tunnel.
-- **Kill-switch — host nftables (Ansible).** A host nftables rule on the gateway node DROPS the steered
-  `vpn`-class traffic whenever `tun0` is down (fail-closed, R-NET-06), scoped to vpn-class only so
-  `internet`-class and node traffic are unaffected.
+- **Kill-switch — Cilium's own drop, NOT host nftables (corrected 2026-09-12).** vpn-class fails
+  closed because Cilium drops the traffic when it cannot use its cached egress interface. The host
+  nftables table is retained for a narrower job. See "The kill-switch does not do what this file
+  used to claim" below — that section is the measurement, not an opinion.
+
+## LIVE ON PROD since 2026-09-12 — `k3s-prod-3`
+
+| | dev | prod |
+|---|---|---|
+| gateway node | `k3s-dev-3` | `k3s-prod-3` |
+| inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) |
+| vpn-class egress IP | `158.173.241.36` (PIA Stockholm) | `158.173.241.36` (PIA Stockholm) |
+| internet-class egress IP | home WAN | `136.51.115.252` (home WAN) |
+
+Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three
+nodes — all egress the PIA address; plex and nextcloud (internet-class) still egress the home
+WAN. **Before the gateway existed, prod's vpn-class pods had no internet at all**: the CEGP was
+reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress list` showed
+`Gateway IP: Not Found` for their `0.0.0.0/0` entry and the traffic was dropped. Arming the
+gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also
+means nothing had been leaking in the meantime.
+
+Both environments currently share one PIA account and, as it happens, one public egress IP.
+That is inside the account's connection limit, but it means the two environments are not
+distinguishable from the outside; a separate prod PIA account is the clean fix if that ever
+matters (see `infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`).
+
+## The kill-switch does not do what this file used to claim (measured 2026-09-12)
+
+This README, `egress-steering.yaml`, and the role all used to say the **host nftables
+kill-switch** is the authoritative fail-closed backstop for `vpn`-class traffic. **It cannot be.**
+With `bpf.masquerade: true` — which Egress Gateway requires — the SNAT and the egress happen in
+eBPF, which `bpf_redirect()`s the skb straight to the egress device. netfilter POSTROUTING is
+never invoked for those packets.
+
+A counters-only nft table installed alongside the real one on the gateway node, while a vpn-class
+pod egressed:
+
+| counter | packets |
+|---|---|
+| `forward`, saddr = pod CIDR | **0** |
+| `forward`, saddr = PIA tunnel range | **0** |
+| `postrouting`, oif tun0, saddr = tunnel range | **0** |
+| `postrouting`, oif WAN, saddr = pod CIDR | **0** |
+| `postrouting`, oif WAN, any (control: the hook IS live) | 45166 |
+| `postrouting`, oif tun0, saddr = tunnel (control: a node-originated curl) | 16 |
+
+**What actually fails closed is Cilium itself.** When it cannot use its cached egress
+interface/IP it drops the packet. Verified with tcpdump on *both* the WAN and tun0, during a
+tunnel stop and during a tunnel restart: **0 packets left either device.** There is no leak — the
+guarantee is real — but the mechanism is Cilium's, and the existing `vpn-killswitch` conformance
+test passes because of it, not because of the nft rules.
+
+The nft table is **kept**, with a `counter` on every rule, for the narrower thing it genuinely
+does cover: **node-originated** traffic sourced from a tunnel address. That was proven on
+`k3s-prod-3` in both directions — a source-bound ping with the tunnel ip-rule present got 3/3
+replies and moved `killswitch_accept`; with the ip-rule removed (so the routing table sent it out
+`eno1`) `killswitch_drop` moved **0 -> 3 packets / 252 bytes** and the ping got nothing back.
+If `killswitch_drop` ever starts counting in normal operation, the datapath has changed — read
+that as news, not reassurance.
+
+Also removed: the `meta mark 0x1c8 oifname != tun0 drop` "contract hook". Nothing stamps that
+mark (the only Cilium mark on the gateway node is `0x200/0xf00`, the L7-proxy rule to table
+2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is
+a hazard to whatever component picks that value next. The role's `vpn_egress_gateway_fwmark`
+variable is gone with it, and the "add the fwmark to the interface contract" escalation in the
+role README is **closed as not-implementable** rather than left open.
+
+## Cilium never re-resolves `tun0` — the gateway dies on every reconnect without help
+
+Cilium resolves `interface: tun0` once into a cached (egress IP, egress ifindex) and does not
+revisit it. Every tunnel restart re-creates tun0 with a **new ifindex**, so the cached one goes
+stale and vpn-class egress is silently blackholed. Found the hard way: the dev gateway had been
+dead for ~80 minutes when this work started, with no alert and nothing in the ledger.
+
+`egress-gateway-reconciliation-trigger-interval: 1s` does **not** help; both clusters carry that
+setting and dev still did not self-heal 20s+ after the tunnel was back. It coalesces
+reconciliations that k8s object events already triggered — netlink link events are not among them.
+
+The Ansible role now publishes `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on
+the gateway node on every connect, plus a 2-minute convergence timer. That node update makes
+Cilium re-process the policy and re-resolve the interface (verified: cached ifindex 1986 -> 1987
+within 10s, egress recovered immediately). A node label rather than an annotation on this
+Flux-managed policy, so GitOps state is never written to from a node.
+
+**Debugging a dead gateway starts here:**
+
+```bash
+kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink
+kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
+  cilium-dbg bpf egress list      # Egress Ifindex must match the node's live `ip link show tun0`
+```
+
+If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
+gateway node is the next stop.
 
 ## Resilience — there is NO gateway failover (spec correction, escalated)
 

platform/vpn-egress/egress-steering.yaml

@@ -59,16 +59,38 @@ spec:
   # Excluded flows fall through to Cilium's normal local datapath (invariant 1).
   destinationCIDRs:
     - "0.0.0.0/0"
+  #
+  # BULK NFS (added to the record 2026-09-12, when plex/sonarr/radarr/nextcloud gained media
+  # mounts): the export is 192.168.2.102:/mnt/Storage/plex on prod (192.168.1.9 on dev), both
+  # inside 192.168.0.0/16 below, so NFS is excluded from the tunnel twice over. The FIRST reason
+  # is the one that actually matters and needs no CIDR at all: an NFS PersistentVolume is mounted
+  # by the NODE KERNEL in the host netns, not by the pod, so `selectors.podSelector` never
+  # matches it and no pod-scoped egress steering can touch it. Verified after arming the prod
+  # gateway: sonarr (k3s-prod-1) and radarr (k3s-prod-3, the gateway node itself) both still list
+  # AND write the export, and the node's own mounts are intact. No RFC1918 destination needs
+  # adding here for the media path.
   excludedCIDRs:
     - "10.0.0.0/8"       # RFC1918 + Cilium cluster-pool pod CIDR (CLUSTER_POD_CIDR) + k3s service net
     - "172.16.0.0/12"    # RFC1918
-    - "192.168.0.0/16"   # RFC1918 (home LAN / node network family)
+    - "192.168.0.0/16"   # RFC1918 (home LAN / node network family / the bulk NFS server)
   # --- Where it egresses --------------------------------------------------
-  # The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`. When
-  # tun0 is absent/down, Cilium has no egress interface here; the host nftables kill-switch (Ansible,
-  # ADR-0015) is the authoritative fail-closed backstop that DROPS this vpn-class traffic — scoped to
-  # vpn-class only, so default/internet-class is unaffected (invariants 3 & 4: vpn fails CLOSED,
-  # internet fails OPEN).
+  # The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`.
+  #
+  # CORRECTION 2026-09-12 — this comment used to say the host nftables kill-switch "is the
+  # authoritative fail-closed backstop that DROPS this vpn-class traffic". It is not, and cannot
+  # be. With `bpf.masquerade: true` the SNAT and egress happen in eBPF, which bpf_redirect()s the
+  # skb to the egress device without invoking netfilter POSTROUTING, so the host nft rules never
+  # see this traffic. Measured with a counters-only nft table on the gateway node while a
+  # vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and
+  # the PIA range, versus 45166 packets on that same hook for WAN traffic (so the hook is live)
+  # and 16 for a node-originated curl out tun0 (so the matching works).
+  # WHAT ACTUALLY FAILS CLOSED: Cilium itself. When the cached egress interface/IP cannot be used
+  # it drops the packet — tcpdump on both the WAN and tun0 during a tunnel stop AND during a
+  # tunnel restart caught 0 packets leaving either device. vpn fails CLOSED (verified, no leak);
+  # internet-class and node traffic are never selected and fail OPEN (invariants 3 & 4 hold).
+  # The host nft table is retained, with counters, for the narrower case it CAN see:
+  # node-originated traffic sourced from a tunnel address (proven on k3s-prod-3 — the drop rule's
+  # counter moved 0 -> 3 packets when such a packet was routed at the WAN).
   #
   # SINGLE GATEWAY ONLY — there is NO failover, and a second labelled node is NOT a standby.
   # `egressGateway` (singular) resolves the nodeSelector to ONE node. Verbatim from the CRD
@@ -89,6 +111,22 @@ spec:
   # ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)"
   # is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see
   # platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06.
+  #
+  # `interface: tun0` IS RESOLVED ONCE — the gateway node has to tell Cilium when tun0 changes.
+  # Measured on k3s-dev-3 2026-09-12: Cilium caches an (egress IP, egress ifindex) pair from the
+  # first resolution and does NOT re-resolve on a netlink change. `systemctl restart
+  # vpn-egress-tunnel` re-creates tun0 with a NEW ifindex; the map kept the old one and vpn-class
+  # egress was silently blackholed (tcpdump: 0 packets out the WAN, 0 out tun0 — fail-closed, but
+  # dead). `egress-gateway-reconciliation-trigger-interval: 1s` does not cover link events.
+  # Upstream: "it does not automatically respond to a change in the gateway node's network
+  # configuration... You can force a fresh selection by re-applying the Egress Gateway policy."
+  # The Ansible role closes this: roles/vpn_egress_gateway publishes
+  # `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node on every
+  # connect (plus a 2-minute convergence timer). That node update makes Cilium re-process this
+  # policy and re-resolve the interface — verified: cached ifindex 1986 -> 1987 within 10s.
+  # DO NOT "fix" this by having the node write to THIS object: it is Flux-managed, and a node
+  # writing to GitOps state is a drift fight. Check the label, not this file, when debugging:
+  #   kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink
   egressGateway:
     nodeSelector:
       matchLabels: