merge: VPN egress gateway on prod, plus three datapath fixes

2026-09-12 18:15:02 -05:00
commit 19fc8bcb27
@@ -2089,3 +2089,245 @@ fail-closed branch fixes the "unmapped path contributes nothing" hole; it does n
 Both belong to `ci/`, so they are left to that branch rather than smuggled in here. The full L0
 layer was run unscoped on this branch to compensate: **12 offline checks, 11 PASS, 1 SKIP**
 (`placeholder-guard-coupling`, pre-existing), 0 FAIL.
+
+## 2026-09-12 — VPN egress on PROD (`k3s-prod-3`), and three things the repo believed that are not true (branch `feat/vpn-egress-prod`)
+
+The operator asked for VPN egress on prod. An earlier assessment this session had recommended
+holding off over three suspected defects; the operator decided, so the defects were fixed first
+and then the gateway was stood up. All three were settled by **measurement on the dev gateway
+node**, not by reading code — and two of the three turned out to be different from what was
+suspected, in one case much worse.
+
+### The headline: the DEV gateway was already dead, silently, and had been for ~80 minutes
+
+`cilium-dbg bpf egress list` on `k3s-dev-3` showed `Egress Ifindex 3`. There is no ifindex 3 on
+that node (the list goes 1, 2, 4, 5, …) — it was a **destroyed** tun0. The live tun0 was ifindex
+1780. vpn-class pods on both the gateway node and a remote node timed out. Nothing alerted, and
+the ledger still recorded the gateway as conformance-green.
+
+**Cilium resolves a `CiliumEgressGatewayPolicy`'s `interface: tun0` exactly once**, caching an
+(egress IP, egress ifindex) pair, and does not re-resolve when tun0 changes. Upstream is explicit:
+*"it does not automatically respond to a change in the gateway node's network configuration… You
+can force a fresh selection by re-applying the Egress Gateway policy."* Every tunnel restart
+re-creates tun0 with a new ifindex, so **the gateway is dead after its first reconnect or reboot.**
+`egress-gateway-reconciliation-trigger-interval: 1s` does not cover it — both clusters carry that
+setting and dev was still stale 20s+ after recovery; the interval coalesces reconciliations that
+k8s object events already triggered, and netlink link events are not among them.
+
+Measured, in order:
+
+| step | observation |
+|---|---|
+| baseline | map ifindex 3, live tun0 1780, vpn-class pods time out |
+| annotate the CEGP (the upstream remedy) | map -> 1780 within 8s; pods return the PIA IP `158.173.241.36` immediately |
+| `systemctl restart vpn-egress-tunnel` | tun0 1780 -> 1985, map STAYS 1780, pods break again |
+| `systemctl stop`, wait 30s, `start` (the exact `vpn-killswitch` conformance sequence) | tun0 -> 1990, map stays 1989, pods still broken 20s+ later |
+| set ANY unrelated label on the gateway NODE | map -> live ifindex within 10s, egress recovers |
+
+**Fix:** `roles/vpn_egress_gateway` now publishes
+`homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node — from the
+OpenVPN up-script on every connect (via `systemctl start --no-block`, so it can never block or
+fail the tunnel and does not need the kube-API up at that instant), plus a 2-minute convergence
+timer for the boot case. A **node label, not an annotation on the policy**: the policy is
+Flux-managed GitOps and a node writing to GitOps state is a drift fight, whereas node labels are
+already this role's territory — and the label's value doubles as the diagnostic. After the fix,
+stop->start recovers unaided (map 1989 -> 1990, PIA IP returns).
+
+**Consequence for the ledger:** the `vpn-killswitch` (R-NET-06) test stops *and restarts* the
+tunnel. Its recovery step **could not have passed** on the implementation as it stood — verified
+twice. Whatever produced the earlier green result needs re-running; the fail-closed half was and
+remains sound.
+
+### Defect as reported vs. as measured
+
+**Reported:** the up-script's `nft flush set` opens a leak window — on reconnect Cilium keeps
+SNAT-ing to the previous address A while only B is in the set, so packets sourced from A match
+neither the ip-rule nor the drop rule and **leave via the WAN**.
+
+**Measured: there is no WAN leak.** Cilium's redirect follows the cached *ifindex*, not a routing
+lookup, so a stale egress IP cannot reach the WAN. Tested directly by swapping tun0's address in
+place (keeping the ifindex, as an OpenVPN `persist-tun` soft reconnect does) while Cilium held the
+old one: the pod's packets left **tun0** carrying the stale source, PIA answered, and tcpdump on
+the WAN caught **0** packets. Same result in the stale-ifindex case: 0 packets out the WAN, 0 out
+tun0 — the redirect to a dead ifindex simply fails and the packet is dropped.
+
+The flush was still wrong, so the set is now **additive** (`add element` is idempotent; the
+down-script already retained it). It cannot leak pod traffic — see the next section for why — but
+node-originated tunnel traffic *does* traverse netfilter, and additive costs nothing. A kill-switch
+reload no longer disarms the guard either: the `.nft` file uses `table` + `flush table`, which
+clears rules but leaves a named set's elements intact.
+
+### The nftables kill-switch cannot see the traffic it was written to guard
+
+Suspected; now proven, with the counters the brief asked for. A counters-only diagnostic nft table
+on the gateway node, while a vpn-class pod egressed:
+
+| counter | packets |
+|---|---|
+| `forward`, saddr = pod CIDR | **0** |
+| `forward`, saddr = PIA tunnel range | **0** |
+| `postrouting`, oif tun0, saddr = tunnel range | **0** |
+| `postrouting`, oif WAN, saddr = pod CIDR | **0** |
+| `postrouting`, oif WAN, any — control, the hook is live | 45166 |
+| `postrouting`, oif tun0, saddr = tunnel — control, a node-originated curl | 16 |
+
+With `bpf.masquerade: true` (which Egress Gateway requires) the SNAT and egress happen in eBPF,
+which `bpf_redirect()`s the skb to the egress device without invoking netfilter POSTROUTING. The
+hook works and the matching works — the steered pod packets are simply never presented to it.
+
+**So what fails closed is Cilium's own drop**, confirmed by tcpdump on both devices during a
+tunnel stop and a tunnel restart: 0 packets left either. The existing `vpn-killswitch` test passes
+because of that, not because of the nft rules. Every comment in the tree that credited the nft
+table has been corrected rather than left to mislead the next reader.
+
+**The table is kept, not deleted**, because it does cover something real — node-originated traffic
+sourced from a tunnel address — and that was proven in **both** directions on `k3s-prod-3`:
+
+- source-bound `ping -I 10.72.192.7` with the tunnel ip-rule present: 3/3 replies at 144 ms,
+  `killswitch_accept` advances;
+- ip-rule removed, so `ip route get` says `via 192.168.2.1 dev eno1`: **`killswitch_drop`
+  0 -> 3 packets / 252 bytes**, 100% loss.
+
+Every rule now carries a `counter` so the claim stays falsifiable. If `killswitch_drop` ever
+counts in normal operation, the datapath has changed.
+
+Also **removed** the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable. Nothing stamps
+that mark — the only Cilium mark on the gateway node is `0x200/0xf00`, which is the L7-proxy rule
+to table 2004 — and the traffic never reaches netfilter anyway. A live rule matching an arbitrary
+mark is a hazard to whatever component picks that value next, not a backstop. This **closes** the
+role README's standing escalation ("add the shared fwmark to the interface contract so the
+CiliumEgressGatewayPolicy marks vpn-class traffic") as **not implementable on Cilium OSS**: there
+is no supported way to make the policy stamp a mark, and no host firewall hook to catch it in.
+
+### Prod stand-up
+
+`[vpn_egress_gateway_prod]` is a **separate** inventory group holding only `k3s-prod-3`, with its
+own `group_vars/vpn_egress_gateway_prod/vault.yml`. Prod is deliberately NOT a member of
+`[vpn_egress_gateway]`, which is the default target of `playbooks/vpn.egress.gateway.yml` — sharing
+the group would mean a flagless run silently reconfigures a prod node's kill-switch datapath. Run
+as `-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3`, with `--list-hosts`
+first (confirmed: exactly `k3s-prod-3`).
+
+Node 3 because node 1's SSD was replaced 2026-09-11 (least operational history) and node 2 carries
+the §0b memattr history. **Exactly one node, forever**: Cilium OSS picks "the first node in lexical
+ordering" among nodeSelector matches, so labelling `k3s-prod-1` would silently take the gateway
+role, and the public egress IP, away from `k3s-prod-3`.
+
+The vault file is a **verbatim copy of dev's ciphertext** — same PIA account per the operator, and
+copying the encrypted blob means the plaintext never touched disk, a shell, or a transcript. The
+two files are independent from here on; rotating means re-encrypting both, which is correct for two
+environments.
+
+**`wlp1s0` did not need bringing down — it does not exist.** The brief and `hosts.ini` both said
+node 3 has wlp1s0 up on 192.168.2.195. Re-verified before choosing it, because a second live uplink
+in a kill-switch datapath is not a footnote: there is no `wl*` netdev on the node, no `phy80211`
+device, the `wlp1s0` stanza in `/etc/network/interfaces` is commented out, 192.168.2.195 is
+assigned nowhere, and the only IPv4 addresses are `lo` and `eno1 192.168.2.12`. The inventory
+comment is corrected.
+
+**Result on prod:**
+
+| | before | after |
+|---|---|---|
+| jackett / qbittorrent / radarr / sonarr (vpn-class, spread across all 3 nodes) | no internet at all — CEGP reconciled with no matching node, `cilium-dbg bpf egress list` showed `Gateway IP: Not Found`, traffic dropped | all four egress `158.173.241.36` (PIA Stockholm) |
+| plex / nextcloud (internet-class) | `136.51.115.252` (home WAN) | `136.51.115.252`, unchanged |
+| node's own egress | WAN | WAN — `default via 192.168.2.1 dev eno1`, k3s `/readyz` ok |
+
+Worth stating plainly: arming the gateway **restored** vpn-class egress rather than moving it from
+the WAN to the tunnel. Nothing had been leaking in the meantime, because nothing had been going
+out.
+
+### The NFS media mounts are unaffected, verified rather than assumed
+
+sonarr, radarr, plex and nextcloud gained NFS mounts from `192.168.2.102:/mnt/Storage/plex` (and
+`/mnt/Storage/nextcloud`) earlier today, so the interaction with pod-scoped egress steering was
+checked before and after arming:
+
+- **The structural reason it cannot break:** an NFS PersistentVolume is mounted by the **node
+  kernel in the host netns**, not by the pod, so the CEGP's `selectors.podSelector` never matches
+  it. No `excludedCIDRs` entry is needed for that.
+- **And belt-and-braces:** the server is `192.168.2.102`, inside the existing
+  `192.168.0.0/16` exclusion anyway.
+- **Empirically, after arming:** sonarr (on `k3s-prod-1`) and radarr (on `k3s-prod-3`, the gateway
+  node itself) both still `ls` **and write** to `/media`; plex lists it; the node's own
+  `findmnt`-visible mounts are intact and readable. qBittorrent, incidentally, has **no** NFS
+  media mount in this repo — only plex, sonarr, radarr and nextcloud do.
+
+### A fourth defect, created by the stand-up itself: one pinned PIA server cannot serve two gateways
+
+Prod came up green, and **dev went down.** Not the connection limit — the operator's "same account,
+well inside its limit" is true. The vaulted `.ovpn` pins **one** server, so both gateways dialled
+it, and PIA handed **both sessions the same tunnel address**. Verbatim from both journals:
+
+```
+[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198
+PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ...
+```
+
+Each reconnect steals `10.72.192.7` on that server; the loser then receives nothing, trips
+`--ping-restart` after 60s, reconnects, and steals it back. Measured: dev restarting at 22:57:05,
+22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC, prod interleaved about a minute off — a ~2-minute
+mutual-eviction cycle with no end. Prod's first probes looked green only because they landed
+inside an up-window; that is a good reminder that a single green probe is not a stability check.
+
+**Fix:** a new role var `vpn_egress_gateway_pia_remote` ("<host> <port>", empty = honour the
+.ovpn) rewrites only the `remote` line of the deployed .ovpn, and
+`group_vars/vpn_egress_gateway_prod/vars.yml` points prod at `158.173.241.135`. That is another
+member of the same PIA Stockholm LB pool — `sweden.privacy.network` resolved from `k3s-prod-3` to
+`158.173.241.{34,55,112,135,239}`, and every server in the pool accepts the same credentials and
+the same inline CA, so only the address differs. Dev keeps the .ovpn's `.112` untouched, which is
+why the override is per-group rather than in the role or a second vaulted .ovpn. A PIA server
+address is not a credential, so it sits in plain group_vars next to the vault, reviewable.
+
+**Standing rule this establishes: one gateway per PIA server.** A third gateway, or a rebuild that
+re-pins, must take an unused address from the pool.
+
+**Dev recovered the moment prod left the shared server.** Dev's last `Inactivity timeout` is
+23:05:06 UTC — the minute prod switched to `.135`. Re-checked at 23:12:03: the 23:05:07 session is
+still the live one, seven minutes unbroken against a previous cycle of two, and a source-bound ping
+through the tunnel returns 4/4 at 146 ms with 0% loss. vpn-class pod egress from dev is back to
+`158.173.241.36` on three consecutive probes.
+
+**And the switch handed us the best possible proof of the re-steer fix, unplanned.** Moving prod to
+`.135` gave it a genuinely NEW tunnel address on a real reconnect — `10.72.192.7` (ifindex 128) ->
+`10.150.192.86` (ifindex 129) — which is exactly the case Cilium does not follow and exactly what
+defect 1 was about. The `egress-uplink` label moved to `10.150.192.86-129` by itself, Cilium
+re-resolved, and all four vpn-class workloads kept egressing (now `158.173.241.11`). No manual
+step, no CEGP edit, no operator. Before this branch that address change would have left prod's
+vpn-class egress dead until someone noticed.
+
+### Left for the operator / not settled here
+
+1. **Re-run `vpn-killswitch` (R-NET-06) against a labelled cluster** and refresh the ledger. It can
+   only run where a node carries the gateway label, and its recovery step is only passable with
+   this branch's re-steer fix. It is an L3 fault-injection test that stops the tunnel on a LIVE
+   cluster, so it was not fired at prod unasked.
+2. **Decide whether prod gets its own PIA account.** After the server split the two environments do
+   have distinct exit IPs (dev `158.173.241.36`, prod `158.173.241.11`), so the immediate problem is
+   solved — but they still share one account, and the per-environment PIA server is now a
+   hand-maintained value in `group_vars/vpn_egress_gateway_prod/vars.yml` that must not collide with
+   dev's. A separate prod account removes that coupling permanently.
+3. **Gateway-node loss is still an outage** — Cilium OSS has no egress-gateway failover, and the
+   ADR-0016 supersede drafted on 2026-09-06 is still open. Nothing here changes that; the
+   single-gateway choice is now recorded in the inventory itself, not just in prose.
+4. **A silent gateway outage still has no alert.** The re-steer timer means it now self-heals, but
+   nothing watches the invariant "the gateway node's `egress-uplink` label matches
+   `cilium-dbg bpf egress list`'s Egress Ifindex". That is a monitoring gap, not a datapath one,
+   and it is the thing that let dev sit dead for 80 minutes. A Prometheus rule over a small
+   node-exporter textfile from the re-steer script would close it.
+5. **Hygiene, unrelated but found on the way:** `/etc/network/interfaces` on `k3s-prod-3` holds a
+   commented-out wireless stanza containing the home WiFi **PSK in plaintext**. It is inert
+   (wireless is gone) and not printed here. Worth scrubbing on all three nodes.
+6. **`ADR-0015` and `docs/03` still describe the nftables kill-switch as the vpn-class
+   fail-closed mechanism**, and specs are immutable to agents. Drafted amendment for the architect:
+   > *R-NET-06's fail-closed guarantee for `egress-class=vpn` is provided by the Cilium Egress
+   > Gateway datapath, which drops steered traffic when the gateway's egress interface cannot be
+   > used; this was verified with packet capture on both the WAN and tunnel devices. The host
+   > nftables table is a defence-in-depth guard for node-originated traffic sourced from the tunnel
+   > address only — with `bpf.masquerade` enabled, which Egress Gateway requires, steered pod
+   > traffic does not traverse netfilter and cannot be guarded there. The "optional fwmark
+   > contract hook" is withdrawn: Cilium OSS provides no way for the policy to stamp such a mark.*
+   > *Additionally, `interface:` on a CiliumEgressGatewayPolicy is resolved once and cached; a
+   > gateway whose tunnel interface is dynamic requires an explicit re-resolution trigger on every
+   > interface change. This is a standing operational requirement of the ADR-0015 design, not an
+   > implementation detail.*
@@ -10,8 +10,31 @@
 #       egress-gateway node is STOPPED (fault injection) —
 # THEN  the vpn-class workload reaches NOTHING on the plain internet: its curl FAILS
 #       and, critically, does NOT return the WAN IP (a WAN IP = LEAK = test FAILS).
-#       The host nftables kill-switch blackholes egress-class=vpn traffic while tun0
-#       is absent (ADR-0015 invariant 3/4: vpn fails CLOSED).
+#       vpn-class egress is blackholed while tun0 is absent (ADR-0015 invariant 3/4:
+#       vpn fails CLOSED).
+#
+#       MECHANISM CORRECTED 2026-09-12: this comment used to credit "the host nftables
+#       kill-switch". That table never sees this traffic — Cilium's eBPF egress-gateway
+#       datapath (bpf.masquerade) bpf_redirect()s the steered skbs to the egress device
+#       without invoking netfilter POSTROUTING, measured with counters on the gateway
+#       node (0 packets for the pod CIDR and the PIA range in forward and postrouting,
+#       against 45166 on the same hook for WAN traffic). What fails closed is CILIUM's
+#       own drop when it cannot use the cached egress interface; tcpdump on both the WAN
+#       and tun0 during the stop caught 0 packets. The ASSERTION below is unchanged and
+#       still the right one — only the explanation was wrong.
+#
+#       PREREQUISITE, AND IT IS NOT OPTIONAL: this test cannot run at all unless a node
+#       carries homelab.keithjackson.dev/egress-gateway=true (the fault Job is pinned to
+#       it by nodeSelector, and with no gateway there is no tunnel to stop).
+#
+#       RECOVERY DEPENDS ON THE RE-STEER PATH. Stopping and starting the tunnel gives
+#       tun0 a NEW ifindex, and Cilium does not re-resolve `interface: tun0` on its own,
+#       so before 2026-09-12 the recovery step could not pass: measured, the map stayed
+#       on the old ifindex and vpn-class egress never came back. roles/vpn_egress_gateway
+#       now publishes the live uplink as a node label on every connect, which makes
+#       Cilium re-resolve; with it, stop->start recovers unaided (map 1989 -> 1990, PIA IP
+#       returns). If this test's recovery step ever fails again, check
+#       `journalctl -u vpn-egress-resteer` on the gateway node first.
 # AND   the `internet`-class workload is UNAFFECTED (invariant 3: internet fails OPEN).
 # WHEN  the tunnel RECOVERS (service restarted), THEN vpn-class egress returns (PIA IP).
 #
@@ -0,0 +1,49 @@
+# group_vars for the `vpn_egress_gateway_prod` node — ADR-0015 host tunnel secrets (PROD)
+
+Prod's egress gateway is `k3s-prod-3`, in its **own** inventory group
+(`[vpn_egress_gateway_prod]`) so that the flagless
+`ansible-playbook playbooks/vpn.egress.gateway.yml` — which defaults to the **dev** group —
+can never touch a prod node. Prod is reached only by asking for it:
+
+```bash
+cd infra/ansible
+ansible-playbook playbooks/vpn.egress.gateway.yml \
+  -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts
+# then drop --list-hosts
+```
+
+A separate group needs its own `group_vars`, because the dev group's vault is attached to the
+dev group and is not visible to a prod host. The required keys are **identical** to dev's —
+see `../vpn_egress_gateway/README.md` for what each one is:
+
+| Vault key |
+|---|
+| `vault_vpn_egress_gateway_pia_username` |
+| `vault_vpn_egress_gateway_pia_password` |
+| `vault_vpn_egress_gateway_pia_ovpn` |
+
+## How this file was created (2026-09-12)
+
+`vault.yml` here is a **verbatim copy of the dev group's encrypted blob**. That is deliberate:
+
+- the operator confirmed the **same PIA account** applies to prod and is well inside its
+  connection limit, so the values are the same values;
+- copying the *ciphertext* means the plaintext was never decrypted to disk, never passed
+  through a shell, and never appeared in an agent transcript. The repo-wide vault passphrase
+  (`ansible.cfg` → `vault_password_file`) decrypts both files identically.
+
+The two files are now **independent**: rotating the PIA credentials means re-encrypting
+*both*, and that is the intended behaviour — prod and dev are separate environments and a
+credential change should be a deliberate act per environment, not an accident of sharing a
+file. If prod is ever moved to its own PIA account (recommended if the connection limit ever
+bites, or to keep the public egress IPs of the two environments independent), re-create this
+file alone:
+
+```bash
+cd infra/ansible
+ansible-vault create inventory/group_vars/vpn_egress_gateway_prod/vault.yml
+```
+
+## Notes
+- `vault.yml` is skipped by the yamllint gate (`ci/yamllint.yaml` ignores `**/vault.yml`).
+- **Never** run `ansible-inventory --list`: it prints DECRYPTED vault values.
@@ -0,0 +1,22 @@
+---
+# Non-secret group_vars for the PROD egress gateway (k3s-prod-3). Credentials live in the
+# vaulted vault.yml beside this file.
+
+# PROD DIALS ITS OWN PIA SERVER — and it has to.
+#
+# The vaulted .ovpn pins ONE server, 158.173.241.112. Measured 2026-09-12: with dev
+# (k3s-dev-3) and prod (k3s-prod-3) both dialling it, PIA handed BOTH sessions the SAME
+# tunnel address — `Server-10779-2a ... ifconfig 10.72.192.7` in both journals. Each
+# reconnect steals that address from the other; the loser then sees no data for 60s, hits
+# `--ping-restart`, reconnects and steals it back. Both tunnels flapped on a ~2-minute cycle
+# (dev 22:57 / 22:59 / 23:01 UTC, prod interleaved a minute off) and neither environment had
+# reliable vpn-class egress. This is NOT the account's connection limit — that has room.
+#
+# 158.173.241.135 is another member of the same PIA Stockholm LB pool: `sweden.privacy.network`
+# resolved from k3s-prod-3 to 158.173.241.{34,55,112,135,239}. Any server in the pool accepts
+# the same credentials and the same inline CA, so only the address differs. Dev keeps the
+# .ovpn's pinned .112 untouched, which is why the override lives HERE and not in the role.
+#
+# A public server address is not a credential, so this is plain group_vars, not vault.
+# If prod is ever moved to its own PIA account, this can go away again.
+vpn_egress_gateway_pia_remote: "158.173.241.135 1198"
@@ -0,0 +1,131 @@
+$ANSIBLE_VAULT;1.1;AES256
+38306636313935333030303664653739316233663139616563663233643262373637306331653264
+6538363539383464353964663632633561323562383038340a396233633232356137313435336231
+36333663363130643532613566643339373365626462313537653033306363336564343139373537
+3164323463616632630a373732333536613936366463313439343664363530383664636535616261
+31306332666666643637346364386537343062383033663530346163373536323639373366333661
+36303831323662333232633034393362333561643637623033656565303461336465333664366332
+36336466656437343431363735646661643837666335633464666130643866313830366465323166
+65616537343630303431623537303766383762646533616539613235343938656237613934623565
+35373835633539653333343232626336326638303961373230663561313330313632393066323130
+31383838386262393730666231376362383132646230383964313439663935613861323137646331
+36626365323566316436386662383434366664356166636431663332393665323734623735303239
+62383031613237333136613837356166343436306137353935613661613264326466346337346530
+65356630373235313131656131383965313366613965386662343961653864373964346337363562
+66626635363438656430306231323764346564666132633966353861643262353131623134656137
+39306134626163346336613732326264396438623630656534396565343763303761333538316661
+30623466366232323531633262623434643366613833353564386163303966316537623934336163
+66323933653036366535663266666531666236646431383066616262353733316165666434616534
+35386633313464363233326663363634626239616364383866653135386435636130316335666430
+38323363386462373862366230393764386338643466363161333733653065383966663935303964
+64333461333437653433373131343736333831313236306262373631353461356237386163623166
+30663761306534316439356365396663336135333537663366353866336536663934646133333332
+65353236366435326630653830346631643236326337626435333236386634333137623138316131
+61323365303461353932626336323534633533646233653135396665373765666337333934356662
+36646339376564366439646137306364623165626263383634326337333566663236346366366334
+37316166393763663139336538663266333736623661383062383732303133326233313232323836
+34356130396131346533633336373736663366366231343162633064343565613937366434613664
+31656365303439643837323135613936646133306163663265373933633231616135353335376435
+61633965356538613636663564346566313261306235313438316639626131303566363365663764
+64316562383361373663653034393333653831666239363037343636666364666536343534613066
+61373335353464633963623161353866333038643062303931653134626334393864663461653531
+65373766336437376435616138666563623337373038623566666136626637633463323965666139
+38336139613635383066336164373061663266326235363731663565613336623936373531383966
+62323330636263306665343462666639643530336633346531636263616436356631343763343432
+61636264316566663933346163373163653734623031393962343635653765653964633137346433
+62616138333536336438363564356636316531346566663066376234356230626165386165336165
+38656166633839656438323966376561306338623237303530376231393663663661343362653664
+37613433653634363534306432383061373165613266386562323632636537373736656137633639
+33666634316465383066333737616636363866626330313232326434666164316439333433633461
+35336161313039646264613835613336326332326464313339636532653837653430393939633462
+66656563626163376134623437633332633535336233656237303962623063363134616431623235
+62323265666136323366343565656236646430393237356333613836343438353164313535616138
+34613134663465333734663337303663316161313439626332326334363835376361366463363631
+30396336303761383465373439313666656237356433303532373162306166356165316164656334
+37326163393230306161653332323462333238313166373134343839323461363936353737613662
+61643938333464366433306363633531383264613030316631303034376131323135313136316461
+31666632393631376133373533616639396361633762636663323463363433626537656333653837
+38633632623139356630616637663331616461383361383733613563383165363139343035383333
+32653563303038336137336333653561363037633634323662616666356534323138353438616463
+64383639353733333537616632346230383330356166663433373734386435376563333038333934
+37616536333132303166366235613265336432616239393730353036616337386463363330306532
+65613066363434346562633630336135313736393137656230306163383730356133326561333431
+35393466323936636562616336353934626262336363306335636331666633623432343134396334
+37343765646134306638313636326531616665323136643264353666323236343239633639356537
+61343037666232386135303363363938353864396461363830373633363730323165303165633430
+32656532633761316132613463616365353066393165353331646665613239333738653861363134
+30343939333633366532656430303133643337356336336638653266653766353637356561663636
+61626632333438313436323465643434666334393034366166343930323661386664623663396435
+61633334656464623330623333346165363563356363656663373335646433343762663964323735
+34316239663164623431353164306235386661666632383730376139626666313935663462326433
+36313039326635396435646466306366343438656334346231393066326638313337373933393231
+35373731336164396165303935316366343565363631353137393333613363656564343063373933
+61643033373839656532373330666436396632626663353436356231323166326365323832346461
+31366630626635323465333238613734313265356231313338356431393338353166633939383334
+64333634373638383937663933336563663132366534643434373032333934383533383063643036
+39323534326139633931623831656565333930323362366539346239646464353536346639343436
+37663337396565363065343532616437333666316266633632653661396535636461653335363538
+37303263383030646233313239643536326434323933616135373835393965346462373739383230
+33383863353761346232643835633139363231663162353937343363653339393838616535393436
+38303737633231626236316336353832306232633032653032633365386563626434646139633063
+31313436343063643766663465626330643366313663633733663762653631633966383736336138
+36353738383865613537366364666665326464626365643032666630643039326538663531353964
+33646561303435666237303763663634613136633662633536303066363764383039373236666135
+30626639313531353134353730323334623631396463323035313733333965623665333466366164
+62646539393966626337333637646438313635373666306538356631396237323739613032353832
+31383439343531373561353038363535353266333265336434303934343239633334383865333436
+31616537666332373332623932623162333138373563363336303735396431396464613364643162
+61323737643134653763373232383134643931343366353437633632326138353237336333616238
+32613239316666343761363863376563613638366432313461643535353731383631633233366234
+39663632613062656337616561636533633264373930376437373132303830313863303734346632
+33313431336661663634663861643364363834626638666565383635343433323961623765633265
+34386631653833653865383865636364653265363162323462376364313964626633313635313765
+38633131653836396463393136333932346532333631626461373739346139306465653532366632
+65363333633964616632613830386566623865356131646338653236373861346430363164393634
+64373434663531643436616664393936396564666561313862636133306537373137636234626232
+61313861626235636235353431353738303161373939373966383139323833346662373731646434
+31356133623331323661666234613466366130336662653764363738626531346531333339396266
+32636564653732396339383533343266633730666434303161626362366134353361643362626533
+32333132656636333736616661323664316234393937313666653762316463306437633163383030
+65383962346233663061356338306164343363323736646165613730373166333533373635373039
+39343262393031626637623030373535663539613361613366333863336531663637323035376564
+30626232623661623065643136353237316333323764663633643132653264303162366134616564
+31366665313830323263373835613964386132333538396165663538653865356236646234373465
+30383533316532363133323465363063373962353763346366383563613862336465366565653637
+62346365373030383135323933366138656463323864663363626330653036623739343035336135
+62643262336436336462386430363430363139396566343963373664373334366430643830313439
+66613362396137313661666635643962346662376137393066653765636330366462633565346663
+38633864366666316534653665303664393863633464373638393463343537386437383039373530
+66623063653536666364303531373931613433383139376664633934376662336231643762353830
+31653065303430393864383532333766613965383831616563653233313338343861663334373465
+39363361313532333034613034666265386430333466396163303263636133316263313538656634
+65313464316463623134356562366261663635323530393137313233316638643364646639343935
+38313131633132636463323231393332363766653935323965373134653336376131636163376566
+33623934346431313433386430656264316566383436663535343532303763316437303436336465
+39626231303931303130643237383663666338313662663231313935643334393362323130356530
+39656330616138373962633732313436666662616165316236323930613134643161613464636536
+36613735636233616164396630333634333935393938623061313963636461316564313662353034
+33666434326165643964303561303062323061633766353761313464376262333361323532616361
+39313832396134643966353239333066346239626534303830396165396530623135333834313439
+37373661643731623935666437313465633565303232366638393837303462613035393031333061
+39666433373662633730363665313666346165366138306636623263633537346631666132306238
+34393363373335663631393135646231313064383837396164613536643533383464313536663561
+38633131303966356534636539333733336464393236623238373766616537613437343232303437
+64323636613862663135323034636239333433333165313239336239666665616631623934316637
+61303237336334316265643964663830336539353164343063636664633430326139356534313036
+65393464653330333365336335363531303536313439363066376438666133376664346336633937
+66346165666337666134333963613662303534656339373731663534656533316338353763643930
+38313239313637326435323463333863663433623366316532333065396635663966343835636630
+34643631613431333431636330623564663662383063353138656436336333333262306633666163
+38303032316133666533393663343366333162626436333762306263616362643936383637653330
+31643334613766636231333461643863333838343961383762316162636634323134313461313962
+35363062663934336632383931333262643534336563376537343265633434336439623131613337
+32626461643263623664353639613361393735663266373930366134633534343466346461313032
+34363162343464353037616637316636393139343563303331376362653238346631333137393164
+30306636366636366562656361346637356364666136623131316138393262633963356138383232
+64383932623831376632336331343333393739613663613763303035643136653965386435366663
+35313161383236356563653931313933633737646363363339336639613961343934373637643034
+63333036343532323166366132656538643331303131393737656338323939396461393764393134
+33663539633930636265393363326638636130313036333536306362663633616563316465616330
+35353539303335336565373962393433333539613439353765643061383132386133396339616137
+3265326634643062346666356263326639373130333362333064
@@ -45,6 +45,34 @@ localhost ansible_connection=local
 [vpn_egress_gateway]
 k3s-dev-3

+# PROD's egress-gateway node — a SEPARATE group on purpose (2026-09-12).
+#
+# It is NOT a member of [vpn_egress_gateway] above, and k3s-prod-3 must never be added
+# there: that group is the DEFAULT target of playbooks/vpn.egress.gateway.yml, so a
+# flagless `ansible-playbook playbooks/vpn.egress.gateway.yml` would then silently
+# reconfigure a prod node's kill-switch datapath. Prod is reached only by asking for it:
+#
+#   ansible-playbook playbooks/vpn.egress.gateway.yml \
+#     -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3
+#
+# (`--list-hosts` first, every time. The play's target var is
+# `vpn_egress_gateway_hosts`, NOT the cluster-wide `k3s_hosts`.)
+#
+# WHY NODE 3, AND WHY EXACTLY ONE:
+#   * node 1's SSD was replaced 2026-09-11 and it has the least operational history;
+#   * node 2 carries the §0b memattr history;
+#   * node 3 has a single uplink (eno1 192.168.2.12 — its wireless is gone: no wl* netdev
+#     exists, the /etc/network/interfaces wlp1s0 stanza is commented out, and the
+#     192.168.2.195 address the earlier note below claimed is NOT present. Verified
+#     2026-09-12) which is what a kill-switch datapath wants.
+#   * LEXICAL ORDERING HAZARD: Cilium OSS picks "the first node in lexical ordering" when
+#     several match the policy's nodeSelector, so labelling k3s-prod-1 would SILENTLY take
+#     the gateway role from k3s-prod-3. Exactly one host here, forever.
+# Its PIA credentials are its own vaulted group_vars/vpn_egress_gateway_prod/vault.yml —
+# the dev group's vault does not apply to a prod host.
+[vpn_egress_gateway_prod]
+k3s-prod-3
+
 # The Proxmox HYPERVISOR host itself (NOT a k3s node) — target of playbooks/backup-s3.yml,
 # which stands up the off-cluster versitygw S3 backup target for Velero + CNPG. Reached DIRECTLY
 # over SSH as root (its own 192.168.2.102:22, the pfSense NAT to the host's 192.168.1.9), so
@@ -116,8 +144,13 @@ k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
 # needs the same >= 6.13 kernel. Its disk is a THIRD distinct model (WDC WDS500G2B0A SATA
 # SSD) — all three nodes differ on disk, so the §2 Longhorn/etcd symmetry gap is now
 # three-way, not a node-1 outlier.
-# NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately
-# the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2.
+# NOTE (CORRECTED 2026-09-12): this used to read "node 3 also has wlp1s0 up on
+# 192.168.2.195 (wifi)". It does not — re-verified before making node 3 the VPN egress
+# gateway, because a second live uplink in a kill-switch datapath is not a detail: there is
+# NO wl* netdev on the node at all, no phy80211 device, the wlp1s0 stanza in
+# /etc/network/interfaces is commented out, 192.168.2.195 is assigned nowhere, and the only
+# IPv4 addresses are lo and eno1 192.168.2.12. `ansible_host` is that WIRED eno1 address —
+# the same directly-routable 192.168.2.x LAN as nodes 1 and 2.
 k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12

 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
@@ -11,13 +11,25 @@
 #   vault_vpn_egress_gateway_pia_username, vault_vpn_egress_gateway_pia_password, vault_vpn_egress_gateway_pia_ovpn
 # Run:  ansible-playbook playbooks/vpn.egress.gateway.yml
 #
-# TARGET (R-ENV-02): the group is a var, defaulting to dev's `vpn_egress_gateway`. Prod
-# has NOT decided whether it runs an egress gateway at all, nor on which node
-# (infra/PROD-PROVISIONING.md §6, last box) — so there is deliberately no prod egress
-# group in the inventory yet. When there is, point this at it with
-#   -e vpn_egress_gateway_hosts=<that group>
-# and it needs its OWN vaulted PIA credentials group_vars; this play's vault is scoped to
-# the dev group.
+# TARGET (R-ENV-02): the group is a var, defaulting to dev's `vpn_egress_gateway`. PROD has
+# its own group as of 2026-09-12 — `vpn_egress_gateway_prod` (k3s-prod-3), with its own
+# vaulted credentials in group_vars/vpn_egress_gateway_prod/:
+#
+#   ansible-playbook playbooks/vpn.egress.gateway.yml \
+#     -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts
+#
+# The var is `vpn_egress_gateway_hosts`, NOT the cluster-wide `k3s_hosts`. Prod is
+# deliberately NOT a member of the default group, so a flagless run cannot reach it — and
+# `--list-hosts` before every run is the habit that catches a mistyped flag, which in this
+# repo has silently targeted all three dev nodes before.
+#
+# THE LABEL IS WHAT ARMS THE DATAPATH. The CiliumEgressGatewayPolicy is already reconciled
+# on both clusters, so the instant a node carries
+# `homelab.keithjackson.dev/egress-gateway=true` steering goes live. The role sequences
+# kill-switch -> tunnel -> re-steer -> label, with the label LAST, on purpose. Never apply
+# that label by hand, and never run this play with tags that skip the tunnel or kill-switch
+# tasks. Exactly ONE node per cluster may carry it (Cilium OSS picks the lexically first
+# match, so a labelled k3s-prod-1 would silently steal the role from k3s-prod-3).
 - name: Configure the node-level VPN egress gateway
  hosts: "{{ vpn_egress_gateway_hosts | default('vpn_egress_gateway') }}"
  become: true
@@ -12,12 +12,19 @@ separate GitOps authored against the shared **interface contract** below.
 > once the tunnel is live in dev/test.

 ## Interface contract (shared with the Cilium-policy agent)
- **Inventory group** `vpn_egress_gateway` — the egress-gateway node(s) (initially `k3s-dev-3`).
+- **Inventory groups** — `vpn_egress_gateway` (dev: `k3s-dev-3`) and `vpn_egress_gateway_prod`
+  (prod: `k3s-prod-3`). Each holds **exactly one** host, and prod is a separate group so the
+  flagless playbook run cannot reach it. Cilium OSS resolves a multi-match nodeSelector to "the
+  first node in lexical ordering", so a second labelled node does not add failover — it silently
+  steals the role.
 - **k8s node label** `homelab.keithjackson.dev/egress-gateway=true` — set by this role; the
-  `CiliumEgressGatewayPolicy` selects the gateway node by it.
+  `CiliumEgressGatewayPolicy` selects the gateway node by it. **This label is what arms the
+  datapath**, which is why the role applies it LAST.
 - **Host tunnel iface** `tun0` — the Cilium policy SNATs vpn-class pods out of it.
- **Optional fwmark** `0x1c8` — a contract *hook* (see kill-switch layering) the Cilium side
-  MAY stamp on vpn-class packets for a tunnel-down-independent host backstop.
+- **k8s node label** `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` — set by this
+  role's re-steer script. Its VALUE is the uplink Cilium was last told about; its CHANGE is what
+  makes Cilium re-resolve `interface: tun0`. Cilium caches that resolution and never revisits it
+  on its own, so without this the gateway is dead after its first reconnect. See "Re-steering".

 ## How the four ADR-0015 invariants are met

@@ -45,19 +52,42 @@ Two independent layers, both scoped so they can only ever affect vpn-class traff
 2. **nftables backstop (`templates/vpn-egress-killswitch.nft.j2`).** Own `inet
   vpn_egress_gateway_killswitch` table, base chain **policy `accept`** (so it can only *drop* the
   narrow matches, never break anything). In `postrouting`:
-   - `oifname tun0 ip saddr @vpn_egress_gateway_src accept` — tunnel up: let steered traffic out.
-   - `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` — **fail closed:** a tunnel-sourced packet
-     trying to leave any other interface is dropped (no WAN leak). This rule stays **armed even
-     while the tunnel is down** because the source set is retained on tunnel-down (see below).
-   - `meta mark 0x1c8 oifname != tun0 drop` — the optional fwmark backstop.
+   - `oifname tun0 ip saddr @vpn_egress_gateway_src counter accept` — tunnel up: let
+     tunnel-sourced traffic out.
+   - `ip saddr @vpn_egress_gateway_src oifname != tun0 counter drop` — **fail closed:** a
+     tunnel-sourced packet trying to leave any other interface is dropped (no WAN leak). Armed
+     even while the tunnel is down, because the source set is retained on tunnel-down (below).

-`@vpn_egress_gateway_src` holds the **last live tun0 address**. The up-script **flushes then
-re-adds** the current tunnel source on every connect; the down-script **intentionally retains
-it** (see "Explicit fail-closed backstop" below). Node traffic and `internet`-class traffic are
-never in the set, never carry the mark, and are never targeted — so they always hit
+   **Both rules carry a `counter`, and both were exercised on k3s-prod-3 (2026-09-12):** a
+   source-bound `ping -I 10.72.192.7` with the ip-rule present got 3/3 replies at 144 ms and
+   moved `killswitch_accept`; with the ip-rule deleted, `ip route get` confirmed the packet
+   would go `via 192.168.2.1 dev eno1` and `killswitch_drop` moved **0 -> 3 packets / 252
+   bytes** with 100% loss. The kill-switch is real, not aspirational — but read the next
+   paragraph for what it can and cannot reach.
+
+   **SCOPE, HONESTLY — it does NOT cover vpn-class POD traffic.** A counters-only diagnostic
+   table on the dev gateway recorded, while a vpn-class pod egressed: 0 packets in `forward`
+   and 0 in `postrouting` for the pod CIDR *and* for the PIA range, against 45166 packets on
+   that same postrouting hook for WAN traffic and 16 for a node-originated curl out tun0.
+   Cilium's eBPF egress-gateway datapath (`bpf.masquerade: true`) `bpf_redirect()`s those skbs
+   straight to the egress device, so netfilter POSTROUTING is never invoked for them. What
+   makes vpn-class fail closed is **Cilium's own drop** when the cached egress interface is
+   unusable — tcpdump during a tunnel stop and during a tunnel restart showed 0 packets leaving
+   either the WAN or tun0. This table's real, narrow guarantee is NODE-originated traffic
+   sourced from a tunnel address.
+
+`@vpn_egress_gateway_src` holds **every tun0 address this node has had**. The up-script **adds**
+the current tunnel source on every connect and never flushes (it used to flush first, which left
+a brief moment where a packet still sourced from the previous address matched neither the
+ip-rule nor the drop rule); the down-script **intentionally retains it**. Node traffic and
+`internet`-class traffic are never in the set and are never targeted — so they always hit
 `policy accept` and **fail open toward the WAN** (invariant 4), tunnel up or down. The opposite
 directions are structural, not incidental.

+A kill-switch **reload no longer disarms the guard**: the `.nft` file uses `table` + `flush
+table`, which clears rules but leaves a named set's elements intact, so an Ansible run cannot
+empty the set mid-flight.
+
 ### Explicit, edge-independent fail-closed backstop (tunnel-down)
 Earlier the down-script flushed `@vpn_egress_gateway_src`, which **disarmed** the
 `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` rule at exactly the moment the tunnel was
@@ -67,12 +97,15 @@ guarantee that could leak if the edge router NATed that stale source. The down-s
 vpn-class packet that would leave the WAN while the tunnel is down is **explicitly dropped** in
 the host nft table — defense-in-depth, independent of edge-router behavior.

-This is safe precisely because the **up-script flushes the set and re-adds the (possibly new)
-tunnel source on every reconnect** (`nft flush set …` then `nft add element …`), so a retained
-stale address self-corrects the instant the tunnel comes back. The set remains scoped to
-vpn-class only; node and internet-class traffic are never members, so **no invariant is
-weakened — only vpn-class is ever affected** (invariants 3 & 4 upheld). The routing blackhole in
-table `51820` remains as the second, independent fail-closed layer.
+The set is now **ADDITIVE** (2026-09-12): the up-script only ever `nft add element`s the live
+source. It used to `nft flush set` first, and that flush was itself a hole — between a reconnect
+and Cilium refreshing its cached egress IP, a packet still sourced from the PREVIOUS address
+matched neither the ip-rule (deleted by the down-script) nor the drop rule (the address had just
+been flushed out of the set). Additive closes it, `add element` is idempotent, and an extra
+element can only ever *tighten* the rule. The set stays scoped to tunnel sources; node and
+internet-class traffic are never members, so **no invariant is weakened** (invariants 3 & 4
+upheld). The routing blackhole in table `51820` remains as the second, independent fail-closed
+layer.

 ## Why the node default route is provably unaffected (design-level verification)
 - OpenVPN runs with `route-noexec` and `pull-filter ignore "redirect-gateway"` → it adds **zero**
@@ -82,40 +115,79 @@ table `51820` remains as the second, independent fail-closed layer.
  ip-rules, never `table main` / `table default`.
 - The ip-rule matches **`from <tun0-addr>`** — an address the node itself never sources traffic
  from — so node-originated and internet-class packets never enter the tunnel table.
- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src` / the fwmark.
+- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src`.
 - Live check (post-deploy): `ip route show table main default` still points at the WAN NIC, and
  `curl ipinfo` from the node shows the WAN IP, with the tunnel both up and forced down.
+  Confirmed on `k3s-prod-3` 2026-09-12: `default via 192.168.2.1 dev eno1` with the tunnel up,
+  node curl = the WAN IP, k3s `/readyz` = ok.

-## Kill-switch layering & ESCALATION (honest limit)
- The nft `@vpn_egress_gateway_src` rule and the routing blackhole robustly cover the case where the
-  tunnel is up but a vpn-class packet is (mis)routed toward the WAN, and flaps.
- The **tunnel fully-down** case relies on **Cilium Egress Gateway itself failing closed** when
-  its egress interface `tun0` is absent (Cilium 1.16 drops egress-gateway-marked traffic with no
-  valid egress path rather than falling back to the node's default route) **plus** the routing
-  blackhole. That is the ADR's expected behavior.
- For a host backstop that is **fully independent of Cilium even in a fail-open regression** when
-  the tunnel is down (no tun0 address to key on), the host needs a stable marker on vpn-class
-  packets. The only contract-available marker is the **`0x1c8` fwmark hook** — which the
-  *Cilium-policy* side must stamp. **ESCALATION for the PM:** either (a) confirm the Cilium
-  Egress Gateway 1.16 fail-closed + routing-blackhole guarantee is accepted as sufficient for
-  R-NET-06, or (b) add the shared fwmark to the interface contract so the CiliumEgressGatewayPolicy
-  marks vpn-class traffic and this role's fwmark rule becomes the tunnel-down-independent backstop.
-  This is flagged rather than worked around — no spec was weakened to reach done.
+## Re-steering — Cilium does not follow tun0, and that is the whole ball game
+
+Cilium resolves the policy's `interface: tun0` **once**, caching an (egress IP, egress ifindex)
+pair in its BPF egress map, and does not re-resolve when tun0 changes. Upstream says so: *"it
+does not automatically respond to a change in the gateway node's network configuration... You can
+force a fresh selection by re-applying the Egress Gateway policy."* Measured on `k3s-dev-3`:
+
+| what | observed |
+|---|---|
+| after `systemctl restart vpn-egress-tunnel` | node tun0 at ifindex 1987, `cilium-dbg bpf egress list` still 1986 |
+| vpn-class pod egress in that state | times out; tcpdump shows **0** packets out the WAN and **0** out tun0 (no leak, no service) |
+| `egress-gateway-reconciliation-trigger-interval: 1s` | does **not** help — dev has that setting; still stale 20s+ later. It coalesces reconciliations that k8s object events already triggered; netlink link events are not among them |
+| stale egress **IP** with a live ifindex | does NOT leak either — the redirect follows the cached *ifindex*, so packets leave tun0 with the previous source (0 packets on the WAN, verified) |
+| touching any node label on the gateway | cached ifindex 1986 -> 1987 within 10s, egress recovers immediately |
+
+So the failure mode is a **silent outage on every tunnel restart or reboot**, not a leak. The fix
+is `templates/vpn-egress-resteer.sh.j2`, which publishes `<tun0 addr>-<ifindex>` as the
+`egress-uplink` node label:
+
+- **a node label, not an annotation on the policy** — the policy is Flux-managed GitOps and
+  writing to it from a node invites drift/ownership churn; node labels are already this role's
+  territory, and the label value doubles as the diagnostic (`kubectl get nodes -L
+  homelab.keithjackson.dev/egress-uplink`);
+- **fired by the OpenVPN up-script** on every connect, via `systemctl start --no-block` so it can
+  never block or fail the tunnel and does not need the kube-API to be up at that instant;
+- **plus `vpn-egress-resteer.timer`** (2 min) as the convergence backstop for the boot case where
+  the API is not up yet and every in-script retry is exhausted;
+- **idempotent** — if the label already records the live uplink it exits without writing, so the
+  timer is free to run often.
+
+Proof it works: `systemctl stop` then `start` of the tunnel now recovers on its own (map
+1989 -> 1990, PIA IP returns). Before the fix the identical sequence stayed broken indefinitely —
+which also means the **recovery half of the `vpn-killswitch` conformance test could not have
+passed** on the pre-fix implementation.

 ## What requires a LIVE tunnel to validate (cannot be tested without operator creds)
 1. The PIA `.ovpn` + username/password actually authenticate and `tun0` comes up as a host iface.
 2. `egress-class`: a vpn-class pod's public IP is the PIA Stockholm IP; internet-class shows the WAN IP.
 3. `vpn-killswitch`: forcing `tun0`/the tunnel service down blackholes vpn-class egress (no leak)
   while internet-class is unaffected — the R-NET-06 fault-injection gate.
-4. That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped
-   rules actually see the packets (eBPF datapath interaction — see escalation).
+4. ~~That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped
+   rules actually see the packets (eBPF datapath interaction).~~ **ANSWERED 2026-09-12: it does
+   NOT.** Counters proved the steered pod packets never enter netfilter at all. See "SCOPE,
+   HONESTLY" above. The question is closed; the guard's claim is narrowed rather than pretended.
 5. MTU/PMTU across `pod → tun0 → PIA` (in-pod gluetun needed 1300 for Cilium VXLAN; on the node
   the tunnel rides the physical WAN, so `vpn_egress_gateway_tun_mtu` defaults empty — verify live).
 6. Port forwarding (PIA NAT-PMP) for qBittorrent — ADR-0015 Phase-6 follow-up, not in this role.

 ## Files
 - `defaults/main.yml` — all `vpn_egress_gateway_*` vars; the three vault-backed credential vars.
- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, node label.
+- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, re-steer, node label
+  (kill-switch -> tunnel -> re-steer -> **label last**, because the label is what arms steering).
 - `handlers/main.yml` — daemon-reload, kill-switch reload, tunnel restart.
- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, both systemd units.
- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys: `inventory/group_vars/vpn_egress_gateway/README.md`.
+- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, the kill-switch and
+  tunnel units, and `vpn-egress-resteer.{sh,service,timer}`.
+- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys:
+  `inventory/group_vars/vpn_egress_gateway/README.md` (dev) and
+  `inventory/group_vars/vpn_egress_gateway_prod/README.md` (prod).
+
+## Running it against prod
+```bash
+cd infra/ansible
+ansible-playbook playbooks/vpn.egress.gateway.yml \
+  -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts
+# confirm it says exactly `k3s-prod-3`, then drop --list-hosts
+```
+The target var is `vpn_egress_gateway_hosts`, **not** the cluster-wide `k3s_hosts`. Never apply
+the `egress-gateway` label by hand, and never run this role with tags that skip the tunnel or
+kill-switch tasks — the label is the last step precisely so nothing is steered at a node that is
+not ready to carry it.
@@ -20,6 +20,23 @@ vpn_egress_gateway_pia_password: "{{ vault_vpn_egress_gateway_pia_password }}"
 # The FULL text of the PIA .ovpn (inline <ca>/<crl-verify>, cipher/auth, remote line).
 vpn_egress_gateway_pia_ovpn: "{{ vault_vpn_egress_gateway_pia_ovpn }}"

+# OVERRIDE THE .ovpn's PINNED `remote` — "<host-or-ip> <port>", empty to honour the .ovpn.
+#
+# WHY THIS EXISTS (measured 2026-09-12): the operator's .ovpn pins ONE PIA server IP. Point
+# two gateways at the same vaulted .ovpn and both dial the SAME server, and PIA hands BOTH
+# THE SAME tunnel address — dev and prod were both `Server-10779-2a`
+# (158.173.241.112:1198) with `ifconfig 10.72.192.7`. Each reconnect then steals that
+# address from the other, the loser sees no data for 60s, hits `--ping-restart`, reconnects
+# and steals it back: both tunnels flapped on a ~2-minute cycle and neither was reliably
+# up. It is NOT a connection-LIMIT problem (the account has room) — it is one pinned server.
+#
+# So each environment's gateway needs its own server. The PIA Stockholm LB pool
+# (`sweden.privacy.network`) resolved to 158.173.241.{34,55,112,135,239}; any PIA server in
+# the pool accepts the same credentials and CA, so this is a per-environment knob rather
+# than a second vaulted .ovpn. NOT a secret — a public server address — so it belongs in
+# group_vars/<group>/vars.yml, not the vault.
+vpn_egress_gateway_pia_remote: ""
+
 # --- on-disk layout -------------------------------------------------------------
 vpn_egress_gateway_config_dir: /etc/openvpn/client
 vpn_egress_gateway_scripts_dir: /etc/openvpn/scripts
@@ -46,17 +63,39 @@ vpn_egress_gateway_blackhole_metric: 4000
 # (Cilium VXLAN 1450 → needed 1300) no shrink is required by default.
 vpn_egress_gateway_tun_mtu: ""

-# --- nftables kill-switch (invariants 3 & 4 — scoped to vpn-class ONLY) ----------
+# --- nftables kill-switch -------------------------------------------------------
+# SCOPE, HONESTLY (measured 2026-09-12, see the .nft template header): this table guards
+# NODE-ORIGINATED traffic sourced from a tunnel address. It does NOT and cannot guard the
+# egress-class=vpn POD traffic, because Cilium's eBPF egress-gateway datapath
+# (bpf.masquerade) redirects those packets to the egress device without ever invoking
+# netfilter POSTROUTING. vpn-class fails closed via Cilium's own drop, not via this table.
 vpn_egress_gateway_nft_table: vpn_egress_gateway_killswitch
-# Named set holding the LIVE tun0 source address (maintained by the up/down scripts).
-# The kill-switch matches ONLY packets SNAT'd to this address by Cilium Egress Gateway
-# — i.e. exactly the egress-class=vpn traffic. Node / internet-class traffic is never
-# in this set and is never matched (fails OPEN toward the WAN).
+# Named set of tunnel source addresses, ADDITIVE (the up-script adds, never flushes; the
+# down-script retains). Every element only tightens the `oifname != tun0 drop` rule.
 vpn_egress_gateway_nft_src_set: vpn_egress_gateway_src
-# Contract-hook fwmark: if the CiliumEgressGatewayPolicy side stamps vpn-class packets
-# with this mark, the kill-switch also drops marked packets not leaving via tun0 — an
-# extra, tunnel-down-independent backstop. See README "Kill-switch layering / escalation".
-vpn_egress_gateway_fwmark: "0x1c8"
+# REMOVED 2026-09-12: vpn_egress_gateway_fwmark ("0x1c8"). It backed a kill-switch rule
+# for a mark that nothing stamps — Cilium OSS does not mark egress-gateway traffic with
+# it (the only Cilium mark on the gateway node is 0x200/0xf00, the L7-proxy rule), and the
+# traffic never reaches netfilter anyway. A dead rule matching an arbitrary mark is a
+# hazard, not a backstop, so both the rule and the variable are gone.
+
+# --- re-steer: keep Cilium's cached egress interface in sync with tun0 -----------
+# Cilium resolves `interface: tun0` ONCE into a cached (egress IP, egress ifindex) and does
+# not re-resolve when tun0 changes; every tunnel restart gives tun0 a NEW ifindex, which
+# silently blackholes vpn-class egress. The re-steer script publishes the live uplink as a
+# node label; that node update makes the gateway node re-process the policy. Verified on
+# k3s-dev-3: cached ifindex moved 1986 -> 1987 within 10s and egress recovered.
+# The label VALUE is "<tun0 address>-<ifindex>", so `kubectl get nodes -L <this key>` shows
+# which uplink Cilium was last told about.
+vpn_egress_gateway_uplink_label_key: homelab.keithjackson.dev/egress-uplink
+# The kube-API is routinely not reachable yet when OpenVPN connects during boot, so the
+# script retries. Bounded so a genuinely broken API does not hang a systemd job forever.
+vpn_egress_gateway_resteer_retries: 30
+vpn_egress_gateway_resteer_retry_seconds: 10
+# Convergence backstop if the event-driven trigger is ever lost (no-op when already in
+# sync, so it is cheap to run often).
+vpn_egress_gateway_resteer_timer_boot: 2min
+vpn_egress_gateway_resteer_timer_interval: 2min

 # --- kubernetes node label (steering contract) ----------------------------------
 # The label the parallel CiliumEgressGatewayPolicy selects the gateway node by.
@@ -28,9 +28,18 @@
    - "{{ vpn_egress_gateway_config_dir }}"
    - "{{ vpn_egress_gateway_scripts_dir }}"

+# `vpn_egress_gateway_pia_remote`, when set, rewrites ONLY the `remote <host> <port>` line so
+# this environment's gateway dials its own PIA server. Two gateways sharing the .ovpn's single
+# pinned server are handed the SAME tunnel address and evict each other every ~2 minutes — see
+# the variable's comment in defaults/main.yml. The regex is anchored to a line that STARTS with
+# `remote` followed by whitespace, so `remote-cert-tls` / `remote-random` are untouched.
 - name: Deploy the operator's proven PIA .ovpn (from Ansible vault)
  ansible.builtin.copy:
-    content: "{{ vpn_egress_gateway_pia_ovpn }}"
+    content: >-
+      {{ vpn_egress_gateway_pia_ovpn if (vpn_egress_gateway_pia_remote | length == 0)
+         else (vpn_egress_gateway_pia_ovpn
+               | regex_replace('(?m)^[ \t]*remote[ \t]+\S+.*$',
+                               'remote ' ~ vpn_egress_gateway_pia_remote)) }}
    dest: "{{ vpn_egress_gateway_ovpn_path }}"
    owner: root
    group: root
@@ -69,6 +78,17 @@
    - vpn-egress-down.sh
  notify: Restart vpn-egress tunnel

+# The re-steer script is what keeps Cilium's cached egress interface in sync with tun0.
+# It is NOT notified to the tunnel: it is triggered BY the tunnel's up-script and by its
+# own timer, and re-deploying it must never bounce the tunnel.
+- name: Deploy the Cilium egress-gateway re-steer script
+  ansible.builtin.template:
+    src: vpn-egress-resteer.sh.j2
+    dest: "{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh"
+    owner: root
+    group: root
+    mode: "0750"
+
 - name: Deploy the scoped nftables kill-switch ruleset
  ansible.builtin.template:
    src: vpn-egress-killswitch.nft.j2
@@ -100,6 +120,18 @@
    - Reload systemd
    - Restart vpn-egress tunnel

+- name: Deploy the egress-gateway re-steer unit and its convergence timer
+  ansible.builtin.template:
+    src: "{{ item }}.j2"
+    dest: "/etc/systemd/system/{{ item }}"
+    owner: root
+    group: root
+    mode: "0644"
+  loop:
+    - vpn-egress-resteer.service
+    - vpn-egress-resteer.timer
+  notify: Reload systemd
+
 - name: Flush handlers so units/routing exist before enable+start
  ansible.builtin.meta: flush_handlers

@@ -115,6 +147,27 @@
    enabled: true
    state: started

+# Enabled BEFORE the node label below: the label is what arms steering, and by then the
+# re-steer path must already exist so Cilium is told about the live tun0 rather than
+# caching whatever it resolves once and never revisiting it.
+- name: Enable and start the re-steer convergence timer
+  ansible.builtin.systemd_service:
+    name: vpn-egress-resteer.timer
+    enabled: true
+    state: started
+
+# Publish the live uplink immediately rather than waiting for the timer's first tick, so a
+# first-time stand-up (or a re-run after a tunnel restart) converges within seconds.
+# `started`, NOT `restarted`: the tunnel's up-script fires this same unit on connect, and
+# `restarted` SIGTERMs that in-flight run (logging a spurious unit failure) instead of
+# letting it finish. The unit is a oneshot without RemainAfterExit, so it is `inactive`
+# once it has run and `started` genuinely re-runs it.
+- name: Re-steer Cilium onto the live tun0 now
+  ansible.builtin.systemd_service:
+    name: vpn-egress-resteer.service
+    state: started
+  changed_when: false
+
 # --- steering contract: label the k8s node the Cilium Egress Gateway policy selects ----
 - name: Read the current egress-gateway node label
  ansible.builtin.command:
@@ -8,15 +8,20 @@
 # then (2) remove the tunnel default from the dedicated table. The permanent
 # `blackhole default` in that table is LEFT in place as one fail-closed fallback.
 #
-# EXPLICIT FAIL-CLOSED BACKSTOP: the kill-switch source set is INTENTIONALLY RETAINED
-# here (we do NOT flush it). Keeping the last live tun0 address in
+# FAIL-CLOSED BACKSTOP: the kill-switch source set is INTENTIONALLY RETAINED here (we do
+# NOT flush it). Keeping every tunnel address this node has had in
 # `{{ vpn_egress_gateway_nft_src_set }}` keeps the nft rule
 # `ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != tun0 drop` ARMED while the
-# tunnel is down, so any vpn-class packet that would leave the WAN is EXPLICITLY dropped
-# — defense-in-depth that does not depend on the stale SNAT source being non-routable or
-# on the edge router refusing to NAT it. This is safe because the up-script FLUSHES then
-# re-adds the (possibly new) tunnel source on every reconnect, so the set self-corrects.
-# The set is scoped to vpn-class only; node and internet-class traffic are never in it.
+# tunnel is down. The up-script is ADDITIVE (2026-09-12 — it used to flush first, which
+# briefly left a previous tunnel address matched by nothing), so the set only ever grows
+# and only ever tightens the rule.
+#
+# SCOPE, HONESTLY: that rule covers NODE-ORIGINATED traffic sourced from a tunnel address.
+# It does NOT cover the egress-class=vpn POD traffic — Cilium's eBPF egress-gateway
+# datapath never presents those packets to netfilter (measured; see the kill-switch .nft
+# header). vpn-class pods fail closed because Cilium drops them when it cannot reach the
+# cached egress interface — verified with tcpdump on both the WAN and tun0 during a tunnel
+# stop and a tunnel restart: zero packets left either device.
 # The node's main-table WAN default is never touched here.
 set -uo pipefail

@@ -30,8 +35,8 @@ SRC="${ifconfig_local:-}"

 # 1) Stop steering to the (now dead) tunnel source by dropping the ip-rule.
 #    NOTE: the nft source set is deliberately NOT flushed here — see header. Retaining the
-#    last tun0 address keeps the `oifname != tun0 drop` rule armed as the explicit,
-#    edge-independent fail-closed backstop until the up-script re-registers on reconnect.
+#    tun0 addresses keeps the `oifname != tun0 drop` rule armed for node-originated
+#    tunnel-sourced traffic until the up-script re-registers on reconnect.
 if [ -n "${SRC}" ]; then
  while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done
 fi
@@ -1,40 +1,93 @@
 #!/usr/sbin/nft -f
 # {{ ansible_managed }}
-# ADR-0015 kill-switch (invariants 3 & 4) — SCOPED TO vpn-class ONLY, fails CLOSED.
+# ADR-0015 kill-switch — SCOPED TO tunnel-sourced traffic ONLY, fails CLOSED.
+#
+# =============================================================================
+# READ THIS BEFORE TRUSTING THIS FILE (measured 2026-09-12 on k3s-dev-3)
+# =============================================================================
+# This table does NOT, and CANNOT, guard the `egress-class=vpn` POD traffic that
+# ADR-0015 asks it to guard. It was measured, not reasoned about:
+#
+#   A diagnostic nft table (counters only) was installed on the gateway node
+#   alongside this one, counting every packet in `forward` and `postrouting` by
+#   (oifname, saddr). While a vpn-class pod egressed through the gateway:
+#       forward  saddr=pod CIDR ................  0 packets
+#       forward  saddr=PIA tunnel range ........  0 packets
+#       postrouting oif tun0 saddr=tunnel ......  0 packets
+#       postrouting oif WAN  saddr=pod CIDR ....  0 packets
+#       postrouting oif WAN  (any) ............. 45166 packets   <- hook IS live
+#   and, as a positive control, a curl from the NODE itself out tun0 counted
+#       postrouting oif tun0 saddr=tunnel ......  16 packets
+#
+#   So the hook is traversed and the matching works — but Cilium's egress-gateway
+#   datapath never presents the steered pod packets to netfilter at all. With
+#   `bpf.masquerade: true` the SNAT and the egress happen in eBPF, which
+#   bpf_redirect()s the skb straight to the egress device: POSTROUTING is not
+#   invoked for those packets, in either direction.
+#
+# WHAT ACTUALLY MAKES vpn-class FAIL CLOSED (also measured):
+#   Cilium caches the gateway's (egress IP, egress ifindex) when it resolves this
+#   policy. If the cached ifindex no longer exists — which is exactly what a tunnel
+#   restart produces, tun0 being re-created with a NEW ifindex — the eBPF redirect
+#   fails and the packet is DROPPED. Verified with tcpdump on BOTH the WAN and tun0
+#   during a tunnel restart and during a full stop: 0 packets left either device.
+#   There is no leak, but the mechanism is Cilium's, not this table's.
+#
+# WHAT THIS TABLE IS STILL GOOD FOR:
+#   NODE-ORIGINATED traffic sourced from a tunnel address (the 16-packet control
+#   above) — e.g. a host-side helper bound to tun0 — which DOES traverse netfilter.
+#   That is a real, if narrow, guarantee, and it is the only one claimed here.
+#   Every rule carries a `counter` so the claim is falsifiable at any moment:
+#     nft list table inet {{ vpn_egress_gateway_nft_table }}
+#   If `killswitch_drop` ever becomes non-zero, something (a Cilium datapath
+#   change, bpf.masquerade turned off) has put this traffic back into netfilter —
+#   read that as new information, not as reassurance.
+#
+# See platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-12.
+# =============================================================================
 #
 # Own dedicated table so re-loading it never disturbs Cilium's or the node's own rules.
-# The base chains are policy ACCEPT: this table can ONLY ever drop the narrow vpn-class
-# traffic matched below; node traffic and internet-class traffic are untouched and keep
-# egressing the WAN (fail OPEN — invariant 4). It matches on:
-#   * @{{ vpn_egress_gateway_nft_src_set }} — the LIVE tun0 address Cilium Egress Gateway SNATs
-#     vpn-class pods to (populated by the OpenVPN up-script; emptied by the down-script).
-#     This IS the egress-class=vpn traffic and nothing else.
-#   * meta mark {{ vpn_egress_gateway_fwmark }} — optional contract-hook: a tunnel-DOWN-independent
-#     backstop, active only if the CiliumEgressGatewayPolicy side stamps this fwmark.
+# The base chain is policy ACCEPT: this table can ONLY ever drop traffic whose source is
+# in the set below; node traffic and internet-class traffic are untouched and keep
+# egressing the WAN (fail OPEN — invariant 4).
+
+# Idempotent reload. `table` + `flush table` clears the RULES only — a named set and its
+# elements are not chains, so the live tunnel address SURVIVES a reload and the guard is
+# not disarmed by an Ansible run. (A plain `nft -f` without this would append a second
+# copy of every rule on each load.)
+table inet {{ vpn_egress_gateway_nft_table }} { }
+flush table inet {{ vpn_egress_gateway_nft_table }}

 table inet {{ vpn_egress_gateway_nft_table }} {
-    # Live tunnel source address(es). Empty whenever the tunnel is down => the two
-    # saddr rules below match nothing => vpn-class has no path at all (fail closed).
+    # Every tunnel source address this gateway has had. The OpenVPN up-script ADDS the
+    # live address on each connect and never flushes (see vpn-egress-up.sh): an element
+    # can only ever tighten the guard, and retaining previous addresses means a packet
+    # still sourced from an older tunnel address is dropped rather than unmatched.
+    # Empty before the very first successful connect => the rules below match nothing,
+    # and vpn-class has no path at all (fail closed, by Cilium's drop — see above).
    set {{ vpn_egress_gateway_nft_src_set }} {
        type ipv4_addr
    }

    chain postrouting {
-        # oifname is resolved at postrouting; SNAT/masquerade has already applied here,
-        # so the Cilium-SNAT'd vpn-class packet's saddr is the tunnel address.
+        # oifname is resolved at postrouting, after any netfilter SNAT.
        type filter hook postrouting priority 100; policy accept;

-        # Tunnel UP: let the steered vpn-class traffic out tun0.
-        oifname "{{ vpn_egress_gateway_tun_iface }}" ip saddr @{{ vpn_egress_gateway_nft_src_set }} accept
+        # Tunnel UP: let tunnel-sourced traffic out tun0.
+        oifname "{{ vpn_egress_gateway_tun_iface }}" ip saddr @{{ vpn_egress_gateway_nft_src_set }} counter accept comment "killswitch_accept"

-        # FAIL CLOSED: a vpn-class (tunnel-sourced) packet trying to leave ANY other
-        # interface (i.e. the WAN) is dropped — never leaks. Scoped by the set, so it
-        # can never match node or internet-class traffic.
-        ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != "{{ vpn_egress_gateway_tun_iface }}" drop
+        # FAIL CLOSED: tunnel-sourced traffic trying to leave ANY other interface (i.e.
+        # the WAN) is dropped — never leaks. Scoped by the set, so it can never match
+        # node or internet-class traffic. The counter is the proof-of-life for the whole
+        # table: see the header for why it reads 0 for POD traffic by construction.
+        ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != "{{ vpn_egress_gateway_tun_iface }}" counter drop comment "killswitch_drop"

-        # Tunnel-down-independent backstop (contract hook, see role README): if vpn-class
-        # packets carry the agreed fwmark, drop any not leaving via tun0 even when the set
-        # is empty (tunnel fully down). Harmless if the Cilium side does not set the mark.
-        meta mark {{ vpn_egress_gateway_fwmark }} oifname != "{{ vpn_egress_gateway_tun_iface }}" drop
+        # REMOVED 2026-09-12 — the `meta mark 0x1c8 oifname != tun0 drop` rule. It was a
+        # "contract hook" for a fwmark that nothing stamps: Cilium OSS does not mark
+        # egress-gateway traffic with 0x1c8 (the only Cilium mark visible on the gateway
+        # node is 0x200/0xf00, which is the L7-proxy rule -> table 2004, unrelated), and
+        # the traffic does not reach netfilter anyway. A dead rule matching an arbitrary
+        # mark is a hazard, not a backstop: any future component that happens to use
+        # 0x1c8 would have had its egress silently dropped.
    }
 }
@@ -0,0 +1,25 @@
+# {{ ansible_managed }}
+# Publishes this node's CURRENT tun0 (address + ifindex) as a node label so Cilium
+# re-resolves the CiliumEgressGatewayPolicy's `interface: tun0` (ADR-0015). Without this,
+# Cilium keeps a cached (egress IP, egress ifindex) from the FIRST resolution and vpn-class
+# egress is silently blackholed after every tunnel restart or reboot — measured, see the
+# script header and the DECISIONS-LOG entry of 2026-09-12.
+#
+# Triggered by the OpenVPN up-script (`systemctl start --no-block`) on every connect, and
+# by vpn-egress-resteer.timer as the convergence backstop. Deliberately NOT ordered
+# Before/After the tunnel: it must never be able to block or fail the tunnel.
+[Unit]
+Description=Re-steer Cilium Egress Gateway onto the live tun0 — ADR-0015
+Documentation=file://{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh
+After=network-online.target
+# The kill-switch owns the nft table this script re-registers the tunnel source into.
+After=vpn-egress-killswitch.service
+
+[Service]
+Type=oneshot
+ExecStart={{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh
+# Bounded: {{ vpn_egress_gateway_resteer_retries }} attempts x {{ vpn_egress_gateway_resteer_retry_seconds }}s waiting for the kube-API, plus slack.
+TimeoutStartSec={{ (vpn_egress_gateway_resteer_retries | int * vpn_egress_gateway_resteer_retry_seconds | int) + 60 }}
+# A failure here means "the kube-API was unreachable"; the timer retries. Do not let a
+# failed re-steer mark the whole gateway degraded.
+SuccessExitStatus=0 1
@@ -0,0 +1,97 @@
+#!/usr/bin/env bash
+# {{ ansible_managed }}
+# Re-steer the Cilium Egress Gateway onto THIS node's CURRENT tun0 (ADR-0015).
+#
+# WHY THIS EXISTS (measured on k3s-dev-3, 2026-09-12 — see DECISIONS-LOG):
+#   Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0` once, caching an
+#   (egress IP, egress ifindex) pair in its BPF egress map, and does NOT re-resolve when
+#   tun0 changes. Observed directly: after `systemctl restart vpn-egress-tunnel` the node
+#   had tun0 at ifindex 1987 while `cilium-dbg bpf egress list` still said 1986, and
+#   vpn-class pod egress was blackholed (tcpdump: 0 packets out the WAN, 0 out tun0 — no
+#   leak, but no service). `egress-gateway-reconciliation-trigger-interval: 1s` does NOT
+#   cover netlink link events; it only coalesces reconciliations that k8s object events
+#   already triggered.
+#
+#   Upstream remedy: "You can force a fresh selection by re-applying the Egress Gateway
+#   policy." Any k8s event that makes the gateway node re-process the policy does it. We
+#   use a NODE LABEL rather than touching the policy object, because:
+#     * the policy is Flux-managed GitOps — writing to it from a node invites drift
+#       fights and ownership churn; the node's labels are already this role's territory;
+#     * it was verified to work: setting an unrelated label on k3s-dev-3 moved the cached
+#       ifindex 1986 -> 1987 within 10s and vpn-class egress recovered (PIA IP returned);
+#     * the label VALUE is the uplink identity, so `kubectl get node -L ...` shows at a
+#       glance which tun0 Cilium was last told about. Diagnosis needs no exec into a pod.
+#
+# Idempotent: if the label already records the live uplink, this exits without touching
+# anything, so the timer can run as often as it likes.
+set -uo pipefail
+
+IFACE="{{ vpn_egress_gateway_tun_iface }}"
+TABLE_ID="{{ vpn_egress_gateway_route_table_id }}"
+RULE_PRIO="{{ vpn_egress_gateway_rule_priority }}"
+TUN_METRIC="{{ vpn_egress_gateway_tun_route_metric }}"
+NFT_TABLE="{{ vpn_egress_gateway_nft_table }}"
+NFT_SET="{{ vpn_egress_gateway_nft_src_set }}"
+NODE="{{ inventory_hostname }}"
+UPLINK_LABEL="{{ vpn_egress_gateway_uplink_label_key }}"
+RETRIES="{{ vpn_egress_gateway_resteer_retries }}"
+RETRY_SLEEP="{{ vpn_egress_gateway_resteer_retry_seconds }}"
+export KUBECONFIG="{{ vpn_egress_gateway_kubeconfig }}"
+
+log() { echo "vpn-egress-resteer: $*"; }
+
+# --- 1) what is the live uplink? -------------------------------------------------
+addr="$(ip -4 -o addr show dev "${IFACE}" 2>/dev/null | awk '{print $4}' | cut -d/ -f1 | head -1)"
+idx="$(cat "/sys/class/net/${IFACE}/ifindex" 2>/dev/null || true)"
+if [ -z "${addr}" ] || [ -z "${idx}" ]; then
+  # Tunnel down. Nothing to publish — and deliberately NOTHING is torn down here either:
+  # vpn-class is already fail-closed (Cilium drops when the cached ifindex is gone), and
+  # the kill-switch set is retained on purpose. The next successful connect re-runs this.
+  log "${IFACE} has no address yet (tunnel down) — nothing to re-steer"
+  exit 0
+fi
+# A k8s label value must be <=63 chars of [A-Za-z0-9] plus - _ . in the middle. An IPv4
+# address and an ifindex satisfy that; the separator is a dash.
+want="${addr}-${idx}"
+
+# --- 2) re-assert the host side (idempotent; repairs a kill-switch reload) --------
+# `flush table` in the .nft file keeps set elements, but a full ExecStop/ExecStart cycle
+# (or a hand-run `nft delete table`) does not, so re-register unconditionally.
+nft add element inet "${NFT_TABLE}" "${NFT_SET}" "{ ${addr} }" 2>/dev/null || \
+  log "WARNING could not add ${addr} to ${NFT_TABLE}/${NFT_SET} (is the kill-switch loaded?)"
+if ! grep -q "from ${addr} lookup" <<<"$(ip rule show)"; then
+  ip rule add from "${addr}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null || true
+fi
+ip route replace default dev "${IFACE}" table "${TABLE_ID}" metric "${TUN_METRIC}" 2>/dev/null || true
+
+# --- 3) has Cilium already been told about THIS uplink? ---------------------------
+# The label is the record of what we last published. Retried because at boot the kube-API
+# is routinely not up yet when OpenVPN connects.
+i=0
+while :; do
+  i=$((i + 1))
+  # -o json + an exact-string match rather than jsonpath: the label key contains dots and
+  # a slash, which jsonpath needs escaped differently across kubectl versions, and getting
+  # that wrong would silently read "" and re-label on every timer tick.
+  node_json="$(k3s kubectl get node "${NODE}" -o json 2>/dev/null)"
+  if [ -n "${node_json}" ]; then
+    # Here-strings, NOT pipes: with `set -o pipefail`, `grep -q` exiting early on a match
+    # SIGPIPEs the writer and the pipeline reports 141 — i.e. a successful match would read
+    # as a failure and this script would re-label on every single run.
+    if grep -qF "\"${UPLINK_LABEL}\": \"${want}\"" <<<"${node_json}"; then
+      log "uplink unchanged (${want}) — Cilium already re-steered, nothing to do"
+      exit 0
+    fi
+    have="$(grep -F "\"${UPLINK_LABEL}\": " <<<"${node_json}" | head -1 | cut -d'"' -f4)"
+    log "uplink changed: label='${have:-<unset>}' live='${want}' — re-steering"
+    if k3s kubectl label node "${NODE}" "${UPLINK_LABEL}=${want}" --overwrite >/dev/null 2>&1; then
+      log "re-steered: ${NODE} ${UPLINK_LABEL}=${want}"
+      exit 0
+    fi
+  fi
+  if [ "${i}" -ge "${RETRIES}" ]; then
+    log "ERROR kube-API unreachable or label write failed after ${i} attempts — the timer will retry"
+    exit 1
+  fi
+  sleep "${RETRY_SLEEP}"
+done
@@ -0,0 +1,19 @@
+# {{ ansible_managed }}
+# Convergence backstop for the event-driven re-steer. The OpenVPN up-script fires
+# vpn-egress-resteer.service on every connect, but that call can be lost (kube-API not up
+# yet at boot and all in-script retries exhausted, an OpenVPN soft restart whose up-script
+# ran before k3s, a hand-run `nft delete table`). This timer makes the gateway converge on
+# its own instead of needing an operator. The script is a no-op when the published label
+# already matches the live tun0, so running it often is free.
+[Unit]
+Description=Periodic re-steer of the Cilium Egress Gateway onto the live tun0 — ADR-0015
+Documentation=file://{{ vpn_egress_gateway_scripts_dir }}/vpn-egress-resteer.sh
+
+[Timer]
+OnBootSec={{ vpn_egress_gateway_resteer_timer_boot }}
+OnUnitInactiveSec={{ vpn_egress_gateway_resteer_timer_interval }}
+AccuracySec=15s
+Unit=vpn-egress-resteer.service
+
+[Install]
+WantedBy=timers.target
@@ -35,9 +35,34 @@ ip route replace blackhole default table "${TABLE_ID}" metric "${BH_METRIC}"
 while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done
 ip rule add from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}"

-# Register the live tunnel source with the kill-switch set so the nft backstop can scope
-# to exactly this traffic. Flush any stale members first (address changes on reconnect).
-nft flush set inet "${NFT_TABLE}" "${NFT_SET}" 2>/dev/null || true
+# Register the live tunnel source with the kill-switch set. ADDITIVE ON PURPOSE — the set
+# is NEVER flushed here (it used to be, see the DECISIONS-LOG entry of 2026-09-12).
+# `nft add element` is idempotent, and every address that has ever been this tunnel's
+# source can only ever TIGHTEN the `oifname != tun0 drop` rule. Flushing first meant that
+# between a reconnect and Cilium re-resolving its cached egress IP there was a moment
+# where a packet still sourced from the PREVIOUS address matched neither the ip-rule nor
+# the drop rule. (Measurement since showed that window cannot leak for POD traffic,
+# because that traffic never enters netfilter at all — see the kill-switch .nft header —
+# but node-originated tunnel traffic DOES, and additive costs nothing.)
 nft add element inet "${NFT_TABLE}" "${NFT_SET}" "{ ${SRC} }"

+# THE LOAD-BEARING FIX (2026-09-12): tell Cilium that this gateway's uplink changed.
+#
+# Cilium resolves the CiliumEgressGatewayPolicy's `interface: tun0` ONCE, into a cached
+# (egress IP, egress ifindex) pair, and — measured on k3s-dev-3 — does NOT re-resolve it
+# when tun0 changes underneath, not even with `egress-gateway-reconciliation-trigger-
+# interval: 1s`. Upstream says as much: "it does not automatically respond to a change in
+# the gateway node's network configuration... You can force a fresh selection by
+# re-applying the Egress Gateway policy."
+#
+# Every tunnel restart re-creates tun0 with a NEW ifindex, so WITHOUT this the gateway is
+# silently dead after its first reconnect or reboot: vpn-class egress is blackholed (no
+# leak — the eBPF redirect to the dead ifindex just fails — but no service, and no alert).
+# Re-steering is done out-of-band by vpn-egress-resteer.service so this hook can never
+# block or fail the tunnel: OpenVPN's up-script must stay fast and must not depend on the
+# kube-API being reachable yet (at boot it usually is not — the resteer unit retries, and
+# its timer converges anyway).
+systemctl start --no-block vpn-egress-resteer.service || \
+  echo "vpn-egress-up: WARNING could not trigger vpn-egress-resteer.service (timer will converge)" >&2
+
 echo "vpn-egress-up: tun=${IFACE} src=${SRC} table=${TABLE_ID} — split-tunnel routing active"
@@ -8,8 +8,12 @@ Egress is realized per **[ADR-0015](../../../home-network-k3s/decisions/adr-0015
 by a **Cilium Egress Gateway** policy, with a **host nftables kill-switch**. There is **no in-cluster
 gateway pod** — the tunnel is host infrastructure, not a Flux workload.

-> **STATUS: live + conformance-green.** The node tunnel was validated end-to-end; the `egress-class`
-> (R-NET-05/07) and `vpn-killswitch` (R-NET-06) conformance tests pass against the live cluster.
+> **STATUS: live on BOTH clusters** — `k3s-dev-3` and, since 2026-09-12, `k3s-prod-3`.
+> The `egress-class` (R-NET-05/07) and `vpn-killswitch` (R-NET-06) tests pass on dev.
+> **Caveat on that "conformance-green":** the `vpn-killswitch` test stops and then restarts the
+> gateway tunnel, and until 2026-09-12 a restarted tunnel never recovered (Cilium kept a stale
+> egress ifindex — see below), so the test's recovery step could not have passed on the
+> implementation as it stood. Whatever produced the earlier green result, re-run it.

 ## How it works (ADR-0015)
 Three layers, two owners:
@@ -24,9 +28,134 @@ Three layers, two owners:
  (Kyverno-enforced) and SNATs them out the gateway node's `tun0`. Native Cilium eBPF — no client-pod
  changes, no sidecar, no overlay. `excludedCIDRs` keeps RFC1918 / cluster / LAN traffic on the local
  datapath; only public destinations leave via the tunnel.
- **Kill-switch — host nftables (Ansible).** A host nftables rule on the gateway node DROPS the steered
-  `vpn`-class traffic whenever `tun0` is down (fail-closed, R-NET-06), scoped to vpn-class only so
-  `internet`-class and node traffic are unaffected.
+- **Kill-switch — Cilium's own drop, NOT host nftables (corrected 2026-09-12).** vpn-class fails
+  closed because Cilium drops the traffic when it cannot use its cached egress interface. The host
+  nftables table is retained for a narrower job. See "The kill-switch does not do what this file
+  used to claim" below — that section is the measurement, not an opinion.
+
+## LIVE ON PROD since 2026-09-12 — `k3s-prod-3`
+
+| | dev | prod |
+|---|---|---|
+| gateway node | `k3s-dev-3` | `k3s-prod-3` |
+| inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) |
+| PIA server | `158.173.241.112` (the .ovpn's pin) | `158.173.241.135` (group_vars override — see below) |
+| vpn-class egress IP | `158.173.241.36` | `158.173.241.11` |
+| internet-class egress IP | home WAN | `136.51.115.252` (home WAN) |
+
+Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three
+nodes — all egress the PIA address; plex and nextcloud (internet-class) still egress the home
+WAN. **Before the gateway existed, prod's vpn-class pods had no internet at all**: the CEGP was
+reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress list` showed
+`Gateway IP: Not Found` for their `0.0.0.0/0` entry and the traffic was dropped. Arming the
+gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also
+means nothing had been leaking in the meantime.
+
+### One PIA account is fine. One pinned PIA *server* is not.
+
+Standing prod up broke dev, and the reason is not the connection limit. The vaulted `.ovpn`
+pins **one** server, so both gateways dialled it — and PIA handed **both sessions the same
+tunnel address**. Verbatim from both journals:
+
+```
+[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198
+PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ...
+```
+
+Each reconnect steals `10.72.192.7` on that server; the loser receives nothing, trips
+`--ping-restart` after 60 s, reconnects and steals it back. Measured cycle: dev restarting at
+22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC with prod interleaved about a minute
+off — every ~2 minutes, indefinitely. Neither environment had reliable vpn-class egress; prod's
+first probes looked green only because they landed inside an up-window.
+
+Dev recovered the minute prod left `.112`: its last `Inactivity timeout` is 23:05:06 UTC, and at
+23:12:03 that same session was still up — seven minutes unbroken against a two-minute cycle — with
+a source-bound ping returning 4/4 at 146 ms and vpn-class pod egress back to `158.173.241.36`.
+
+**Fix:** `vpn_egress_gateway_pia_remote` in `group_vars/vpn_egress_gateway_prod/vars.yml`
+rewrites just the `remote` line, pointing prod at `158.173.241.135` — another member of the
+same PIA Stockholm LB pool (`sweden.privacy.network` resolves to
+`158.173.241.{34,55,112,135,239}`; every server in it takes the same credentials and CA). Dev
+keeps `.112`.
+
+**So: one gateway per PIA server.** A third gateway, or a rebuild that re-pins a server, must
+pick an unused address from the pool. A separate prod PIA account would also solve it — see
+`infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`.
+
+That switch also produced the cleanest possible live proof of the re-steer path, for free: prod's
+tunnel address changed from `10.72.192.7` (ifindex 128) to `10.150.192.86` (ifindex 129) — a real
+new PIA address on a real reconnect, the exact case Cilium does not follow. The `egress-uplink`
+label moved to `10.150.192.86-129` on its own and all four vpn-class workloads kept egressing,
+now via `158.173.241.11`. No manual step, no CEGP edit.
+
+## The kill-switch does not do what this file used to claim (measured 2026-09-12)
+
+This README, `egress-steering.yaml`, and the role all used to say the **host nftables
+kill-switch** is the authoritative fail-closed backstop for `vpn`-class traffic. **It cannot be.**
+With `bpf.masquerade: true` — which Egress Gateway requires — the SNAT and the egress happen in
+eBPF, which `bpf_redirect()`s the skb straight to the egress device. netfilter POSTROUTING is
+never invoked for those packets.
+
+A counters-only nft table installed alongside the real one on the gateway node, while a vpn-class
+pod egressed:
+
+| counter | packets |
+|---|---|
+| `forward`, saddr = pod CIDR | **0** |
+| `forward`, saddr = PIA tunnel range | **0** |
+| `postrouting`, oif tun0, saddr = tunnel range | **0** |
+| `postrouting`, oif WAN, saddr = pod CIDR | **0** |
+| `postrouting`, oif WAN, any (control: the hook IS live) | 45166 |
+| `postrouting`, oif tun0, saddr = tunnel (control: a node-originated curl) | 16 |
+
+**What actually fails closed is Cilium itself.** When it cannot use its cached egress
+interface/IP it drops the packet. Verified with tcpdump on *both* the WAN and tun0, during a
+tunnel stop and during a tunnel restart: **0 packets left either device.** There is no leak — the
+guarantee is real — but the mechanism is Cilium's, and the existing `vpn-killswitch` conformance
+test passes because of it, not because of the nft rules.
+
+The nft table is **kept**, with a `counter` on every rule, for the narrower thing it genuinely
+does cover: **node-originated** traffic sourced from a tunnel address. That was proven on
+`k3s-prod-3` in both directions — a source-bound ping with the tunnel ip-rule present got 3/3
+replies and moved `killswitch_accept`; with the ip-rule removed (so the routing table sent it out
+`eno1`) `killswitch_drop` moved **0 -> 3 packets / 252 bytes** and the ping got nothing back.
+If `killswitch_drop` ever starts counting in normal operation, the datapath has changed — read
+that as news, not reassurance.
+
+Also removed: the `meta mark 0x1c8 oifname != tun0 drop` "contract hook". Nothing stamps that
+mark (the only Cilium mark on the gateway node is `0x200/0xf00`, the L7-proxy rule to table
+2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is
+a hazard to whatever component picks that value next. The role's `vpn_egress_gateway_fwmark`
+variable is gone with it, and the "add the fwmark to the interface contract" escalation in the
+role README is **closed as not-implementable** rather than left open.
+
+## Cilium never re-resolves `tun0` — the gateway dies on every reconnect without help
+
+Cilium resolves `interface: tun0` once into a cached (egress IP, egress ifindex) and does not
+revisit it. Every tunnel restart re-creates tun0 with a **new ifindex**, so the cached one goes
+stale and vpn-class egress is silently blackholed. Found the hard way: the dev gateway had been
+dead for ~80 minutes when this work started, with no alert and nothing in the ledger.
+
+`egress-gateway-reconciliation-trigger-interval: 1s` does **not** help; both clusters carry that
+setting and dev still did not self-heal 20s+ after the tunnel was back. It coalesces
+reconciliations that k8s object events already triggered — netlink link events are not among them.
+
+The Ansible role now publishes `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on
+the gateway node on every connect, plus a 2-minute convergence timer. That node update makes
+Cilium re-process the policy and re-resolve the interface (verified: cached ifindex 1986 -> 1987
+within 10s, egress recovered immediately). A node label rather than an annotation on this
+Flux-managed policy, so GitOps state is never written to from a node.
+
+**Debugging a dead gateway starts here:**
+
+```bash
+kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink
+kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
+  cilium-dbg bpf egress list      # Egress Ifindex must match the node's live `ip link show tun0`
+```
+
+If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
+gateway node is the next stop.

 ## Resilience — there is NO gateway failover (spec correction, escalated)

@@ -59,16 +59,38 @@ spec:
  # Excluded flows fall through to Cilium's normal local datapath (invariant 1).
  destinationCIDRs:
    - "0.0.0.0/0"
+  #
+  # BULK NFS (added to the record 2026-09-12, when plex/sonarr/radarr/nextcloud gained media
+  # mounts): the export is 192.168.2.102:/mnt/Storage/plex on prod (192.168.1.9 on dev), both
+  # inside 192.168.0.0/16 below, so NFS is excluded from the tunnel twice over. The FIRST reason
+  # is the one that actually matters and needs no CIDR at all: an NFS PersistentVolume is mounted
+  # by the NODE KERNEL in the host netns, not by the pod, so `selectors.podSelector` never
+  # matches it and no pod-scoped egress steering can touch it. Verified after arming the prod
+  # gateway: sonarr (k3s-prod-1) and radarr (k3s-prod-3, the gateway node itself) both still list
+  # AND write the export, and the node's own mounts are intact. No RFC1918 destination needs
+  # adding here for the media path.
  excludedCIDRs:
    - "10.0.0.0/8"       # RFC1918 + Cilium cluster-pool pod CIDR (CLUSTER_POD_CIDR) + k3s service net
    - "172.16.0.0/12"    # RFC1918
-    - "192.168.0.0/16"   # RFC1918 (home LAN / node network family)
+    - "192.168.0.0/16"   # RFC1918 (home LAN / node network family / the bulk NFS server)
  # --- Where it egresses --------------------------------------------------
-  # The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`. When
-  # tun0 is absent/down, Cilium has no egress interface here; the host nftables kill-switch (Ansible,
-  # ADR-0015) is the authoritative fail-closed backstop that DROPS this vpn-class traffic — scoped to
-  # vpn-class only, so default/internet-class is unaffected (invariants 3 & 4: vpn fails CLOSED,
-  # internet fails OPEN).
+  # The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`.
+  #
+  # CORRECTION 2026-09-12 — this comment used to say the host nftables kill-switch "is the
+  # authoritative fail-closed backstop that DROPS this vpn-class traffic". It is not, and cannot
+  # be. With `bpf.masquerade: true` the SNAT and egress happen in eBPF, which bpf_redirect()s the
+  # skb to the egress device without invoking netfilter POSTROUTING, so the host nft rules never
+  # see this traffic. Measured with a counters-only nft table on the gateway node while a
+  # vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and
+  # the PIA range, versus 45166 packets on that same hook for WAN traffic (so the hook is live)
+  # and 16 for a node-originated curl out tun0 (so the matching works).
+  # WHAT ACTUALLY FAILS CLOSED: Cilium itself. When the cached egress interface/IP cannot be used
+  # it drops the packet — tcpdump on both the WAN and tun0 during a tunnel stop AND during a
+  # tunnel restart caught 0 packets leaving either device. vpn fails CLOSED (verified, no leak);
+  # internet-class and node traffic are never selected and fail OPEN (invariants 3 & 4 hold).
+  # The host nft table is retained, with counters, for the narrower case it CAN see:
+  # node-originated traffic sourced from a tunnel address (proven on k3s-prod-3 — the drop rule's
+  # counter moved 0 -> 3 packets when such a packet was routed at the WAN).
  #
  # SINGLE GATEWAY ONLY — there is NO failover, and a second labelled node is NOT a standby.
  # `egressGateway` (singular) resolves the nodeSelector to ONE node. Verbatim from the CRD
@@ -89,6 +111,22 @@ spec:
  # ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)"
  # is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see
  # platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06.
+  #
+  # `interface: tun0` IS RESOLVED ONCE — the gateway node has to tell Cilium when tun0 changes.
+  # Measured on k3s-dev-3 2026-09-12: Cilium caches an (egress IP, egress ifindex) pair from the
+  # first resolution and does NOT re-resolve on a netlink change. `systemctl restart
+  # vpn-egress-tunnel` re-creates tun0 with a NEW ifindex; the map kept the old one and vpn-class
+  # egress was silently blackholed (tcpdump: 0 packets out the WAN, 0 out tun0 — fail-closed, but
+  # dead). `egress-gateway-reconciliation-trigger-interval: 1s` does not cover link events.
+  # Upstream: "it does not automatically respond to a change in the gateway node's network
+  # configuration... You can force a fresh selection by re-applying the Egress Gateway policy."
+  # The Ansible role closes this: roles/vpn_egress_gateway publishes
+  # `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node on every
+  # connect (plus a 2-minute convergence timer). That node update makes Cilium re-process this
+  # policy and re-resolve the interface — verified: cached ifindex 1986 -> 1987 within 10s.
+  # DO NOT "fix" this by having the node write to THIS object: it is Flux-managed, and a node
+  # writing to GitOps state is a drift fight. Check the label, not this file, when debugging:
+  #   kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink
  egressGateway:
    nodeSelector:
      matchLabels: