docs(vpn): stop crediting the nftables kill-switch for a drop it never makes

Four places in the tree told the reader that the host nftables table is the
authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress
README, egress-steering.yaml, the role README, and the vpn-killswitch test's own
GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway
datapath bpf_redirect()s those skbs to the egress device without invoking
netfilter POSTROUTING. Corrected with the measurement in each place rather than
softened, because a guard the repo trusts and that does not fire is worse than no
guard.

What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops
the traffic when it cannot use its cached egress interface, verified by tcpdump on
both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out
either). The nft table keeps a narrower, now-proven job: node-originated traffic
sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on
k3s-prod-3 when such a packet was routed at the WAN).

Also recorded:

* egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the
  gateway node is what tells Cilium it changed (the egress-uplink node label). A
  future reader's instinct will be to have the node write to that object; the
  comment says not to, because it is Flux-managed, and points at the label and
  `cilium-dbg bpf egress list` as the debugging entry points instead.
* egress-steering.yaml — why the new NFS media mounts are not affected by
  pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns,
  so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server
  too, but that is the second reason, not the first.
* vpn-killswitch test — its recovery step depends on the re-steer path, and could
  not have passed before it existed. Its no-leak assertion was always right; only
  the explanation was wrong. Also states the prerequisite plainly: the test cannot
  run at all without a labelled gateway node.
* platform README — prod is live on k3s-prod-3 with the before/after egress IPs,
  and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no
  matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway
  restored egress rather than moving it off the WAN.
* role README — the standing "add the shared fwmark to the interface contract"
  escalation is closed as not implementable on Cilium OSS, rather than left open
  for someone to attempt.

DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 /
docs-03 amendment for the architect (specs are immutable to agents) and the five
things left for the operator — chief among them that a silent gateway outage still
has no alert, which is what let the dev gateway sit dead for 80 minutes.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -2089,3 +2089,200 @@ fail-closed branch fixes the "unmapped path contributes nothing" hole; it does n
 Both belong to `ci/`, so they are left to that branch rather than smuggled in here. The full L0
 layer was run unscoped on this branch to compensate: **12 offline checks, 11 PASS, 1 SKIP**
 (`placeholder-guard-coupling`, pre-existing), 0 FAIL.
+
+## 2026-09-12 — VPN egress on PROD (`k3s-prod-3`), and three things the repo believed that are not true (branch `feat/vpn-egress-prod`)
+
+The operator asked for VPN egress on prod. An earlier assessment this session had recommended
+holding off over three suspected defects; the operator decided, so the defects were fixed first
+and then the gateway was stood up. All three were settled by **measurement on the dev gateway
+node**, not by reading code — and two of the three turned out to be different from what was
+suspected, in one case much worse.
+
+### The headline: the DEV gateway was already dead, silently, and had been for ~80 minutes
+
+`cilium-dbg bpf egress list` on `k3s-dev-3` showed `Egress Ifindex 3`. There is no ifindex 3 on
+that node (the list goes 1, 2, 4, 5, …) — it was a **destroyed** tun0. The live tun0 was ifindex
+1780. vpn-class pods on both the gateway node and a remote node timed out. Nothing alerted, and
+the ledger still recorded the gateway as conformance-green.
+
+**Cilium resolves a `CiliumEgressGatewayPolicy`'s `interface: tun0` exactly once**, caching an
+(egress IP, egress ifindex) pair, and does not re-resolve when tun0 changes. Upstream is explicit:
+*"it does not automatically respond to a change in the gateway node's network configuration… You
+can force a fresh selection by re-applying the Egress Gateway policy."* Every tunnel restart
+re-creates tun0 with a new ifindex, so **the gateway is dead after its first reconnect or reboot.**
+`egress-gateway-reconciliation-trigger-interval: 1s` does not cover it — both clusters carry that
+setting and dev was still stale 20s+ after recovery; the interval coalesces reconciliations that
+k8s object events already triggered, and netlink link events are not among them.
+
+Measured, in order:
+
+| step | observation |
+|---|---|
+| baseline | map ifindex 3, live tun0 1780, vpn-class pods time out |
+| annotate the CEGP (the upstream remedy) | map -> 1780 within 8s; pods return the PIA IP `158.173.241.36` immediately |
+| `systemctl restart vpn-egress-tunnel` | tun0 1780 -> 1985, map STAYS 1780, pods break again |
+| `systemctl stop`, wait 30s, `start` (the exact `vpn-killswitch` conformance sequence) | tun0 -> 1990, map stays 1989, pods still broken 20s+ later |
+| set ANY unrelated label on the gateway NODE | map -> live ifindex within 10s, egress recovers |
+
+**Fix:** `roles/vpn_egress_gateway` now publishes
+`homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node — from the
+OpenVPN up-script on every connect (via `systemctl start --no-block`, so it can never block or
+fail the tunnel and does not need the kube-API up at that instant), plus a 2-minute convergence
+timer for the boot case. A **node label, not an annotation on the policy**: the policy is
+Flux-managed GitOps and a node writing to GitOps state is a drift fight, whereas node labels are
+already this role's territory — and the label's value doubles as the diagnostic. After the fix,
+stop->start recovers unaided (map 1989 -> 1990, PIA IP returns).
+
+**Consequence for the ledger:** the `vpn-killswitch` (R-NET-06) test stops *and restarts* the
+tunnel. Its recovery step **could not have passed** on the implementation as it stood — verified
+twice. Whatever produced the earlier green result needs re-running; the fail-closed half was and
+remains sound.
+
+### Defect as reported vs. as measured
+
+**Reported:** the up-script's `nft flush set` opens a leak window — on reconnect Cilium keeps
+SNAT-ing to the previous address A while only B is in the set, so packets sourced from A match
+neither the ip-rule nor the drop rule and **leave via the WAN**.
+
+**Measured: there is no WAN leak.** Cilium's redirect follows the cached *ifindex*, not a routing
+lookup, so a stale egress IP cannot reach the WAN. Tested directly by swapping tun0's address in
+place (keeping the ifindex, as an OpenVPN `persist-tun` soft reconnect does) while Cilium held the
+old one: the pod's packets left **tun0** carrying the stale source, PIA answered, and tcpdump on
+the WAN caught **0** packets. Same result in the stale-ifindex case: 0 packets out the WAN, 0 out
+tun0 — the redirect to a dead ifindex simply fails and the packet is dropped.
+
+The flush was still wrong, so the set is now **additive** (`add element` is idempotent; the
+down-script already retained it). It cannot leak pod traffic — see the next section for why — but
+node-originated tunnel traffic *does* traverse netfilter, and additive costs nothing. A kill-switch
+reload no longer disarms the guard either: the `.nft` file uses `table` + `flush table`, which
+clears rules but leaves a named set's elements intact.
+
+### The nftables kill-switch cannot see the traffic it was written to guard
+
+Suspected; now proven, with the counters the brief asked for. A counters-only diagnostic nft table
+on the gateway node, while a vpn-class pod egressed:
+
+| counter | packets |
+|---|---|
+| `forward`, saddr = pod CIDR | **0** |
+| `forward`, saddr = PIA tunnel range | **0** |
+| `postrouting`, oif tun0, saddr = tunnel range | **0** |
+| `postrouting`, oif WAN, saddr = pod CIDR | **0** |
+| `postrouting`, oif WAN, any — control, the hook is live | 45166 |
+| `postrouting`, oif tun0, saddr = tunnel — control, a node-originated curl | 16 |
+
+With `bpf.masquerade: true` (which Egress Gateway requires) the SNAT and egress happen in eBPF,
+which `bpf_redirect()`s the skb to the egress device without invoking netfilter POSTROUTING. The
+hook works and the matching works — the steered pod packets are simply never presented to it.
+
+**So what fails closed is Cilium's own drop**, confirmed by tcpdump on both devices during a
+tunnel stop and a tunnel restart: 0 packets left either. The existing `vpn-killswitch` test passes
+because of that, not because of the nft rules. Every comment in the tree that credited the nft
+table has been corrected rather than left to mislead the next reader.
+
+**The table is kept, not deleted**, because it does cover something real — node-originated traffic
+sourced from a tunnel address — and that was proven in **both** directions on `k3s-prod-3`:
+
+- source-bound `ping -I 10.72.192.7` with the tunnel ip-rule present: 3/3 replies at 144 ms,
+  `killswitch_accept` advances;
+- ip-rule removed, so `ip route get` says `via 192.168.2.1 dev eno1`: **`killswitch_drop`
+  0 -> 3 packets / 252 bytes**, 100% loss.
+
+Every rule now carries a `counter` so the claim stays falsifiable. If `killswitch_drop` ever
+counts in normal operation, the datapath has changed.
+
+Also **removed** the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable. Nothing stamps
+that mark — the only Cilium mark on the gateway node is `0x200/0xf00`, which is the L7-proxy rule
+to table 2004 — and the traffic never reaches netfilter anyway. A live rule matching an arbitrary
+mark is a hazard to whatever component picks that value next, not a backstop. This **closes** the
+role README's standing escalation ("add the shared fwmark to the interface contract so the
+CiliumEgressGatewayPolicy marks vpn-class traffic") as **not implementable on Cilium OSS**: there
+is no supported way to make the policy stamp a mark, and no host firewall hook to catch it in.
+
+### Prod stand-up
+
+`[vpn_egress_gateway_prod]` is a **separate** inventory group holding only `k3s-prod-3`, with its
+own `group_vars/vpn_egress_gateway_prod/vault.yml`. Prod is deliberately NOT a member of
+`[vpn_egress_gateway]`, which is the default target of `playbooks/vpn.egress.gateway.yml` — sharing
+the group would mean a flagless run silently reconfigures a prod node's kill-switch datapath. Run
+as `-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3`, with `--list-hosts`
+first (confirmed: exactly `k3s-prod-3`).
+
+Node 3 because node 1's SSD was replaced 2026-09-11 (least operational history) and node 2 carries
+the §0b memattr history. **Exactly one node, forever**: Cilium OSS picks "the first node in lexical
+ordering" among nodeSelector matches, so labelling `k3s-prod-1` would silently take the gateway
+role, and the public egress IP, away from `k3s-prod-3`.
+
+The vault file is a **verbatim copy of dev's ciphertext** — same PIA account per the operator, and
+copying the encrypted blob means the plaintext never touched disk, a shell, or a transcript. The
+two files are independent from here on; rotating means re-encrypting both, which is correct for two
+environments.
+
+**`wlp1s0` did not need bringing down — it does not exist.** The brief and `hosts.ini` both said
+node 3 has wlp1s0 up on 192.168.2.195. Re-verified before choosing it, because a second live uplink
+in a kill-switch datapath is not a footnote: there is no `wl*` netdev on the node, no `phy80211`
+device, the `wlp1s0` stanza in `/etc/network/interfaces` is commented out, 192.168.2.195 is
+assigned nowhere, and the only IPv4 addresses are `lo` and `eno1 192.168.2.12`. The inventory
+comment is corrected.
+
+**Result on prod:**
+
+| | before | after |
+|---|---|---|
+| jackett / qbittorrent / radarr / sonarr (vpn-class, spread across all 3 nodes) | no internet at all — CEGP reconciled with no matching node, `cilium-dbg bpf egress list` showed `Gateway IP: Not Found`, traffic dropped | all four egress `158.173.241.36` (PIA Stockholm) |
+| plex / nextcloud (internet-class) | `136.51.115.252` (home WAN) | `136.51.115.252`, unchanged |
+| node's own egress | WAN | WAN — `default via 192.168.2.1 dev eno1`, k3s `/readyz` ok |
+
+Worth stating plainly: arming the gateway **restored** vpn-class egress rather than moving it from
+the WAN to the tunnel. Nothing had been leaking in the meantime, because nothing had been going
+out.
+
+### The NFS media mounts are unaffected, verified rather than assumed
+
+sonarr, radarr, plex and nextcloud gained NFS mounts from `192.168.2.102:/mnt/Storage/plex` (and
+`/mnt/Storage/nextcloud`) earlier today, so the interaction with pod-scoped egress steering was
+checked before and after arming:
+
+- **The structural reason it cannot break:** an NFS PersistentVolume is mounted by the **node
+  kernel in the host netns**, not by the pod, so the CEGP's `selectors.podSelector` never matches
+  it. No `excludedCIDRs` entry is needed for that.
+- **And belt-and-braces:** the server is `192.168.2.102`, inside the existing
+  `192.168.0.0/16` exclusion anyway.
+- **Empirically, after arming:** sonarr (on `k3s-prod-1`) and radarr (on `k3s-prod-3`, the gateway
+  node itself) both still `ls` **and write** to `/media`; plex lists it; the node's own
+  `findmnt`-visible mounts are intact and readable. qBittorrent, incidentally, has **no** NFS
+  media mount in this repo — only plex, sonarr, radarr and nextcloud do.
+
+### Left for the operator / not settled here
+
+1. **Re-run `vpn-killswitch` (R-NET-06) against a labelled cluster** and refresh the ledger. It can
+   only run where a node carries the gateway label, and its recovery step is only passable with
+   this branch's re-steer fix. It is an L3 fault-injection test that stops the tunnel on a LIVE
+   cluster, so it was not fired at prod unasked.
+2. **Both clusters now exit from the same PIA public IP** (`158.173.241.36`) on one account. Inside
+   the connection limit, but the two environments are indistinguishable from outside. A separate
+   prod PIA account is the clean fix if that ever matters.
+3. **Gateway-node loss is still an outage** — Cilium OSS has no egress-gateway failover, and the
+   ADR-0016 supersede drafted on 2026-09-06 is still open. Nothing here changes that; the
+   single-gateway choice is now recorded in the inventory itself, not just in prose.
+4. **A silent gateway outage still has no alert.** The re-steer timer means it now self-heals, but
+   nothing watches the invariant "the gateway node's `egress-uplink` label matches
+   `cilium-dbg bpf egress list`'s Egress Ifindex". That is a monitoring gap, not a datapath one,
+   and it is the thing that let dev sit dead for 80 minutes. A Prometheus rule over a small
+   node-exporter textfile from the re-steer script would close it.
+5. **Hygiene, unrelated but found on the way:** `/etc/network/interfaces` on `k3s-prod-3` holds a
+   commented-out wireless stanza containing the home WiFi **PSK in plaintext**. It is inert
+   (wireless is gone) and not printed here. Worth scrubbing on all three nodes.
+6. **`ADR-0015` and `docs/03` still describe the nftables kill-switch as the vpn-class
+   fail-closed mechanism**, and specs are immutable to agents. Drafted amendment for the architect:
+   > *R-NET-06's fail-closed guarantee for `egress-class=vpn` is provided by the Cilium Egress
+   > Gateway datapath, which drops steered traffic when the gateway's egress interface cannot be
+   > used; this was verified with packet capture on both the WAN and tunnel devices. The host
+   > nftables table is a defence-in-depth guard for node-originated traffic sourced from the tunnel
+   > address only — with `bpf.masquerade` enabled, which Egress Gateway requires, steered pod
+   > traffic does not traverse netfilter and cannot be guarded there. The "optional fwmark
+   > contract hook" is withdrawn: Cilium OSS provides no way for the policy to stamp such a mark.*
+   > *Additionally, `interface:` on a CiliumEgressGatewayPolicy is resolved once and cached; a
+   > gateway whose tunnel interface is dynamic requires an explicit re-resolution trigger on every
+   > interface change. This is a standing operational requirement of the ADR-0015 design, not an
+   > implementation detail.*