docs(vpn): stop crediting the nftables kill-switch for a drop it never makes

Four places in the tree told the reader that the host nftables table is the
authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress
README, egress-steering.yaml, the role README, and the vpn-killswitch test's own
GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway
datapath bpf_redirect()s those skbs to the egress device without invoking
netfilter POSTROUTING. Corrected with the measurement in each place rather than
softened, because a guard the repo trusts and that does not fire is worse than no
guard.

What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops
the traffic when it cannot use its cached egress interface, verified by tcpdump on
both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out
either). The nft table keeps a narrower, now-proven job: node-originated traffic
sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on
k3s-prod-3 when such a packet was routed at the WAN).

Also recorded:

* egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the
  gateway node is what tells Cilium it changed (the egress-uplink node label). A
  future reader's instinct will be to have the node write to that object; the
  comment says not to, because it is Flux-managed, and points at the label and
  `cilium-dbg bpf egress list` as the debugging entry points instead.
* egress-steering.yaml — why the new NFS media mounts are not affected by
  pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns,
  so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server
  too, but that is the second reason, not the first.
* vpn-killswitch test — its recovery step depends on the re-steer path, and could
  not have passed before it existed. Its no-leak assertion was always right; only
  the explanation was wrong. Also states the prerequisite plainly: the test cannot
  run at all without a labelled gateway node.
* platform README — prod is live on k3s-prod-3 with the before/after egress IPs,
  and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no
  matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway
  restored egress rather than moving it off the WAN.
* role README — the standing "add the shared fwmark to the interface contract"
  escalation is closed as not implementable on Cilium OSS, rather than left open
  for someone to attempt.

DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 /
docs-03 amendment for the architect (specs are immutable to agents) and the five
things left for the operator — chief among them that a silent gateway outage still
has no alert, which is what let the dev gateway sit dead for 80 minutes.

Co-Authored-By: Claude Opus 5 <[email protected]>

conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml

@@ -10,8 +10,31 @@
 #       egress-gateway node is STOPPED (fault injection) —
 # THEN  the vpn-class workload reaches NOTHING on the plain internet: its curl FAILS
 #       and, critically, does NOT return the WAN IP (a WAN IP = LEAK = test FAILS).
-#       The host nftables kill-switch blackholes egress-class=vpn traffic while tun0
-#       is absent (ADR-0015 invariant 3/4: vpn fails CLOSED).
+#       vpn-class egress is blackholed while tun0 is absent (ADR-0015 invariant 3/4:
+#       vpn fails CLOSED).
+#
+#       MECHANISM CORRECTED 2026-09-12: this comment used to credit "the host nftables
+#       kill-switch". That table never sees this traffic — Cilium's eBPF egress-gateway
+#       datapath (bpf.masquerade) bpf_redirect()s the steered skbs to the egress device
+#       without invoking netfilter POSTROUTING, measured with counters on the gateway
+#       node (0 packets for the pod CIDR and the PIA range in forward and postrouting,
+#       against 45166 on the same hook for WAN traffic). What fails closed is CILIUM's
+#       own drop when it cannot use the cached egress interface; tcpdump on both the WAN
+#       and tun0 during the stop caught 0 packets. The ASSERTION below is unchanged and
+#       still the right one — only the explanation was wrong.
+#
+#       PREREQUISITE, AND IT IS NOT OPTIONAL: this test cannot run at all unless a node
+#       carries homelab.keithjackson.dev/egress-gateway=true (the fault Job is pinned to
+#       it by nodeSelector, and with no gateway there is no tunnel to stop).
+#
+#       RECOVERY DEPENDS ON THE RE-STEER PATH. Stopping and starting the tunnel gives
+#       tun0 a NEW ifindex, and Cilium does not re-resolve `interface: tun0` on its own,
+#       so before 2026-09-12 the recovery step could not pass: measured, the map stayed
+#       on the old ifindex and vpn-class egress never came back. roles/vpn_egress_gateway
+#       now publishes the live uplink as a node label on every connect, which makes
+#       Cilium re-resolve; with it, stop->start recovers unaided (map 1989 -> 1990, PIA IP
+#       returns). If this test's recovery step ever fails again, check
+#       `journalctl -u vpn-egress-resteer` on the gateway node first.
 # AND   the `internet`-class workload is UNAFFECTED (invariant 3: internet fails OPEN).
 # WHEN  the tunnel RECOVERS (service restarted), THEN vpn-class egress returns (PIA IP).
 #