docs(vpn): record the PIA server collision, and the accidental proof it handed us

The stand-up broke dev, so the "one gateway per PIA server" rule and the
measurement behind it are now written down where the next person will look:
platform/vpn-egress/README.md and the session's DECISIONS-LOG entry. Both carry
the verbatim journal lines — same server, same pushed `ifconfig 10.72.192.7` on
both gateways — and the flap timestamps, because "same account, inside the
connection limit" is true and is exactly what makes this non-obvious.

Also corrected: the dev/prod table now shows the two environments on different
servers with different exit IPs (dev 158.173.241.36, prod 158.173.241.11), and the
open item about them sharing one public IP is replaced with the real remaining
decision — whether prod gets its own PIA account, which would remove the
hand-maintained per-environment server value entirely.

The unplanned bonus, recorded in both places: moving prod to .135 gave it a
genuinely NEW tunnel address on a real reconnect (10.72.192.7 ifindex 128 ->
10.150.192.86 ifindex 129) — the precise case Cilium does not follow and the one
defect 1 was about. The egress-uplink label moved on its own, Cilium re-resolved,
and all four vpn-class workloads kept egressing with no manual step. Better
evidence than the synthetic address swap used to characterise it, and it cost
nothing to capture.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/vpn-egress/README.md

@@ -39,7 +39,8 @@ Three layers, two owners:
 |---|---|---|
 | gateway node | `k3s-dev-3` | `k3s-prod-3` |
 | inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) |
-| vpn-class egress IP | `158.173.241.36` (PIA Stockholm) | `158.173.241.36` (PIA Stockholm) |
+| PIA server | `158.173.241.112` (the .ovpn's pin) | `158.173.241.135` (group_vars override — see below) |
+| vpn-class egress IP | `158.173.241.36` | `158.173.241.11` |
 | internet-class egress IP | home WAN | `136.51.115.252` (home WAN) |
 
 Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three
@@ -50,10 +51,38 @@ reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress lis
 gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also
 means nothing had been leaking in the meantime.
 
-Both environments currently share one PIA account and, as it happens, one public egress IP.
-That is inside the account's connection limit, but it means the two environments are not
-distinguishable from the outside; a separate prod PIA account is the clean fix if that ever
-matters (see `infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`).
+### One PIA account is fine. One pinned PIA *server* is not.
+
+Standing prod up broke dev, and the reason is not the connection limit. The vaulted `.ovpn`
+pins **one** server, so both gateways dialled it — and PIA handed **both sessions the same
+tunnel address**. Verbatim from both journals:
+
+```
+[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198
+PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ...
+```
+
+Each reconnect steals `10.72.192.7` on that server; the loser receives nothing, trips
+`--ping-restart` after 60 s, reconnects and steals it back. Measured cycle: dev restarting at
+22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC with prod interleaved about a minute
+off — every ~2 minutes, indefinitely. Neither environment had reliable vpn-class egress; prod's
+first probes looked green only because they landed inside an up-window.
+
+**Fix:** `vpn_egress_gateway_pia_remote` in `group_vars/vpn_egress_gateway_prod/vars.yml`
+rewrites just the `remote` line, pointing prod at `158.173.241.135` — another member of the
+same PIA Stockholm LB pool (`sweden.privacy.network` resolves to
+`158.173.241.{34,55,112,135,239}`; every server in it takes the same credentials and CA). Dev
+keeps `.112`.
+
+**So: one gateway per PIA server.** A third gateway, or a rebuild that re-pins a server, must
+pick an unused address from the pool. A separate prod PIA account would also solve it — see
+`infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`.
+
+That switch also produced the cleanest possible live proof of the re-steer path, for free: prod's
+tunnel address changed from `10.72.192.7` (ifindex 128) to `10.150.192.86` (ifindex 129) — a real
+new PIA address on a real reconnect, the exact case Cilium does not follow. The `egress-uplink`
+label moved to `10.150.192.86-129` on its own and all four vpn-class workloads kept egressing,
+now via `158.173.241.11`. No manual step, no CEGP edit.
 
 ## The kill-switch does not do what this file used to claim (measured 2026-09-12)