From 38e4bf1459ca4998c1328d74bd1adbc2f4f93387 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 18:09:31 -0500 Subject: [PATCH] docs(vpn): record the PIA server collision, and the accidental proof it handed us MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The stand-up broke dev, so the "one gateway per PIA server" rule and the measurement behind it are now written down where the next person will look: platform/vpn-egress/README.md and the session's DECISIONS-LOG entry. Both carry the verbatim journal lines — same server, same pushed `ifconfig 10.72.192.7` on both gateways — and the flap timestamps, because "same account, inside the connection limit" is true and is exactly what makes this non-obvious. Also corrected: the dev/prod table now shows the two environments on different servers with different exit IPs (dev 158.173.241.36, prod 158.173.241.11), and the open item about them sharing one public IP is replaced with the real remaining decision — whether prod gets its own PIA account, which would remove the hand-maintained per-environment server value entirely. The unplanned bonus, recorded in both places: moving prod to .135 gave it a genuinely NEW tunnel address on a real reconnect (10.72.192.7 ifindex 128 -> 10.150.192.86 ifindex 129) — the precise case Cilium does not follow and the one defect 1 was about. The egress-uplink label moved on its own, Cilium re-resolved, and all four vpn-class workloads kept egressing with no manual step. Better evidence than the synthetic address swap used to characterise it, and it cost nothing to capture. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 46 ++++++++++++++++++++++++++++++++--- platform/vpn-egress/README.md | 39 +++++++++++++++++++++++++---- 2 files changed, 77 insertions(+), 8 deletions(-) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index c2df2c5..8e7fe31 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -2253,15 +2253,55 @@ checked before and after arming: `findmnt`-visible mounts are intact and readable. qBittorrent, incidentally, has **no** NFS media mount in this repo — only plex, sonarr, radarr and nextcloud do. +### A fourth defect, created by the stand-up itself: one pinned PIA server cannot serve two gateways + +Prod came up green, and **dev went down.** Not the connection limit — the operator's "same account, +well inside its limit" is true. The vaulted `.ovpn` pins **one** server, so both gateways dialled +it, and PIA handed **both sessions the same tunnel address**. Verbatim from both journals: + +``` +[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198 +PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ... +``` + +Each reconnect steals `10.72.192.7` on that server; the loser then receives nothing, trips +`--ping-restart` after 60s, reconnects, and steals it back. Measured: dev restarting at 22:57:05, +22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC, prod interleaved about a minute off — a ~2-minute +mutual-eviction cycle with no end. Prod's first probes looked green only because they landed +inside an up-window; that is a good reminder that a single green probe is not a stability check. + +**Fix:** a new role var `vpn_egress_gateway_pia_remote` (" ", empty = honour the +.ovpn) rewrites only the `remote` line of the deployed .ovpn, and +`group_vars/vpn_egress_gateway_prod/vars.yml` points prod at `158.173.241.135`. That is another +member of the same PIA Stockholm LB pool — `sweden.privacy.network` resolved from `k3s-prod-3` to +`158.173.241.{34,55,112,135,239}`, and every server in the pool accepts the same credentials and +the same inline CA, so only the address differs. Dev keeps the .ovpn's `.112` untouched, which is +why the override is per-group rather than in the role or a second vaulted .ovpn. A PIA server +address is not a credential, so it sits in plain group_vars next to the vault, reviewable. + +**Standing rule this establishes: one gateway per PIA server.** A third gateway, or a rebuild that +re-pins, must take an unused address from the pool. + +**And the switch handed us the best possible proof of the re-steer fix, unplanned.** Moving prod to +`.135` gave it a genuinely NEW tunnel address on a real reconnect — `10.72.192.7` (ifindex 128) -> +`10.150.192.86` (ifindex 129) — which is exactly the case Cilium does not follow and exactly what +defect 1 was about. The `egress-uplink` label moved to `10.150.192.86-129` by itself, Cilium +re-resolved, and all four vpn-class workloads kept egressing (now `158.173.241.11`). No manual +step, no CEGP edit, no operator. Before this branch that address change would have left prod's +vpn-class egress dead until someone noticed. + ### Left for the operator / not settled here 1. **Re-run `vpn-killswitch` (R-NET-06) against a labelled cluster** and refresh the ledger. It can only run where a node carries the gateway label, and its recovery step is only passable with this branch's re-steer fix. It is an L3 fault-injection test that stops the tunnel on a LIVE cluster, so it was not fired at prod unasked. -2. **Both clusters now exit from the same PIA public IP** (`158.173.241.36`) on one account. Inside - the connection limit, but the two environments are indistinguishable from outside. A separate - prod PIA account is the clean fix if that ever matters. +2. **Decide whether prod gets its own PIA account.** After the server split the two environments do + have distinct exit IPs (dev `158.173.241.36`, prod `158.173.241.11`), so the immediate problem is + solved — but they still share one account, and the per-environment PIA server is now a + hand-maintained value in `group_vars/vpn_egress_gateway_prod/vars.yml` that must not collide with + dev's. A separate prod account removes that coupling permanently. **Verify dev recovered**: dev + was still inside its flap window when this session ended (see the open item below). 3. **Gateway-node loss is still an outage** — Cilium OSS has no egress-gateway failover, and the ADR-0016 supersede drafted on 2026-09-06 is still open. Nothing here changes that; the single-gateway choice is now recorded in the inventory itself, not just in prose. diff --git a/platform/vpn-egress/README.md b/platform/vpn-egress/README.md index 907e29a..b1c9aa1 100644 --- a/platform/vpn-egress/README.md +++ b/platform/vpn-egress/README.md @@ -39,7 +39,8 @@ Three layers, two owners: |---|---|---| | gateway node | `k3s-dev-3` | `k3s-prod-3` | | inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) | -| vpn-class egress IP | `158.173.241.36` (PIA Stockholm) | `158.173.241.36` (PIA Stockholm) | +| PIA server | `158.173.241.112` (the .ovpn's pin) | `158.173.241.135` (group_vars override — see below) | +| vpn-class egress IP | `158.173.241.36` | `158.173.241.11` | | internet-class egress IP | home WAN | `136.51.115.252` (home WAN) | Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three @@ -50,10 +51,38 @@ reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress lis gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also means nothing had been leaking in the meantime. -Both environments currently share one PIA account and, as it happens, one public egress IP. -That is inside the account's connection limit, but it means the two environments are not -distinguishable from the outside; a separate prod PIA account is the clean fix if that ever -matters (see `infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`). +### One PIA account is fine. One pinned PIA *server* is not. + +Standing prod up broke dev, and the reason is not the connection limit. The vaulted `.ovpn` +pins **one** server, so both gateways dialled it — and PIA handed **both sessions the same +tunnel address**. Verbatim from both journals: + +``` +[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198 +PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ... +``` + +Each reconnect steals `10.72.192.7` on that server; the loser receives nothing, trips +`--ping-restart` after 60 s, reconnects and steals it back. Measured cycle: dev restarting at +22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC with prod interleaved about a minute +off — every ~2 minutes, indefinitely. Neither environment had reliable vpn-class egress; prod's +first probes looked green only because they landed inside an up-window. + +**Fix:** `vpn_egress_gateway_pia_remote` in `group_vars/vpn_egress_gateway_prod/vars.yml` +rewrites just the `remote` line, pointing prod at `158.173.241.135` — another member of the +same PIA Stockholm LB pool (`sweden.privacy.network` resolves to +`158.173.241.{34,55,112,135,239}`; every server in it takes the same credentials and CA). Dev +keeps `.112`. + +**So: one gateway per PIA server.** A third gateway, or a rebuild that re-pins a server, must +pick an unused address from the pool. A separate prod PIA account would also solve it — see +`infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`. + +That switch also produced the cleanest possible live proof of the re-steer path, for free: prod's +tunnel address changed from `10.72.192.7` (ifindex 128) to `10.150.192.86` (ifindex 129) — a real +new PIA address on a real reconnect, the exact case Cilium does not follow. The `egress-uplink` +label moved to `10.150.192.86-129` on its own and all four vpn-class workloads kept egressing, +now via `158.173.241.11`. No manual step, no CEGP edit. ## The kill-switch does not do what this file used to claim (measured 2026-09-12)