diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index c2df2c5..8e7fe31 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -2253,15 +2253,55 @@ checked before and after arming: `findmnt`-visible mounts are intact and readable. qBittorrent, incidentally, has **no** NFS media mount in this repo — only plex, sonarr, radarr and nextcloud do. +### A fourth defect, created by the stand-up itself: one pinned PIA server cannot serve two gateways + +Prod came up green, and **dev went down.** Not the connection limit — the operator's "same account, +well inside its limit" is true. The vaulted `.ovpn` pins **one** server, so both gateways dialled +it, and PIA handed **both sessions the same tunnel address**. Verbatim from both journals: + +``` +[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198 +PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ... +``` + +Each reconnect steals `10.72.192.7` on that server; the loser then receives nothing, trips +`--ping-restart` after 60s, reconnects, and steals it back. Measured: dev restarting at 22:57:05, +22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC, prod interleaved about a minute off — a ~2-minute +mutual-eviction cycle with no end. Prod's first probes looked green only because they landed +inside an up-window; that is a good reminder that a single green probe is not a stability check. + +**Fix:** a new role var `vpn_egress_gateway_pia_remote` (" ", empty = honour the +.ovpn) rewrites only the `remote` line of the deployed .ovpn, and +`group_vars/vpn_egress_gateway_prod/vars.yml` points prod at `158.173.241.135`. That is another +member of the same PIA Stockholm LB pool — `sweden.privacy.network` resolved from `k3s-prod-3` to +`158.173.241.{34,55,112,135,239}`, and every server in the pool accepts the same credentials and +the same inline CA, so only the address differs. Dev keeps the .ovpn's `.112` untouched, which is +why the override is per-group rather than in the role or a second vaulted .ovpn. A PIA server +address is not a credential, so it sits in plain group_vars next to the vault, reviewable. + +**Standing rule this establishes: one gateway per PIA server.** A third gateway, or a rebuild that +re-pins, must take an unused address from the pool. + +**And the switch handed us the best possible proof of the re-steer fix, unplanned.** Moving prod to +`.135` gave it a genuinely NEW tunnel address on a real reconnect — `10.72.192.7` (ifindex 128) -> +`10.150.192.86` (ifindex 129) — which is exactly the case Cilium does not follow and exactly what +defect 1 was about. The `egress-uplink` label moved to `10.150.192.86-129` by itself, Cilium +re-resolved, and all four vpn-class workloads kept egressing (now `158.173.241.11`). No manual +step, no CEGP edit, no operator. Before this branch that address change would have left prod's +vpn-class egress dead until someone noticed. + ### Left for the operator / not settled here 1. **Re-run `vpn-killswitch` (R-NET-06) against a labelled cluster** and refresh the ledger. It can only run where a node carries the gateway label, and its recovery step is only passable with this branch's re-steer fix. It is an L3 fault-injection test that stops the tunnel on a LIVE cluster, so it was not fired at prod unasked. -2. **Both clusters now exit from the same PIA public IP** (`158.173.241.36`) on one account. Inside - the connection limit, but the two environments are indistinguishable from outside. A separate - prod PIA account is the clean fix if that ever matters. +2. **Decide whether prod gets its own PIA account.** After the server split the two environments do + have distinct exit IPs (dev `158.173.241.36`, prod `158.173.241.11`), so the immediate problem is + solved — but they still share one account, and the per-environment PIA server is now a + hand-maintained value in `group_vars/vpn_egress_gateway_prod/vars.yml` that must not collide with + dev's. A separate prod account removes that coupling permanently. **Verify dev recovered**: dev + was still inside its flap window when this session ended (see the open item below). 3. **Gateway-node loss is still an outage** — Cilium OSS has no egress-gateway failover, and the ADR-0016 supersede drafted on 2026-09-06 is still open. Nothing here changes that; the single-gateway choice is now recorded in the inventory itself, not just in prose. diff --git a/platform/vpn-egress/README.md b/platform/vpn-egress/README.md index 907e29a..b1c9aa1 100644 --- a/platform/vpn-egress/README.md +++ b/platform/vpn-egress/README.md @@ -39,7 +39,8 @@ Three layers, two owners: |---|---|---| | gateway node | `k3s-dev-3` | `k3s-prod-3` | | inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) | -| vpn-class egress IP | `158.173.241.36` (PIA Stockholm) | `158.173.241.36` (PIA Stockholm) | +| PIA server | `158.173.241.112` (the .ovpn's pin) | `158.173.241.135` (group_vars override — see below) | +| vpn-class egress IP | `158.173.241.36` | `158.173.241.11` | | internet-class egress IP | home WAN | `136.51.115.252` (home WAN) | Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three @@ -50,10 +51,38 @@ reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress lis gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also means nothing had been leaking in the meantime. -Both environments currently share one PIA account and, as it happens, one public egress IP. -That is inside the account's connection limit, but it means the two environments are not -distinguishable from the outside; a separate prod PIA account is the clean fix if that ever -matters (see `infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`). +### One PIA account is fine. One pinned PIA *server* is not. + +Standing prod up broke dev, and the reason is not the connection limit. The vaulted `.ovpn` +pins **one** server, so both gateways dialled it — and PIA handed **both sessions the same +tunnel address**. Verbatim from both journals: + +``` +[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198 +PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ... +``` + +Each reconnect steals `10.72.192.7` on that server; the loser receives nothing, trips +`--ping-restart` after 60 s, reconnects and steals it back. Measured cycle: dev restarting at +22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC with prod interleaved about a minute +off — every ~2 minutes, indefinitely. Neither environment had reliable vpn-class egress; prod's +first probes looked green only because they landed inside an up-window. + +**Fix:** `vpn_egress_gateway_pia_remote` in `group_vars/vpn_egress_gateway_prod/vars.yml` +rewrites just the `remote` line, pointing prod at `158.173.241.135` — another member of the +same PIA Stockholm LB pool (`sweden.privacy.network` resolves to +`158.173.241.{34,55,112,135,239}`; every server in it takes the same credentials and CA). Dev +keeps `.112`. + +**So: one gateway per PIA server.** A third gateway, or a rebuild that re-pins a server, must +pick an unused address from the pool. A separate prod PIA account would also solve it — see +`infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`. + +That switch also produced the cleanest possible live proof of the re-steer path, for free: prod's +tunnel address changed from `10.72.192.7` (ifindex 128) to `10.150.192.86` (ifindex 129) — a real +new PIA address on a real reconnect, the exact case Cilium does not follow. The `egress-uplink` +label moved to `10.150.192.86-129` on its own and all four vpn-class workloads kept egressing, +now via `158.173.241.11`. No manual step, no CEGP edit. ## The kill-switch does not do what this file used to claim (measured 2026-09-12)