Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

docs(vpn): record the PIA server collision, and the accidental proof it handed us

Browse Source 
The stand-up broke dev, so the "one gateway per PIA server" rule and the
measurement behind it are now written down where the next person will look:
platform/vpn-egress/README.md and the session's DECISIONS-LOG entry. Both carry
the verbatim journal lines — same server, same pushed `ifconfig 10.72.192.7` on
both gateways — and the flap timestamps, because "same account, inside the
connection limit" is true and is exactly what makes this non-obvious.

Also corrected: the dev/prod table now shows the two environments on different
servers with different exit IPs (dev 158.173.241.36, prod 158.173.241.11), and the
open item about them sharing one public IP is replaced with the real remaining
decision — whether prod gets its own PIA account, which would remove the
hand-maintained per-environment server value entirely.

The unplanned bonus, recorded in both places: moving prod to .135 gave it a
genuinely NEW tunnel address on a real reconnect (10.72.192.7 ifindex 128 ->
10.150.192.86 ifindex 129) — the precise case Cilium does not follow and the one
defect 1 was about. The egress-uplink label moved on its own, Cilium re-resolved,
and all four vpn-class workloads kept egressing with no manual step. Better
evidence than the synthetic address swap used to characterise it, and it cost
nothing to capture.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-12 18:09:31 -05:00
1 parent 37a6c94168
commit 38e4bf1459
2 files changed
+77 -8
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

DECISIONS-LOG.md
+43 -3
View File
@@ -2253,15 +2253,55 @@ checked before and after arming:
`findmnt`-visible mounts are intact and readable. qBittorrent, incidentally, has **no** NFS
media mount in this repo — only plex, sonarr, radarr and nextcloud do.
### A fourth defect, created by the stand-up itself: one pinned PIA server cannot serve two gateways
Prod came up green, and **dev went down.** Not the connection limit — the operator's "same account,
well inside its limit" is true. The vaulted `.ovpn` pins **one** server, so both gateways dialled
it, and PIA handed **both sessions the same tunnel address**. Verbatim from both journals:
```
[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198
PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ...
```
Each reconnect steals `10.72.192.7` on that server; the loser then receives nothing, trips
`--ping-restart` after 60s, reconnects, and steals it back. Measured: dev restarting at 22:57:05,
22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC, prod interleaved about a minute off — a ~2-minute
mutual-eviction cycle with no end. Prod's first probes looked green only because they landed
inside an up-window; that is a good reminder that a single green probe is not a stability check.
**Fix:** a new role var `vpn_egress_gateway_pia_remote` ("<host> <port>", empty = honour the
.ovpn) rewrites only the `remote` line of the deployed .ovpn, and
`group_vars/vpn_egress_gateway_prod/vars.yml` points prod at `158.173.241.135`. That is another
member of the same PIA Stockholm LB pool — `sweden.privacy.network` resolved from `k3s-prod-3` to
`158.173.241.{34,55,112,135,239}`, and every server in the pool accepts the same credentials and
the same inline CA, so only the address differs. Dev keeps the .ovpn's `.112` untouched, which is
why the override is per-group rather than in the role or a second vaulted .ovpn. A PIA server
address is not a credential, so it sits in plain group_vars next to the vault, reviewable.
**Standing rule this establishes: one gateway per PIA server.** A third gateway, or a rebuild that
re-pins, must take an unused address from the pool.
**And the switch handed us the best possible proof of the re-steer fix, unplanned.** Moving prod to
`.135` gave it a genuinely NEW tunnel address on a real reconnect — `10.72.192.7` (ifindex 128) ->
`10.150.192.86` (ifindex 129) — which is exactly the case Cilium does not follow and exactly what
defect 1 was about. The `egress-uplink` label moved to `10.150.192.86-129` by itself, Cilium
re-resolved, and all four vpn-class workloads kept egressing (now `158.173.241.11`). No manual
step, no CEGP edit, no operator. Before this branch that address change would have left prod's
vpn-class egress dead until someone noticed.
### Left for the operator / not settled here
1. **Re-run `vpn-killswitch` (R-NET-06) against a labelled cluster** and refresh the ledger. It can
only run where a node carries the gateway label, and its recovery step is only passable with
this branch's re-steer fix. It is an L3 fault-injection test that stops the tunnel on a LIVE
cluster, so it was not fired at prod unasked.
2. **Both clusters now exit from the same PIA public IP** (`158.173.241.36`) on one account. Inside
the connection limit, but the two environments are indistinguishable from outside. A separate
prod PIA account is the clean fix if that ever matters.
2. **Decide whether prod gets its own PIA account.** After the server split the two environments do
have distinct exit IPs (dev `158.173.241.36`, prod `158.173.241.11`), so the immediate problem is
solved — but they still share one account, and the per-environment PIA server is now a
hand-maintained value in `group_vars/vpn_egress_gateway_prod/vars.yml` that must not collide with
dev's. A separate prod account removes that coupling permanently. **Verify dev recovered**: dev
was still inside its flap window when this session ended (see the open item below).
3. **Gateway-node loss is still an outage** — Cilium OSS has no egress-gateway failover, and the
ADR-0016 supersede drafted on 2026-09-06 is still open. Nothing here changes that; the
single-gateway choice is now recorded in the inventory itself, not just in prose.
platform/vpn-egress/README.md
+34 -5
View File
@@ -39,7 +39,8 @@ Three layers, two owners:
|---|---|---|
| gateway node | `k3s-dev-3` | `k3s-prod-3` |
| inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) |
| vpn-class egress IP | `158.173.241.36` (PIA Stockholm) | `158.173.241.36` (PIA Stockholm) |
| PIA server | `158.173.241.112` (the .ovpn's pin) | `158.173.241.135` (group_vars override — see below) |
| vpn-class egress IP | `158.173.241.36` | `158.173.241.11` |
| internet-class egress IP | home WAN | `136.51.115.252` (home WAN) |
Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three
@@ -50,10 +51,38 @@ reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress lis
gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also
means nothing had been leaking in the meantime.
Both environments currently share one PIA account and, as it happens, one public egress IP.
That is inside the account's connection limit, but it means the two environments are not
distinguishable from the outside; a separate prod PIA account is the clean fix if that ever
matters (see `infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`).
### One PIA account is fine. One pinned PIA *server* is not.
Standing prod up broke dev, and the reason is not the connection limit. The vaulted `.ovpn`
pins **one** server, so both gateways dialled it — and PIA handed **both sessions the same
tunnel address**. Verbatim from both journals:
```
[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198
PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ...
```
Each reconnect steals `10.72.192.7` on that server; the loser receives nothing, trips
`--ping-restart` after 60 s, reconnects and steals it back. Measured cycle: dev restarting at
22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC with prod interleaved about a minute
off — every ~2 minutes, indefinitely. Neither environment had reliable vpn-class egress; prod's
first probes looked green only because they landed inside an up-window.
**Fix:** `vpn_egress_gateway_pia_remote` in `group_vars/vpn_egress_gateway_prod/vars.yml`
rewrites just the `remote` line, pointing prod at `158.173.241.135` — another member of the
same PIA Stockholm LB pool (`sweden.privacy.network` resolves to
`158.173.241.{34,55,112,135,239}`; every server in it takes the same credentials and CA). Dev
keeps `.112`.
**So: one gateway per PIA server.** A third gateway, or a rebuild that re-pins a server, must
pick an unused address from the pool. A separate prod PIA account would also solve it — see
`infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`.
That switch also produced the cleanest possible live proof of the re-steer path, for free: prod's
tunnel address changed from `10.72.192.7` (ifindex 128) to `10.150.192.86` (ifindex 129) — a real
new PIA address on a real reconnect, the exact case Cilium does not follow. The `egress-uplink`
label moved to `10.150.192.86-129` on its own and all four vpn-class workloads kept egressing,
now via `158.173.241.11`. No manual step, no CEGP edit.
## The kill-switch does not do what this file used to claim (measured 2026-09-12)