From 457edb7a385e705d6714dd6d44a8101e4b2e3241 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 22 Aug 2026 23:08:18 -0500 Subject: [PATCH] fix(vpn): enable gluetun server-list updater (bundled list was stale) + re-enable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CORRECTION: the connection failures were gluetun dialing RETIRED PIA IPs (46.246.x = dead → timeouts, which I mis-read as a network block). A fresh PIA .ovpn connects from the same host, and PIA's current SE Stockholm servers (158.173.x) are reachable. Cilium was independently cleared (default-allow, host-fw off, Hubble FORWARDED). Fix: UPDATER_PERIOD=24h so gluetun fetches PIA's current servers; keep OPENVPN_MTU/MSSFIX=1350 for the real Cilium-VXLAN 1450 pod MTU (data channel); PIA default UDP port. Re-enable vpn-egress. Full correction in DECISIONS-LOG. Co-Authored-By: Claude Opus 4.8 --- DECISIONS-LOG.md | 15 +++++++++++++++ clusters/base/cluster-vars.yaml | 4 ++-- platform/kustomization.yaml | 12 +++++------- platform/vpn-egress/gluetun-gateway.yaml | 17 +++++++++++++---- 4 files changed, 35 insertions(+), 13 deletions(-) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index a94931c..51c2026 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -581,3 +581,18 @@ WireGuard/thrnz would NOT help — same PIA IPs are blocked; if switching provid one whose ranges this network permits. Reverted the port/MTU red-herring env from gluetun-gateway.yaml (back to PIA defaults); `vpn-egress` disabled so it doesn't crashloop/alert. **Config is correct and complete — the only blocker is the network.** + +### Phase 3 VPN — CORRECTION: it was gluetun's STALE server list, NOT a network block (2026-08-22) +The "network blackholes PIA" conclusion above was WRONG. The operator installed a fresh PIA `.ovpn` in GNOME +on the same host/network and it connected fine → the network permits PIA. Verified against PIA's live server +list: current SE Stockholm servers are **158.173.241.x** and ARE reachable (tcp/8080 → RST / "connection +refused" = the server answered). The IPs gluetun was dialing (**46.246.x / 212.112.x**) are **retired** +(timeout = dead hosts). So gluetun's **bundled server list was stale**, and my host reachability test used +those dead IPs → false "blackhole." (Cilium was independently ruled out per the operator's request: policy +mode `default` = default-allow, host firewall disabled, no policy selects these pods, and Hubble showed +pod→PIA `to-network FORWARDED`, no drops.) + +**Fix:** gluetun `UPDATER_PERIOD=24h` (fetch PIA's current server list at startup); keep `OPENVPN_MTU=1350` ++ `OPENVPN_MSSFIX=1350` for the real Cilium-VXLAN 1450 pod MTU (data channel); PIA default OpenVPN UDP port. +Re-enabled `vpn-egress`. **Lesson:** when gluetun "can't reach" a provider, suspect a stale bundled server +list FIRST — validate against the provider's live server list before blaming the network/MTU/ports. diff --git a/clusters/base/cluster-vars.yaml b/clusters/base/cluster-vars.yaml index f0c916e..a1773e1 100644 --- a/clusters/base/cluster-vars.yaml +++ b/clusters/base/cluster-vars.yaml @@ -43,8 +43,8 @@ data: # which lacks PF; NOT a US region, which also lack PF) CLUSTER_POD_CIDR: "10.0.0.0/8" # Cilium cluster-pool pod CIDR (pods get 10.x); Gluetun # FIREWALL_OUTBOUND_SUBNETS = allow these to egress via the tunnel - # NOTE: Phase-3 VPN (vpn-egress) is DISABLED — the network blackholes traffic to PIA's IP ranges - # (operator fix needed, not a config issue). Uses PIA's default OpenVPN UDP port. See DECISIONS-LOG. + # NOTE: Phase-3 VPN uses PIA's default OpenVPN UDP port + gluetun's server-list UPDATER (the bundled list + # was stale, dialing retired IPs). Port/region are not cluster-vars (see gluetun-gateway.yaml). See DECISIONS-LOG. # --- identity / auth (ADR-0014) --- KEYCLOAK_REPLICAS: "1" # dev 1 | prod 2 (HA) diff --git a/platform/kustomization.yaml b/platform/kustomization.yaml index e73e609..d7fb3b4 100644 --- a/platform/kustomization.yaml +++ b/platform/kustomization.yaml @@ -17,13 +17,11 @@ resources: - cert-manager # Let's Encrypt DNS-01 wildcard — needs SOPS `cloudflare-api-token` - externaldns # auto DNS records — needs SOPS `cloudflare-api-token` - gateway # Gateway API listeners (ingress; replaces Traefik) - # ---- Phase 3 — Egress / VPN — DISABLED: BLOCKED at the network (operator action needed). Definitive - # finding: this 192.168.2.x WiFi segment BLACKHOLES traffic to PIA's IP ranges — TCP 501/443 to PIA - # time out, UDP + ICMP get no reply across many PIA servers, while Google/Cloudflare work on all. - # It also intercepts DNS/:53. Not MTU, not ports, not the cluster — the config is correct. Fix at - # the network (move to the unfiltered 192.168.1.x LAN / wired, or unblock PIA on the router), then - # re-enable. See DECISIONS-LOG "Phase 3 VPN — network blocks PIA". ---- - # - vpn-egress # re-enable once the network permits PIA (no code change needed) + # ---- Phase 3 — Egress / VPN (PIA OpenVPN-UDP, ADR-0013 amendment). Real root cause of the earlier + # failures: gluetun's bundled PIA server list was STALE (dialing retired 46.246.x IPs → timeouts); + # fix = UPDATER_PERIOD (fetch current servers) + OPENVPN_MTU/MSSFIX 1350 for the Cilium-VXLAN 1450 + # pod MTU. See DECISIONS-LOG "Phase 3 VPN". ---- + - vpn-egress # shared Gluetun PIA/OpenVPN-UDP egress gateway + kill-switch (ADR-0013) # ---- Phase 4 — Observability (creds provisioned via SOPS: grafana-admin, alertmanager-smtp) ---- - monitoring # kube-prometheus-stack + Loki + Alloy + Alertmanager # ---- Later phases: enabled when their phase opens ---- diff --git a/platform/vpn-egress/gluetun-gateway.yaml b/platform/vpn-egress/gluetun-gateway.yaml index 6164f33..ef8f81b 100644 --- a/platform/vpn-egress/gluetun-gateway.yaml +++ b/platform/vpn-egress/gluetun-gateway.yaml @@ -40,10 +40,19 @@ spec: value: "private internet access" - name: VPN_TYPE value: "openvpn" # OpenVPN-UDP — satisfies "protocol != TCP" (R-NET); ADR-0013 amended - # NOTE (2026-08-22): PIA's default OpenVPN UDP (1197) is used. Earlier port/MTU overrides were - # removed — they were chasing symptoms. The real blocker is that this network BLACKHOLES traffic - # to PIA's IP ranges (TCP 501/443 time out, UDP/ICMP no reply — while Google/Cloudflare work). - # Fixable only at the network, not here. See DECISIONS-LOG "Phase 3 VPN — network blocks PIA". + # Root cause of the connection failures: gluetun's BUNDLED PIA server list was STALE — it dialed + # retired IPs (46.246.x → timeouts, which I initially mis-read as a network block). PIA's CURRENT + # Sweden servers are 158.173.x and are reachable; a fresh .ovpn connects fine. Enable the updater + # so gluetun fetches PIA's current server list at startup + periodically: + - name: UPDATER_PERIOD + value: "24h" + # Pod MTU is 1450 (Cilium VXLAN: 1500 − 50 encap; node/host/WAN are 1500). Shrink OpenVPN so + # DATA-channel packets fit (the handshake is small so this wasn't the connect blocker, but the + # data channel needs it). Hardcoded strings (int-coercion gotcha). See DECISIONS-LOG. + - name: OPENVPN_MTU + value: "1350" + - name: OPENVPN_MSSFIX + value: "1350" # PIA credentials from the SOPS secret (Gluetun's PIA OpenVPN creds env): - name: OPENVPN_USER valueFrom: