From 91aa7acab89a144494b107cefa76137a6375f531 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 23 Aug 2026 00:40:08 -0500 Subject: [PATCH] wip(vpn): gluetun custom-provider mode with operator's PIA .ovpn PIA-provider server selection is flaky here (random dead IPs); the operator's .ovpn uses sweden.privacy.network (load-balanced, reliable). Switch to custom mode (VPN_SERVICE_PROVIDER=custom, OPENVPN_CUSTOM_CONFIG=/config/pia.ovpn), mount the pia-custom-ovpn Secret, keep creds + kill-switch + tun-mtu 1300. Drops PIA-provider server list/updater and native PF (Phase-6 concern). Pending the operator-provisioned pia-custom-ovpn Secret (not merged until it exists). Co-Authored-By: Claude Opus 4.8 --- DECISIONS-LOG.md | 22 ++++- .../pia-custom-ovpn.yaml.example | 28 ++++++ platform/vpn-egress/gluetun-gateway.yaml | 87 ++++++++----------- platform/vpn-egress/kustomization.yaml | 1 + 4 files changed, 87 insertions(+), 51 deletions(-) create mode 100644 bootstrap/sops/secret-templates/pia-custom-ovpn.yaml.example diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 51c2026..df4ad65 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -555,7 +555,27 @@ return traffic) and had me check MTU at each layer. Findings via `ping -M do` sw overhead → **pod MTU 1450**. OpenVPN emits ~1500-byte handshake packets → exceed 1450 → dropped (broken PMTUD = no ICMP back). This is why HTTPS (MSS-clamped) + small UDP (DNS/NTP) work but OpenVPN doesn't. -**Fix (operator's choice: "shrink OpenVPN MTU in gluetun"):** set gluetun `OPENVPN_MTU=1350` (tun-mtu) and +### Phase 3 VPN — custom .ovpn mode (the real cause: flaky server selection) (2026-08-23) +After the MTU red-herrings, packet capture (node ens3 + host wlp4s0, GNOME OFF so it wasn't the host's own +VPN) proved the pod CAN complete a **full** PIA handshake — a working flow (`50017 → 158.173.241.22:1198`) +showed 1222-byte cert packets BOTH ways, reaching the node and pod end-to-end through the double NAT +(Cilium SNAT → libvirt masq → router). It's **intermittent per-server**: gluetun's PIA-provider mode +random-picks server IPs and many don't answer this cluster's flows (→ TLS timeouts, exit-0 restarts ~every +2min), while the operator's GNOME client is reliable because its `.ovpn` targets **`sweden.privacy.network`** +— a load-balanced hostname that always lands on a healthy server. NOT MTU, NOT encryption preset (strong AND +normal both fail identically once the env name `PRIVATE_INTERNET_ACCESS_OPENVPN_ENCRYPTION_PRESET` was +corrected), NOT Cilium (policy default-allow, host-fw off, Hubble FORWARDED), NOT a network block. + +**Decision (operator's choice):** switch gluetun to **custom-provider mode** feeding the operator's exact +`.ovpn` (`VPN_SERVICE_PROVIDER=custom`, `OPENVPN_CUSTOM_CONFIG=/config/pia.ovpn`, .ovpn from the +`pia-custom-ovpn` SOPS Secret; creds still via OPENVPN_USER/PASSWORD; kill-switch kept; tun-mtu 1300 for the +1450 pod MTU). Dropped the PIA-provider server list + updater initContainer. **Trade-off:** custom mode +loses gluetun's PIA-native port forwarding (Phase-6/qBittorrent concern) — revisit then. This is an +implementation choice within ADR-0013's "single shared Gluetun PIA OpenVPN gateway" model (architecture +unchanged), noted here rather than escalated. Pending operator provisioning of the `.ovpn` Secret. + + +**(superseded) Fix (operator's choice: "shrink OpenVPN MTU in gluetun"):** set gluetun `OPENVPN_MTU=1350` (tun-mtu) and `OPENVPN_MSSFIX=1350` so encrypted packets (~1350+52 ≈ 1402) fit under the 1450 pod MTU. Keeps OpenVPN + Cilium VXLAN unchanged. Port kept at 8080 (confirmed PIA responds). Re-enabled `vpn-egress`. Values hardcoded as quoted strings (int-coercion gotcha). Alternatives not taken: Cilium native routing (pod MTU→1500) or diff --git a/bootstrap/sops/secret-templates/pia-custom-ovpn.yaml.example b/bootstrap/sops/secret-templates/pia-custom-ovpn.yaml.example new file mode 100644 index 0000000..589788f --- /dev/null +++ b/bootstrap/sops/secret-templates/pia-custom-ovpn.yaml.example @@ -0,0 +1,28 @@ +# pia-custom-ovpn — the operator's PIA OpenVPN .ovpn file for gluetun custom-provider mode. +# +# WHY: gluetun's PIA-provider server selection random-picks server IPs, many of which don't answer +# this cluster's flows (intermittent TLS timeouts). The downloaded PIA .ovpn targets the load-balanced +# hostname `sweden.privacy.network`, which always lands on a healthy server (reliable on the host). +# See DECISIONS-LOG "Phase 3 VPN — custom .ovpn". +# +# The .ovpn should use `auth-user-pass` (no inline creds) — gluetun injects OPENVPN_USER/OPENVPN_PASSWORD +# from the pia-credentials Secret. It contains only the (public) CA cert + config, but we keep it in a +# SOPS-encrypted Secret for consistency. +# +# PROVISION (operator runs SOPS; agents never handle secret values): +# 1. Generate the Secret manifest straight from your .ovpn file: +# kubectl create secret generic pia-custom-ovpn -n vpn-egress \ +# --from-file=pia.ovpn=/path/to/your-pia.ovpn \ +# --dry-run=client -o yaml > platform/vpn-egress/pia-custom-ovpn.sops.yaml +# 2. Encrypt in place (matches .sops.yaml creation_rules for *.sops.yaml): +# sops -e -i platform/vpn-egress/pia-custom-ovpn.sops.yaml +# 3. Commit. Flux + the SOPS decryption provider apply it; gluetun mounts it at /config/pia.ovpn. +# +# The generated (pre-encryption) shape looks like: +apiVersion: v1 +kind: Secret +metadata: + name: pia-custom-ovpn + namespace: vpn-egress +data: + pia.ovpn: # `kubectl create secret --from-file` fills this in for you diff --git a/platform/vpn-egress/gluetun-gateway.yaml b/platform/vpn-egress/gluetun-gateway.yaml index 92f95dd..269748a 100644 --- a/platform/vpn-egress/gluetun-gateway.yaml +++ b/platform/vpn-egress/gluetun-gateway.yaml @@ -1,13 +1,19 @@ --- # Gluetun — the single shared in-cluster VPN egress gateway (ADR-0013): PIA over OpenVPN-UDP, with -# Gluetun's built-in kill-switch/firewall (fail-closed), health checks, auto-reconnect, and native -# port forwarding for qBittorrent. `vpn`-class pods are steered through this gateway (see README.md -# "Egress steering" — the Cilium-native mechanism, pending live validation with PIA creds). +# Gluetun's built-in kill-switch/firewall (fail-closed), health checks, auto-reconnect. `vpn`-class +# pods are steered through this gateway (see README.md "Egress steering"). # -# Credentials: the `pia-credentials` SOPS Secret (keys: username, password). Gluetun uses PIA's -# username/password (via OPENVPN_USER/OPENVPN_PASSWORD) to authenticate to PIA. Gluetun ships PIA's -# OpenVPN server configs, so no manual `.ovpn` is needed (ADR-0013 amendment: OpenVPN-UDP — Gluetun -# does NOT support PIA over WireGuard). +# CUSTOM-PROVIDER mode (DECISIONS-LOG "Phase 3 VPN — custom .ovpn"): gluetun's PIA-provider server +# selection random-picks server IPs, many of which don't answer this cluster's flows (intermittent +# TLS timeouts — verified by packet capture: the pod CAN complete a full handshake, but only against +# responsive servers). The operator's downloaded PIA `.ovpn` targets `sweden.privacy.network` — a +# load-balanced hostname that always lands on a healthy server (reliable in GNOME on the host). So we +# feed gluetun that exact `.ovpn`. Trade-off: custom mode loses gluetun's PIA-native port forwarding +# (needed only at Phase 6 for qBittorrent) — revisit then. +# +# Credentials: `pia-credentials` SOPS Secret (username/password) → OPENVPN_USER/OPENVPN_PASSWORD (the +# .ovpn uses auth-user-pass). Config: `pia-custom-ovpn` SOPS Secret (the .ovpn file), mounted at +# /config/pia.ovpn. Both operator-provisioned. apiVersion: apps/v1 kind: Deployment metadata: @@ -27,49 +33,20 @@ spec: labels: app: gluetun spec: - initContainers: - # gluetun's BUNDLED PIA server list is STALE (retired 46.246.x IPs → timeouts; current PIA Sweden - # is 158.173.x), and UPDATER_PERIOD only refreshes AFTER the tunnel is ready (chicken-and-egg). So - # fetch PIA's CURRENT list into the shared /gluetun volume BEFORE the main container starts. - - name: update-servers - image: qmcgaw/gluetun:v3.41.0 - args: ["update", "-enduser", "-providers", "private internet access"] - resources: - requests: { cpu: 25m, memory: 32Mi } - limits: { memory: 128Mi } - volumeMounts: - - name: gluetun-data - mountPath: /gluetun containers: - name: gluetun - image: qmcgaw/gluetun:v3.41.0 # pinned; Renovate bumps. PIA is OpenVPN-only in Gluetun - # (WireGuard for PIA is unsupported upstream — ADR-0013 - # amendment; thrnz/docker-wireguard-pia is the WG fallback). + image: qmcgaw/gluetun:v3.41.0 # pinned; Renovate bumps. securityContext: capabilities: add: ["NET_ADMIN"] # required for the tunnel + firewall env: - name: VPN_SERVICE_PROVIDER - value: "private internet access" + value: "custom" # custom .ovpn (sweden.privacy.network) — see header + DECISIONS-LOG - name: VPN_TYPE - value: "openvpn" # OpenVPN-UDP — satisfies "protocol != TCP" (R-NET); ADR-0013 amended - - name: PRIVATE_INTERNET_ACCESS_OPENVPN_ENCRYPTION_PRESET - value: "normal" # PIA "normal": RSA-2048 + AES-128-CBC + SHA1, port 1198 (matches the - # operator's downloaded lower-encryption .ovpn). Chosen over "strong" - # (RSA-4096): the smaller handshake fits the pod's 1450 MTU (Cilium - # VXLAN) reliably, where RSA-4096 was marginal and wouldn't hold. - # Fresh server list comes from the update-servers initContainer (see above). UPDATER_PERIOD then - # keeps it fresh for long-running pods (gluetun docs: >=360h to be nice to the provider API): - - name: UPDATER_PERIOD - value: "720h" - # Pod MTU is 1450 (Cilium VXLAN: 1500 − 50 encap; node/host/WAN are 1500). Shrink OpenVPN so - # DATA-channel packets fit (the handshake is small so this wasn't the connect blocker, but the - # data channel needs it). Hardcoded strings (int-coercion gotcha). See DECISIONS-LOG. - - name: OPENVPN_MTU - value: "1300" # tun-mtu: extra margin under the 1450 pod MTU (data channel) - - name: OPENVPN_MSSFIX - value: "1300" # clamp TCP-in-tunnel to match - # PIA credentials from the SOPS secret (Gluetun's PIA OpenVPN creds env): + value: "openvpn" + - name: OPENVPN_CUSTOM_CONFIG + value: "/config/pia.ovpn" # the operator's PIA .ovpn (pia-custom-ovpn Secret) + # PIA username/password for the .ovpn's auth-user-pass (pia-credentials SOPS Secret): - name: OPENVPN_USER valueFrom: secretKeyRef: @@ -80,10 +57,12 @@ spec: secretKeyRef: name: pia-credentials key: password - - name: SERVER_REGIONS - value: "${PIA_REGION}" # cluster-vars: a PF-CAPABLE PIA region (ADR-0013 PF caveat) - - name: VPN_PORT_FORWARDING - value: "on" # PIA native port forwarding for qBittorrent + # Pod MTU is 1450 (Cilium VXLAN: 1500 − 50 encap). Shrink OpenVPN's tun-mtu so data-channel + # packets fit (the .ovpn likely defaults to 1500). Hardcoded strings (int-coercion gotcha). + - name: OPENVPN_MTU + value: "1300" + - name: OPENVPN_MSSFIX + value: "1300" - name: FIREWALL value: "on" # KILL-SWITCH: block all egress except the tunnel (fail-closed, R-NET-06) - name: FIREWALL_OUTBOUND_SUBNETS @@ -92,7 +71,7 @@ spec: value: "off" # DNS via cluster CoreDNS, not Gluetun's DoT ports: - name: control - containerPort: 8000 # Gluetun control API (health + forwarded-port lookup) + containerPort: 8000 # Gluetun control API (health + tunnel-state lookup) livenessProbe: httpGet: { path: /v1/openvpn/status, port: 8000 } # Gluetun control API health initialDelaySeconds: 20 @@ -107,7 +86,10 @@ spec: - name: tun mountPath: /dev/net/tun - name: gluetun-data - mountPath: /gluetun # shared with update-servers initContainer (fresh servers.json) + mountPath: /gluetun # gluetun runtime state + - name: custom-ovpn + mountPath: /config # the .ovpn (read-only) + readOnly: true volumes: - name: tun hostPath: @@ -115,9 +97,14 @@ spec: type: CharDevice - name: gluetun-data emptyDir: {} + - name: custom-ovpn + secret: + secretName: pia-custom-ovpn + items: + - key: pia.ovpn + path: pia.ovpn --- -# Control API Service — the port-sync helper (qbittorrent-natmap, README) reads the PIA-assigned -# forwarded port from here; also used for tunnel-state health/alerting (ADR-0013 "alert on tunnel state"). +# Control API Service — used for tunnel-state health/alerting (ADR-0013 "alert on tunnel state"). apiVersion: v1 kind: Service metadata: diff --git a/platform/vpn-egress/kustomization.yaml b/platform/vpn-egress/kustomization.yaml index ea80163..85bc449 100644 --- a/platform/vpn-egress/kustomization.yaml +++ b/platform/vpn-egress/kustomization.yaml @@ -5,3 +5,4 @@ resources: - namespace.yaml - gluetun-gateway.yaml - pia-credentials.sops.yaml + - pia-custom-ovpn.sops.yaml # operator-provisioned: the PIA .ovpn (custom-provider mode). See README.