wip(vpn): gluetun custom-provider mode with operator's PIA .ovpn

PIA-provider server selection is flaky here (random dead IPs); the operator's .ovpn
uses sweden.privacy.network (load-balanced, reliable). Switch to custom mode
(VPN_SERVICE_PROVIDER=custom, OPENVPN_CUSTOM_CONFIG=/config/pia.ovpn), mount the
pia-custom-ovpn Secret, keep creds + kill-switch + tun-mtu 1300. Drops PIA-provider
server list/updater and native PF (Phase-6 concern). Pending the operator-provisioned
pia-custom-ovpn Secret (not merged until it exists).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

DECISIONS-LOG.md

@@ -555,7 +555,27 @@ return traffic) and had me check MTU at each layer. Findings via `ping -M do` sw
   overhead → **pod MTU 1450**. OpenVPN emits ~1500-byte handshake packets → exceed 1450 → dropped (broken
   PMTUD = no ICMP back). This is why HTTPS (MSS-clamped) + small UDP (DNS/NTP) work but OpenVPN doesn't.
 
-**Fix (operator's choice: "shrink OpenVPN MTU in gluetun"):** set gluetun `OPENVPN_MTU=1350` (tun-mtu) and
+### Phase 3 VPN — custom .ovpn mode (the real cause: flaky server selection) (2026-08-23)
+After the MTU red-herrings, packet capture (node ens3 + host wlp4s0, GNOME OFF so it wasn't the host's own
+VPN) proved the pod CAN complete a **full** PIA handshake — a working flow (`50017 → 158.173.241.22:1198`)
+showed 1222-byte cert packets BOTH ways, reaching the node and pod end-to-end through the double NAT
+(Cilium SNAT → libvirt masq → router). It's **intermittent per-server**: gluetun's PIA-provider mode
+random-picks server IPs and many don't answer this cluster's flows (→ TLS timeouts, exit-0 restarts ~every
+2min), while the operator's GNOME client is reliable because its `.ovpn` targets **`sweden.privacy.network`**
+— a load-balanced hostname that always lands on a healthy server. NOT MTU, NOT encryption preset (strong AND
+normal both fail identically once the env name `PRIVATE_INTERNET_ACCESS_OPENVPN_ENCRYPTION_PRESET` was
+corrected), NOT Cilium (policy default-allow, host-fw off, Hubble FORWARDED), NOT a network block.
+
+**Decision (operator's choice):** switch gluetun to **custom-provider mode** feeding the operator's exact
+`.ovpn` (`VPN_SERVICE_PROVIDER=custom`, `OPENVPN_CUSTOM_CONFIG=/config/pia.ovpn`, .ovpn from the
+`pia-custom-ovpn` SOPS Secret; creds still via OPENVPN_USER/PASSWORD; kill-switch kept; tun-mtu 1300 for the
+1450 pod MTU). Dropped the PIA-provider server list + updater initContainer. **Trade-off:** custom mode
+loses gluetun's PIA-native port forwarding (Phase-6/qBittorrent concern) — revisit then. This is an
+implementation choice within ADR-0013's "single shared Gluetun PIA OpenVPN gateway" model (architecture
+unchanged), noted here rather than escalated. Pending operator provisioning of the `.ovpn` Secret.
+
+<!-- superseded strong/normal-preset attempt below, kept for the record -->
+**(superseded) Fix (operator's choice: "shrink OpenVPN MTU in gluetun"):** set gluetun `OPENVPN_MTU=1350` (tun-mtu) and
 `OPENVPN_MSSFIX=1350` so encrypted packets (~1350+52 ≈ 1402) fit under the 1450 pod MTU. Keeps OpenVPN +
 Cilium VXLAN unchanged. Port kept at 8080 (confirmed PIA responds). Re-enabled `vpn-egress`. Values hardcoded
 as quoted strings (int-coercion gotcha). Alternatives not taken: Cilium native routing (pod MTU→1500) or

bootstrap/sops/secret-templates/pia-custom-ovpn.yaml.example

@@ -0,0 +1,28 @@
+# pia-custom-ovpn — the operator's PIA OpenVPN .ovpn file for gluetun custom-provider mode.
+#
+# WHY: gluetun's PIA-provider server selection random-picks server IPs, many of which don't answer
+# this cluster's flows (intermittent TLS timeouts). The downloaded PIA .ovpn targets the load-balanced
+# hostname `sweden.privacy.network`, which always lands on a healthy server (reliable on the host).
+# See DECISIONS-LOG "Phase 3 VPN — custom .ovpn".
+#
+# The .ovpn should use `auth-user-pass` (no inline creds) — gluetun injects OPENVPN_USER/OPENVPN_PASSWORD
+# from the pia-credentials Secret. It contains only the (public) CA cert + config, but we keep it in a
+# SOPS-encrypted Secret for consistency.
+#
+# PROVISION (operator runs SOPS; agents never handle secret values):
+#   1. Generate the Secret manifest straight from your .ovpn file:
+#        kubectl create secret generic pia-custom-ovpn -n vpn-egress \
+#          --from-file=pia.ovpn=/path/to/your-pia.ovpn \
+#          --dry-run=client -o yaml > platform/vpn-egress/pia-custom-ovpn.sops.yaml
+#   2. Encrypt in place (matches .sops.yaml creation_rules for *.sops.yaml):
+#        sops -e -i platform/vpn-egress/pia-custom-ovpn.sops.yaml
+#   3. Commit. Flux + the SOPS decryption provider apply it; gluetun mounts it at /config/pia.ovpn.
+#
+# The generated (pre-encryption) shape looks like:
+apiVersion: v1
+kind: Secret
+metadata:
+  name: pia-custom-ovpn
+  namespace: vpn-egress
+data:
+  pia.ovpn: <base64 of the .ovpn file>   # `kubectl create secret --from-file` fills this in for you

platform/vpn-egress/gluetun-gateway.yaml

@@ -1,13 +1,19 @@
 ---
 # Gluetun — the single shared in-cluster VPN egress gateway (ADR-0013): PIA over OpenVPN-UDP, with
-# Gluetun's built-in kill-switch/firewall (fail-closed), health checks, auto-reconnect, and native
+# Gluetun's built-in kill-switch/firewall (fail-closed), health checks, auto-reconnect. `vpn`-class
-# port forwarding for qBittorrent. `vpn`-class pods are steered through this gateway (see README.md
+# pods are steered through this gateway (see README.md "Egress steering").
-# "Egress steering" — the Cilium-native mechanism, pending live validation with PIA creds).
 #
-# Credentials: the `pia-credentials` SOPS Secret (keys: username, password). Gluetun uses PIA's
+# CUSTOM-PROVIDER mode (DECISIONS-LOG "Phase 3 VPN — custom .ovpn"): gluetun's PIA-provider server
-# username/password (via OPENVPN_USER/OPENVPN_PASSWORD) to authenticate to PIA. Gluetun ships PIA's
+# selection random-picks server IPs, many of which don't answer this cluster's flows (intermittent
-# OpenVPN server configs, so no manual `.ovpn` is needed (ADR-0013 amendment: OpenVPN-UDP — Gluetun
+# TLS timeouts — verified by packet capture: the pod CAN complete a full handshake, but only against
-# does NOT support PIA over WireGuard).
+# responsive servers). The operator's downloaded PIA `.ovpn` targets `sweden.privacy.network` — a
+# load-balanced hostname that always lands on a healthy server (reliable in GNOME on the host). So we
+# feed gluetun that exact `.ovpn`. Trade-off: custom mode loses gluetun's PIA-native port forwarding
+# (needed only at Phase 6 for qBittorrent) — revisit then.
+#
+# Credentials: `pia-credentials` SOPS Secret (username/password) → OPENVPN_USER/OPENVPN_PASSWORD (the
+# .ovpn uses auth-user-pass). Config: `pia-custom-ovpn` SOPS Secret (the .ovpn file), mounted at
+# /config/pia.ovpn. Both operator-provisioned.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -27,49 +33,20 @@ spec:
       labels:
         app: gluetun
     spec:
-      initContainers:
-        # gluetun's BUNDLED PIA server list is STALE (retired 46.246.x IPs → timeouts; current PIA Sweden
-        # is 158.173.x), and UPDATER_PERIOD only refreshes AFTER the tunnel is ready (chicken-and-egg). So
-        # fetch PIA's CURRENT list into the shared /gluetun volume BEFORE the main container starts.
-        - name: update-servers
-          image: qmcgaw/gluetun:v3.41.0
-          args: ["update", "-enduser", "-providers", "private internet access"]
-          resources:
-            requests: { cpu: 25m, memory: 32Mi }
-            limits: { memory: 128Mi }
-          volumeMounts:
-            - name: gluetun-data
-              mountPath: /gluetun
       containers:
         - name: gluetun
-          image: qmcgaw/gluetun:v3.41.0        # pinned; Renovate bumps. PIA is OpenVPN-only in Gluetun
+          image: qmcgaw/gluetun:v3.41.0        # pinned; Renovate bumps.
-                                               # (WireGuard for PIA is unsupported upstream — ADR-0013
-                                               # amendment; thrnz/docker-wireguard-pia is the WG fallback).
           securityContext:
             capabilities:
               add: ["NET_ADMIN"]               # required for the tunnel + firewall
           env:
             - name: VPN_SERVICE_PROVIDER
-              value: "private internet access"
+              value: "custom"                # custom .ovpn (sweden.privacy.network) — see header + DECISIONS-LOG
             - name: VPN_TYPE
-              value: "openvpn"               # OpenVPN-UDP — satisfies "protocol != TCP" (R-NET); ADR-0013 amended
+              value: "openvpn"
-            - name: PRIVATE_INTERNET_ACCESS_OPENVPN_ENCRYPTION_PRESET
+            - name: OPENVPN_CUSTOM_CONFIG
-              value: "normal"                # PIA "normal": RSA-2048 + AES-128-CBC + SHA1, port 1198 (matches the
+              value: "/config/pia.ovpn"      # the operator's PIA .ovpn (pia-custom-ovpn Secret)
-                                             # operator's downloaded lower-encryption .ovpn). Chosen over "strong"
+            # PIA username/password for the .ovpn's auth-user-pass (pia-credentials SOPS Secret):
-                                             # (RSA-4096): the smaller handshake fits the pod's 1450 MTU (Cilium
-                                             # VXLAN) reliably, where RSA-4096 was marginal and wouldn't hold.
-            # Fresh server list comes from the update-servers initContainer (see above). UPDATER_PERIOD then
-            # keeps it fresh for long-running pods (gluetun docs: >=360h to be nice to the provider API):
-            - name: UPDATER_PERIOD
-              value: "720h"
-            # Pod MTU is 1450 (Cilium VXLAN: 1500 − 50 encap; node/host/WAN are 1500). Shrink OpenVPN so
-            # DATA-channel packets fit (the handshake is small so this wasn't the connect blocker, but the
-            # data channel needs it). Hardcoded strings (int-coercion gotcha). See DECISIONS-LOG.
-            - name: OPENVPN_MTU
-              value: "1300"                  # tun-mtu: extra margin under the 1450 pod MTU (data channel)
-            - name: OPENVPN_MSSFIX
-              value: "1300"                  # clamp TCP-in-tunnel to match
-            # PIA credentials from the SOPS secret (Gluetun's PIA OpenVPN creds env):
             - name: OPENVPN_USER
               valueFrom:
                 secretKeyRef:
@@ -80,10 +57,12 @@ spec:
                 secretKeyRef:
                   name: pia-credentials
                   key: password
-            - name: SERVER_REGIONS
+            # Pod MTU is 1450 (Cilium VXLAN: 1500 − 50 encap). Shrink OpenVPN's tun-mtu so data-channel
-              value: "${PIA_REGION}"           # cluster-vars: a PF-CAPABLE PIA region (ADR-0013 PF caveat)
+            # packets fit (the .ovpn likely defaults to 1500). Hardcoded strings (int-coercion gotcha).
-            - name: VPN_PORT_FORWARDING
+            - name: OPENVPN_MTU
-              value: "on"                      # PIA native port forwarding for qBittorrent
+              value: "1300"
+            - name: OPENVPN_MSSFIX
+              value: "1300"
             - name: FIREWALL
               value: "on"                      # KILL-SWITCH: block all egress except the tunnel (fail-closed, R-NET-06)
             - name: FIREWALL_OUTBOUND_SUBNETS
@@ -92,7 +71,7 @@ spec:
               value: "off"                     # DNS via cluster CoreDNS, not Gluetun's DoT
           ports:
             - name: control
-              containerPort: 8000              # Gluetun control API (health + forwarded-port lookup)
+              containerPort: 8000              # Gluetun control API (health + tunnel-state lookup)
           livenessProbe:
             httpGet: { path: /v1/openvpn/status, port: 8000 }   # Gluetun control API health
             initialDelaySeconds: 20
@@ -107,7 +86,10 @@ spec:
             - name: tun
               mountPath: /dev/net/tun
             - name: gluetun-data
-              mountPath: /gluetun            # shared with update-servers initContainer (fresh servers.json)
+              mountPath: /gluetun            # gluetun runtime state
+            - name: custom-ovpn
+              mountPath: /config             # the .ovpn (read-only)
+              readOnly: true
       volumes:
         - name: tun
           hostPath:
@@ -115,9 +97,14 @@ spec:
             type: CharDevice
         - name: gluetun-data
           emptyDir: {}
+        - name: custom-ovpn
+          secret:
+            secretName: pia-custom-ovpn
+            items:
+              - key: pia.ovpn
+                path: pia.ovpn
 ---
-# Control API Service — the port-sync helper (qbittorrent-natmap, README) reads the PIA-assigned
+# Control API Service — used for tunnel-state health/alerting (ADR-0013 "alert on tunnel state").
-# forwarded port from here; also used for tunnel-state health/alerting (ADR-0013 "alert on tunnel state").
 apiVersion: v1
 kind: Service
 metadata:

platform/vpn-egress/kustomization.yaml

@@ -5,3 +5,4 @@ resources:
   - namespace.yaml
   - gluetun-gateway.yaml
   - pia-credentials.sops.yaml
+  - pia-custom-ovpn.sops.yaml   # operator-provisioned: the PIA .ovpn (custom-provider mode). See README.