Merge feature/vpn-custom-ovpn into main

This commit is contained in:
jakman85 committed 2026-08-23 00:45:39 -05:00
commit 6c34fb636d
5 files changed
+115 -51

No files matched your search

+21 -1
View File
@@ -555,7 +555,27 @@ return traffic) and had me check MTU at each layer. Findings via `ping -M do` sw
overhead → **pod MTU 1450**. OpenVPN emits ~1500-byte handshake packets → exceed 1450 → dropped (broken
PMTUD = no ICMP back). This is why HTTPS (MSS-clamped) + small UDP (DNS/NTP) work but OpenVPN doesn't.
**Fix (operator's choice: "shrink OpenVPN MTU in gluetun"):** set gluetun `OPENVPN_MTU=1350` (tun-mtu) and
### Phase 3 VPN — custom .ovpn mode (the real cause: flaky server selection) (2026-08-23)
After the MTU red-herrings, packet capture (node ens3 + host wlp4s0, GNOME OFF so it wasn't the host's own
VPN) proved the pod CAN complete a **full** PIA handshake — a working flow (`50017 → 158.173.241.22:1198`)
showed 1222-byte cert packets BOTH ways, reaching the node and pod end-to-end through the double NAT
(Cilium SNAT → libvirt masq → router). It's **intermittent per-server**: gluetun's PIA-provider mode
random-picks server IPs and many don't answer this cluster's flows (→ TLS timeouts, exit-0 restarts ~every
2min), while the operator's GNOME client is reliable because its `.ovpn` targets **`sweden.privacy.network`**
— a load-balanced hostname that always lands on a healthy server. NOT MTU, NOT encryption preset (strong AND
normal both fail identically once the env name `PRIVATE_INTERNET_ACCESS_OPENVPN_ENCRYPTION_PRESET` was
corrected), NOT Cilium (policy default-allow, host-fw off, Hubble FORWARDED), NOT a network block.
**Decision (operator's choice):** switch gluetun to **custom-provider mode** feeding the operator's exact
`.ovpn` (`VPN_SERVICE_PROVIDER=custom`, `OPENVPN_CUSTOM_CONFIG=/config/pia.ovpn`, .ovpn from the
`pia-custom-ovpn` SOPS Secret; creds still via OPENVPN_USER/PASSWORD; kill-switch kept; tun-mtu 1300 for the
1450 pod MTU). Dropped the PIA-provider server list + updater initContainer. **Trade-off:** custom mode
loses gluetun's PIA-native port forwarding (Phase-6/qBittorrent concern) — revisit then. This is an
implementation choice within ADR-0013's "single shared Gluetun PIA OpenVPN gateway" model (architecture
unchanged), noted here rather than escalated. Pending operator provisioning of the `.ovpn` Secret.
<!-- superseded strong/normal-preset attempt below, kept for the record -->
**(superseded) Fix (operator's choice: "shrink OpenVPN MTU in gluetun"):** set gluetun `OPENVPN_MTU=1350` (tun-mtu) and
`OPENVPN_MSSFIX=1350` so encrypted packets (~1350+52 ≈ 1402) fit under the 1450 pod MTU. Keeps OpenVPN +
Cilium VXLAN unchanged. Port kept at 8080 (confirmed PIA responds). Re-enabled `vpn-egress`. Values hardcoded
as quoted strings (int-coercion gotcha). Alternatives not taken: Cilium native routing (pod MTU→1500) or
@@ -0,0 +1,28 @@
# pia-custom-ovpn — the operator's PIA OpenVPN .ovpn file for gluetun custom-provider mode.
#
# WHY: gluetun's PIA-provider server selection random-picks server IPs, many of which don't answer
# this cluster's flows (intermittent TLS timeouts). The downloaded PIA .ovpn targets the load-balanced
# hostname `sweden.privacy.network`, which always lands on a healthy server (reliable on the host).
# See DECISIONS-LOG "Phase 3 VPN — custom .ovpn".
#
# The .ovpn should use `auth-user-pass` (no inline creds) — gluetun injects OPENVPN_USER/OPENVPN_PASSWORD
# from the pia-credentials Secret. It contains only the (public) CA cert + config, but we keep it in a
# SOPS-encrypted Secret for consistency.
#
# PROVISION (operator runs SOPS; agents never handle secret values):
# 1. Generate the Secret manifest straight from your .ovpn file:
# kubectl create secret generic pia-custom-ovpn -n vpn-egress \
# --from-file=pia.ovpn=/path/to/your-pia.ovpn \
# --dry-run=client -o yaml > platform/vpn-egress/pia-custom-ovpn.sops.yaml
# 2. Encrypt in place (matches .sops.yaml creation_rules for *.sops.yaml):
# sops -e -i platform/vpn-egress/pia-custom-ovpn.sops.yaml
# 3. Commit. Flux + the SOPS decryption provider apply it; gluetun mounts it at /config/pia.ovpn.
#
# The generated (pre-encryption) shape looks like:
apiVersion: v1
kind: Secret
metadata:
name: pia-custom-ovpn
namespace: vpn-egress
data:
pia.ovpn: <base64 of the .ovpn file> # `kubectl create secret --from-file` fills this in for you
+37 -50
View File
@@ -1,13 +1,19 @@
---
# Gluetun — the single shared in-cluster VPN egress gateway (ADR-0013): PIA over OpenVPN-UDP, with
# Gluetun's built-in kill-switch/firewall (fail-closed), health checks, auto-reconnect, and native
# port forwarding for qBittorrent. `vpn`-class pods are steered through this gateway (see README.md
# "Egress steering" — the Cilium-native mechanism, pending live validation with PIA creds).
# Gluetun's built-in kill-switch/firewall (fail-closed), health checks, auto-reconnect. `vpn`-class
# pods are steered through this gateway (see README.md "Egress steering").
#
# Credentials: the `pia-credentials` SOPS Secret (keys: username, password). Gluetun uses PIA's
# username/password (via OPENVPN_USER/OPENVPN_PASSWORD) to authenticate to PIA. Gluetun ships PIA's
# OpenVPN server configs, so no manual `.ovpn` is needed (ADR-0013 amendment: OpenVPN-UDP — Gluetun
# does NOT support PIA over WireGuard).
# CUSTOM-PROVIDER mode (DECISIONS-LOG "Phase 3 VPN — custom .ovpn"): gluetun's PIA-provider server
# selection random-picks server IPs, many of which don't answer this cluster's flows (intermittent
# TLS timeouts — verified by packet capture: the pod CAN complete a full handshake, but only against
# responsive servers). The operator's downloaded PIA `.ovpn` targets `sweden.privacy.network` — a
# load-balanced hostname that always lands on a healthy server (reliable in GNOME on the host). So we
# feed gluetun that exact `.ovpn`. Trade-off: custom mode loses gluetun's PIA-native port forwarding
# (needed only at Phase 6 for qBittorrent) — revisit then.
#
# Credentials: `pia-credentials` SOPS Secret (username/password) → OPENVPN_USER/OPENVPN_PASSWORD (the
# .ovpn uses auth-user-pass). Config: `pia-custom-ovpn` SOPS Secret (the .ovpn file), mounted at
# /config/pia.ovpn. Both operator-provisioned.
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -27,49 +33,20 @@ spec:
labels:
app: gluetun
spec:
initContainers:
# gluetun's BUNDLED PIA server list is STALE (retired 46.246.x IPs → timeouts; current PIA Sweden
# is 158.173.x), and UPDATER_PERIOD only refreshes AFTER the tunnel is ready (chicken-and-egg). So
# fetch PIA's CURRENT list into the shared /gluetun volume BEFORE the main container starts.
- name: update-servers
image: qmcgaw/gluetun:v3.41.0
args: ["update", "-enduser", "-providers", "private internet access"]
resources:
requests: { cpu: 25m, memory: 32Mi }
limits: { memory: 128Mi }
volumeMounts:
- name: gluetun-data
mountPath: /gluetun
containers:
- name: gluetun
image: qmcgaw/gluetun:v3.41.0 # pinned; Renovate bumps. PIA is OpenVPN-only in Gluetun
# (WireGuard for PIA is unsupported upstream — ADR-0013
# amendment; thrnz/docker-wireguard-pia is the WG fallback).
image: qmcgaw/gluetun:v3.41.0 # pinned; Renovate bumps.
securityContext:
capabilities:
add: ["NET_ADMIN"] # required for the tunnel + firewall
env:
- name: VPN_SERVICE_PROVIDER
value: "private internet access"
value: "custom" # custom .ovpn (sweden.privacy.network) — see header + DECISIONS-LOG
- name: VPN_TYPE
value: "openvpn" # OpenVPN-UDP — satisfies "protocol != TCP" (R-NET); ADR-0013 amended
- name: PRIVATE_INTERNET_ACCESS_OPENVPN_ENCRYPTION_PRESET
value: "normal" # PIA "normal": RSA-2048 + AES-128-CBC + SHA1, port 1198 (matches the
# operator's downloaded lower-encryption .ovpn). Chosen over "strong"
# (RSA-4096): the smaller handshake fits the pod's 1450 MTU (Cilium
# VXLAN) reliably, where RSA-4096 was marginal and wouldn't hold.
# Fresh server list comes from the update-servers initContainer (see above). UPDATER_PERIOD then
# keeps it fresh for long-running pods (gluetun docs: >=360h to be nice to the provider API):
- name: UPDATER_PERIOD
value: "720h"
# Pod MTU is 1450 (Cilium VXLAN: 1500 − 50 encap; node/host/WAN are 1500). Shrink OpenVPN so
# DATA-channel packets fit (the handshake is small so this wasn't the connect blocker, but the
# data channel needs it). Hardcoded strings (int-coercion gotcha). See DECISIONS-LOG.
- name: OPENVPN_MTU
value: "1300" # tun-mtu: extra margin under the 1450 pod MTU (data channel)
- name: OPENVPN_MSSFIX
value: "1300" # clamp TCP-in-tunnel to match
# PIA credentials from the SOPS secret (Gluetun's PIA OpenVPN creds env):
value: "openvpn"
- name: OPENVPN_CUSTOM_CONFIG
value: "/config/pia.ovpn" # the operator's PIA .ovpn (pia-custom-ovpn Secret)
# PIA username/password for the .ovpn's auth-user-pass (pia-credentials SOPS Secret):
- name: OPENVPN_USER
valueFrom:
secretKeyRef:
@@ -80,10 +57,12 @@ spec:
secretKeyRef:
name: pia-credentials
key: password
- name: SERVER_REGIONS
value: "${PIA_REGION}" # cluster-vars: a PF-CAPABLE PIA region (ADR-0013 PF caveat)
- name: VPN_PORT_FORWARDING
value: "on" # PIA native port forwarding for qBittorrent
# Pod MTU is 1450 (Cilium VXLAN: 1500 − 50 encap). Shrink OpenVPN's tun-mtu so data-channel
# packets fit (the .ovpn likely defaults to 1500). Hardcoded strings (int-coercion gotcha).
- name: OPENVPN_MTU
value: "1300"
- name: OPENVPN_MSSFIX
value: "1300"
- name: FIREWALL
value: "on" # KILL-SWITCH: block all egress except the tunnel (fail-closed, R-NET-06)
- name: FIREWALL_OUTBOUND_SUBNETS
@@ -92,7 +71,7 @@ spec:
value: "off" # DNS via cluster CoreDNS, not Gluetun's DoT
ports:
- name: control
containerPort: 8000 # Gluetun control API (health + forwarded-port lookup)
containerPort: 8000 # Gluetun control API (health + tunnel-state lookup)
livenessProbe:
httpGet: { path: /v1/openvpn/status, port: 8000 } # Gluetun control API health
initialDelaySeconds: 20
@@ -107,7 +86,10 @@ spec:
- name: tun
mountPath: /dev/net/tun
- name: gluetun-data
mountPath: /gluetun # shared with update-servers initContainer (fresh servers.json)
mountPath: /gluetun # gluetun runtime state
- name: custom-ovpn
mountPath: /config # the .ovpn (read-only)
readOnly: true
volumes:
- name: tun
hostPath:
@@ -115,9 +97,14 @@ spec:
type: CharDevice
- name: gluetun-data
emptyDir: {}
- name: custom-ovpn
secret:
secretName: pia-custom-ovpn
items:
- key: pia.ovpn
path: pia.ovpn
---
# Control API Service — the port-sync helper (qbittorrent-natmap, README) reads the PIA-assigned
# forwarded port from here; also used for tunnel-state health/alerting (ADR-0013 "alert on tunnel state").
# Control API Service — used for tunnel-state health/alerting (ADR-0013 "alert on tunnel state").
apiVersion: v1
kind: Service
metadata:
+1
View File
@@ -5,3 +5,4 @@ resources:
- namespace.yaml
- gluetun-gateway.yaml
- pia-credentials.sops.yaml
- pia-custom-ovpn.sops.yaml # operator-provisioned: the PIA .ovpn (custom-provider mode). See README.
@@ -0,0 +1,28 @@
apiVersion: v1
data:
sweden-aes-128-cbc-udp-dns.ovpn: ENC[AES256_GCM,data:C0dM2ULhPgU9w2YkP8046Zl2iew+PzTYhSJ9VyaiDTUnSDX6s+Aw0/XzQJ7j+YCcENEj+/kiqjJXK+APyLU7F7ngGKwuNH7H9bkQxVYWCvbZEVqg7fVLvhCotgF/ICqtbSAA2YiAuymR5gGJ1x16Ql+Lm/pyxiKUK8xKZ2bwGGiUfuG5rqYtEmmn3KzCst+acBw7CbuAAvDQelipLzLG/TXQMjsZqq7zUGkEjA8zBNRYVacHC+whN1V56jjC6IMlkolLD1LD0KN9Htv6R+Mwvz79nvNxFspYQMJa57OzarW6C73oJaNK0j0So3spb5JKQDHOYUBRb1esRn5TZUT0F8foywD/0TwHKePfHItW4sewzZIukGmZ9i5rHuuc6Y7QrGXdQ5uPwScNaXFdBa0k9V6VO5YGc6LNZf0ftDgX0M8KzyzBakmqzz5xN0+ZoI07kXHcrgabXrONToVh28lnFj84IgmgT/MnHgJE0L0ChaFtkQhiEihCP+V/hJuNXx5Kc63z7kl5Wvy5IaNsrXM2jyBI9ecOpXn3qOeOUI/o1r33WZe4wAbEFz8ECSB6MLi8WDPg/aLBpPbaKRW5d22k4awJ6peGC1IUYoDLwFtrr1Sc5/OmTuCGlVWjFyApEkSbhnj4q/volQEdbzS/P+t4yVdl4aAQBvP9GaAsn+R1fT4Mm5rOehbtepUHjcV93VaTZ3RQv7j0/QlOt5qUFjGke/6ttux6thKo6Oi0Kv6jLujd8lF8rBS92v0BXHH4k3N6ucOcJkkpP+KjixGUOYNY59AJJE6kbCrpsh05LiOqNMqNPs5EUP6QGWsUtCxewiZz4EEOgV63wm7xrUm1gHlVpIRfuBjgMtfngGf1wXjqCNu/M7+i1t38dO04jHMCIGC3YHzw2Ze95+5QCiCJCKTJdRt6k85XJjHjvTHnXDdvm2zrVKnngqUvemp6hL+JjUIqab576JBGzol6oTuT4F5rrRA7jGPiCAw10LXw8Z9Fv3Qi/lYa372lRIU+0kOdC9YXmqlCFkH8EoW8PsT5RrFpZ1CrtlZPw+Ifqg+B5AMMlWRmQlfKjRM2bTx7Qjgy8C5AWlEugTXAtjiFAtbpnf3h5Kr1ap1iK4AiSGpZSR7319GH4ztE4TnPDb5PM+g4ZnNp/KYmTe4HL1nM3iUvi5q8CQwwYv7Ucv1n0ctC//0bXQnakxdkk6K9BFGIfPRvRPGnQe1vNIwSrksxEWl9czVMEFoh8kCwqVNys2BD8IiP5d40UUV4+jphi18mdAlHXOXWCxW6SOYk5OIIaI12BB6Nz04LVBJevaGQ34CjkBIFGhqguZlovw6zt6bVr8FARZ2ARGwejRJeVCx5LcQrcWqj66TqXE3DqT3z8pd82Lzxem3d0PmYNRJ/O47iXkA1rVIACETaCYB/aBGXretnWI8LE7mRh6NqYF5KDg9F2WSzYvdfPE9ab+pcjinDeMNYvV5x1RgfGECrIErLzlYF1OpO1a+xrIr4Fgwct2oydlqh4k0w7MrGicyf3ptLleeXkMvTAHYoi1Jn9JslbjGTFNkmVFIvt+97B0NEgHnJPDazQBcYIqyWFPvckavm43j6/NnUle4N1Uw3aoPqz0zFrWfzP3sKh+YNBOwhbFdS1v8C08KdXYawVUfmzc76i9aGeg/DCo//WzvtvhSmZ/q6ki8VxQ58oijIZlwN6K0Xx2U3Ifmf6FZcNVnqy5Artu3S5M1rKqUmflLzBuUXT7f/pmgXoGvt+EQXh8YD7Rj/Hib5xwnY5knN26qpNakqc5aN0am6TdNXbAzStdQade3IPLw0zsIRLze0O7FPQwW1B+OQXExuaNWhPhcXMz/L2AGcgxbtfQ/rTM50pC4VEmXxeyZa7tOlIgzukHVxx3cyLxSpobh3ZXXgZVC0c3p5cM/BObnDEXS+4s9RkKSykuHcx0pblt4qnheY4tNS6iSnAe21aEEAUBns775EvSGw5hqz9GwcAFgKgKApjx22gEKqI89k+AyK5eDlW44xPWzm6WidWqiS2Fv+pbA+1DPz+loiYErRK5k61HNDoZzU2rF0cfa1GP2tnOKL1/jI95xJpmX7WGYOgnvrlel0w/SlJd3mSiF1BVV/vSbcL+oCZE/7sEz+Q2K4oewGo7JJyMoMzJrp8oiokKrS6wYhZs6iNDYzpJkTaYd9WcgMyutR19c2EIvtIzJzzmOjdp8eP123cWgv3P/KNrDJUOsYLbCvm82Ne16c5+uoXOSx+q8KvujKr0mt71lJy7sklHd18yanH9z+bbGn/RIMhQEHSkpiPEyitUMqnkyNZogNREpnuM/zVaVbkR7ff4UHSd6jqLpkpd3Y61HdOUN8rJQGMzZuG9h7Dg3cygaqPJbnF5WlRCRcKOfR8t3duhD/7tBKR0FVybHxZeL65fhEo0PAvTArkYWwXJ36imrF587TxVCz5rA5fThFgSU8HI8dzo1e7fwa4trypbfy4SppX9H+O1iMeNVSK/DeFjQFCqbzrmNRfQZXFXqMS/3UCgO49NhLJ+09tS64SJDC6NRNDfgYFRgPYNKCKNRWxz5izst2enlobFIg7AHsRMOT5wGd7crmsNyWaGiqtH8WGSAhNaSkm+exI+Q/Wk44Ta/DV97HK01uGBdo0xrHfG4kr2RjcMI7KwYoo870Lhh7yOgYJvXkcYiTs6aFHG7tq3AM+AT6HLvhjb6mE80EheMhQ5iI5UHyD8Zci7RbM4DcpwG9SlfiAXNJJh2DAvJSzI/ozONB5G5AQWq5BA2p878TIjTmiL0yCBr3qHaf/USozZHMFiIallLjiz4kp4wloSYnrTF/sYj6gxRDD4rtzxaO6TEnOS63u3+dDd1HwLilVQdTP+ve06C+Diy9k6qLz7iIA7XWxXbt0Yo1SkFh+VzWK2JIkKaeJaj4eqdr/eNufRCbgCx6q3bj8eFFEfnKr/yFJ+ptlpXbxbxxMhZx5eLXhLiLhOpoDk4jU11L0kW2HzdWymA9xtwIHVqAqPGFzAszsdx5YkvC+qZp6RqUpfTecYaebDXFWGI6PYjWDOIB4K3l/NrtWwY0DLRsRj+u7XVokOLBXkL5bxq6oul2ISS2EOmkrXtIiLdbi0llxLcfZ5jpe7+RSmd/i3eeROdAPGmZQX12dzOdppcTVU4VIZ1UtGQZS8+7aJyRAoDc3MLJsNY8SMjqWr+xnzcrHPP1E2VAuX+k/UF+pEyIM8ohs+Y2R0HsVwn23YKg345KZrq0TlGJ1w3tUeM8OnUENwRvXgg73i8LPTmgdDMng7oxQ3z3FlFP4fMpCBGK/awuMQDE3AQohhEQIBaio0hGhp1hva6EbuHUI9Dwgwjxm8Ar3l3pUYyzKPxWHV5K/EXVju2s7c57t4A2AHV/iNxgswEKac+xXE5YUsL3q/n+RvY+LmK2g5ReSYpTGBLy/8VVR+ckFO+I66TJHjpVGFvEdlcEzUA4sMTZU2xsYSOQJUAHzgxz2EJmfj0uIDNYGCX7+t5deEhxnIQrl/cXdwtq3y3BOpgTpUI0xx4HiZhbriaUMOG4YsbA+EPHqQMLmoK02+H6F7sbExbYjZjEiO63z0vyffyXCinKii4Z9TpMAxM8hImqNO88cN1mDbragSgwpVJVMaRbbyXdOwpMns5F8Ar+MGlw9gHrix9PzJMN/z6kOkMD0Ke/2zT+IBBRSmttTpufQc0xw5s4y0NqRmdQ751BWsO3AYFlK3e6T8ltBoA9K6I4/FM5F5iWKgmF4nWcrkUBrEZzb5S40vBcTq76BzxQ83BULMtl1HVlyfOdao1kW0EJLwzc3h47OfZmjVZV0c2UBuak3rKgruTb/Z20Y+jDuAqbBOUBIP/wmHNDcMrNG3ydYsf+O6EwpQOb4kpsKue6p3Em4YC5hyKhDxyqhVFF86VKAucvVwXwmtS/cQOTX6duR4wOCt57qQYHafK7qlvud5iBZ2IpBQLNoXG9AZmZFaR7yMKZe6fYX3zYcOAyAnIciWWY1f8UOzE7TIvPP4dhW8v6CFfxeATLbv2XA7UlEnhCKw==,iv:iBNeBQdohOdxcIOA49tb+stZY2JsfCSDR6lfyxr+nvQ=,tag:W5Nsi6D264DW1GIJ6tanTA==,type:str]
kind: Secret
metadata:
creationTimestamp: null
name: pia-custom-ovpn
namespace: vpn-egress
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBWVDhUbWtoSVJla2FLOHlF
bmdRdTlFWHZxVmJZWnZMYU50b0sxaGF3L1RRCjlNYWpSdk1CN1F3S3REM3NVcUZ1
dXg2ZjZBZWZwaGtXVEt2OStWTXNjdHcKLS0tIHZDbyt5ZnkxbWlrcWN6OUR1ZHhO
MXZjaE9ER1hlallOZjVYTnN3bmhZSmMKrIBjfP5AEZfBXgJTAL0UaO3LVdG2o5S1
eRyR3vqik0GMOUBEMGAjED6okoBe4x9IQ9PMWgqy9HRm2KQx4WH4gw==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-08-23T05:44:49Z"
mac: ENC[AES256_GCM,data:Vtde8ZTXgc17w2ilZNcIesB3U6+qm2Vkee02pg1oRfiLaqugLuJjwO3YfxP0ud8blz6jBp1npOOKltl43Wv/tLOpimstjnrlf5y2pg+8f1+3dbKhCcQo2OCUbGHi1NV21I9TiV3yZ1iJAefwMGWvzsjBCw5SE3x5rUOvtRFlur4=,iv:uAWe3MiGcmmslxh33JPtOntkBFkPqUEpF6I2FnXB7fM=,tag:460PD8F8pF2q72TCsjfRrw==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4