jakman85andClaude Opus 4.8
f17c46b767
Actual root cause (operator-directed layered MTU check): Cilium runs VXLAN (routing-mode tunnel) so pods get MTU 1450 (1500-50 encap) while node, hypervisor, and the Google Fiber WAN are all 1500. OpenVPN's ~1500-byte handshake packets exceed 1450 and are dropped (PMTUD broken). The ports were a red herring. Fix: OPENVPN_MTU=1350 + OPENVPN_MSSFIX=1350 so encrypted packets (~1402) fit under 1450. Keeps OpenVPN + VXLAN unchanged. Re-enable vpn-egress. Full layered trace (pod/node/host) in DECISIONS-LOG. Co-Authored-By: Claude Opus 4.8 <[email protected]>
114 lines
5.1 KiB
YAML
114 lines
5.1 KiB
YAML
---
|
||
# Gluetun — the single shared in-cluster VPN egress gateway (ADR-0013): PIA over OpenVPN-UDP, with
|
||
# Gluetun's built-in kill-switch/firewall (fail-closed), health checks, auto-reconnect, and native
|
||
# port forwarding for qBittorrent. `vpn`-class pods are steered through this gateway (see README.md
|
||
# "Egress steering" — the Cilium-native mechanism, pending live validation with PIA creds).
|
||
#
|
||
# Credentials: the `pia-credentials` SOPS Secret (keys: username, password). Gluetun uses PIA's
|
||
# username/password (via OPENVPN_USER/OPENVPN_PASSWORD) to authenticate to PIA. Gluetun ships PIA's
|
||
# OpenVPN server configs, so no manual `.ovpn` is needed (ADR-0013 amendment: OpenVPN-UDP — Gluetun
|
||
# does NOT support PIA over WireGuard).
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
metadata:
|
||
name: gluetun
|
||
namespace: vpn-egress
|
||
labels:
|
||
app: gluetun
|
||
spec:
|
||
replicas: 1 # ONE shared tunnel (ADR-0013). Active/standby is a later enhancement.
|
||
strategy:
|
||
type: Recreate # a tunnel is a singleton — never run two overlapping during a rollout
|
||
selector:
|
||
matchLabels:
|
||
app: gluetun
|
||
template:
|
||
metadata:
|
||
labels:
|
||
app: gluetun
|
||
spec:
|
||
containers:
|
||
- name: gluetun
|
||
image: qmcgaw/gluetun:v3.41.0 # pinned; Renovate bumps. PIA is OpenVPN-only in Gluetun
|
||
# (WireGuard for PIA is unsupported upstream — ADR-0013
|
||
# amendment; thrnz/docker-wireguard-pia is the WG fallback).
|
||
securityContext:
|
||
capabilities:
|
||
add: ["NET_ADMIN"] # required for the tunnel + firewall
|
||
env:
|
||
- name: VPN_SERVICE_PROVIDER
|
||
value: "private internet access"
|
||
- name: VPN_TYPE
|
||
value: "openvpn" # OpenVPN-UDP — satisfies "protocol != TCP" (R-NET); ADR-0013 amended
|
||
- name: OPENVPN_ENDPOINT_PORT
|
||
value: "8080" # 8080: PIA confirmed responding here; avoid 53 (this network
|
||
# intercepts DNS/:53). Port is NOT the root cause — MTU is (below).
|
||
# Hardcoded string (not ${var}): Flux/kustomize strips quotes on
|
||
# "${VAR}" → int coercion k8s rejects (Longhorn-fix gotcha).
|
||
# --- MTU: Cilium runs VXLAN (routing-mode tunnel), so pods get MTU 1450 (1500 − 50 encap) while
|
||
# the node/host/WAN are all 1500. OpenVPN's default 1500-byte packets exceed 1450 and are
|
||
# dropped (PMTUD broken). Shrink tun-mtu so packets fit: 1350 + ~52 overhead ≈ 1402 < 1450. ---
|
||
- name: OPENVPN_MTU
|
||
value: "1350" # tun-mtu; keeps encrypted UDP packets under the 1450 pod MTU
|
||
- name: OPENVPN_MSSFIX
|
||
value: "1350" # clamp TCP-in-tunnel to match, avoid in-tunnel fragmentation
|
||
# PIA credentials from the SOPS secret (Gluetun's PIA OpenVPN creds env):
|
||
- name: OPENVPN_USER
|
||
valueFrom:
|
||
secretKeyRef:
|
||
name: pia-credentials
|
||
key: username
|
||
- name: OPENVPN_PASSWORD
|
||
valueFrom:
|
||
secretKeyRef:
|
||
name: pia-credentials
|
||
key: password
|
||
- name: SERVER_REGIONS
|
||
value: "${PIA_REGION}" # cluster-vars: a PF-CAPABLE PIA region (ADR-0013 PF caveat)
|
||
- name: VPN_PORT_FORWARDING
|
||
value: "on" # PIA native port forwarding for qBittorrent
|
||
- name: FIREWALL
|
||
value: "on" # KILL-SWITCH: block all egress except the tunnel (fail-closed, R-NET-06)
|
||
- name: FIREWALL_OUTBOUND_SUBNETS
|
||
value: "${CLUSTER_POD_CIDR}" # allow steered client pods (pod CIDR) to route out via the tunnel
|
||
- name: DOT
|
||
value: "off" # DNS via cluster CoreDNS, not Gluetun's DoT
|
||
ports:
|
||
- name: control
|
||
containerPort: 8000 # Gluetun control API (health + forwarded-port lookup)
|
||
livenessProbe:
|
||
httpGet: { path: /v1/openvpn/status, port: 8000 } # Gluetun control API health
|
||
initialDelaySeconds: 20
|
||
periodSeconds: 30
|
||
resources:
|
||
requests:
|
||
cpu: 50m
|
||
memory: 64Mi
|
||
limits:
|
||
memory: 160Mi
|
||
volumeMounts:
|
||
- name: tun
|
||
mountPath: /dev/net/tun
|
||
volumes:
|
||
- name: tun
|
||
hostPath:
|
||
path: /dev/net/tun
|
||
type: CharDevice
|
||
---
|
||
# Control API Service — the port-sync helper (qbittorrent-natmap, README) reads the PIA-assigned
|
||
# forwarded port from here; also used for tunnel-state health/alerting (ADR-0013 "alert on tunnel state").
|
||
apiVersion: v1
|
||
kind: Service
|
||
metadata:
|
||
name: gluetun-control
|
||
namespace: vpn-egress
|
||
labels:
|
||
app: gluetun
|
||
spec:
|
||
selector:
|
||
app: gluetun
|
||
ports:
|
||
- name: control
|
||
port: 8000
|
||
targetPort: 8000
|