RSA-4096 strong-preset handshake is marginal over the 1450 pod MTU (Cilium VXLAN).
--mtu-disc no clears DF so oversized control packets fragment through instead of
being silently dropped. Last MTU-path lever (OpenVPN 2.6 has no --tls-mtu).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
OpenVPN 2.6.16 rejects --tls-mtu (not a real option) and PIA doesn't support
--fragment, so control-channel packets can't be shrunk. Keep --hand-window 120 to
give the RSA-4096 handshake more time over a lossy/marginal path, plus tun-mtu 1300.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
RSA-4096 strong-preset handshake is marginal over the pod's 1450 MTU (Cilium VXLAN).
Lower tun-mtu to 1300 and add --tls-mtu 1200 (smaller control packets) + --hand-window
120 (more time to complete the handshake over a lossy path). Empirical test of the
MTU-tuning path (operator's choice over Cilium native routing).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
PIA's PF gateway 10.31.128.1 sits inside the /8 cluster-pool, so
FIREWALL_OUTBOUND_SUBNETS=10.0.0.0/8 made gluetun route it around the tunnel and
port forwarding timed out (getSignature to 10.31.128.1:19999). /16 covers actual
pod allocations and excludes PIA's 10.31.x.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator's working config is sweden.privacy.network:1197 with RSA-4096, AES-256-CBC,
SHA256 = PIA's 'strong' preset. gluetun defaults PIA to 'normal' (AES-128-GCM), a
cipher mismatch against the 1197 (strong) endpoint, so the handshake never completed
(silent after UDP link). PIA_ENCRYPTION_PRESET=strong matches it and selects 1197.
Reverts the 8080 port red herring.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
UPDATER_PERIOD only refreshes AFTER the tunnel is ready (chicken-and-egg: can't
connect with stale servers), and gluetun's bundled PIA Sweden list is retired
46.246.x (current is 158.173.x). Add an `update-servers` initContainer running
`gluetun update -providers "private internet access"` into a shared /gluetun
emptyDir the main container reads, so it dials current servers. UPDATER_PERIOD
raised to 720h (docs-compliant) for ongoing in-pod refresh.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CORRECTION: the connection failures were gluetun dialing RETIRED PIA IPs (46.246.x
= dead → timeouts, which I mis-read as a network block). A fresh PIA .ovpn connects
from the same host, and PIA's current SE Stockholm servers (158.173.x) are reachable.
Cilium was independently cleared (default-allow, host-fw off, Hubble FORWARDED).
Fix: UPDATER_PERIOD=24h so gluetun fetches PIA's current servers; keep
OPENVPN_MTU/MSSFIX=1350 for the real Cilium-VXLAN 1450 pod MTU (data channel);
PIA default UDP port. Re-enable vpn-egress. Full correction in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Decisive diagnosis: OpenVPN's outbound init packets are 14 bytes with zero
replies (so MTU was never the blocker), pod UDP egress is healthy (NTP replies),
but TCP 501/443 to PIA IPs time out while Google/Cloudflare are OPEN — across many
PIA servers. This 192.168.2.x WiFi segment blackholes PIA's IP ranges (and
intercepts DNS). Not code-fixable. Reverted the port/MTU red-herring env to PIA
defaults; disabled vpn-egress pending an operator network fix (move to the
unfiltered 192.168.1.x LAN, or unblock PIA on the router). Full trace in
DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Actual root cause (operator-directed layered MTU check): Cilium runs VXLAN
(routing-mode tunnel) so pods get MTU 1450 (1500-50 encap) while node, hypervisor,
and the Google Fiber WAN are all 1500. OpenVPN's ~1500-byte handshake packets
exceed 1450 and are dropped (PMTUD broken). The ports were a red herring.
Fix: OPENVPN_MTU=1350 + OPENVPN_MSSFIX=1350 so encrypted packets (~1402) fit under
1450. Keeps OpenVPN + VXLAN unchanged. Re-enable vpn-egress. Full layered trace
(pod/node/host) in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
OpenVPN can't connect on this network (each port fails differently, so it's
OpenVPN-specific not a UDP block — NTP/123 works): 1197 = home OpenVPN-server
conflict; 53 = DNS interception; 8080 = TLS handshake fails across servers (MTU
on WiFi+triple-NAT). Recommend pivoting to WireGuard via thrnz/docker-wireguard-pia
(UDP 1337, MTU-robust, ADR-0013 fallback, operator's original preference) — a
spec-level (ADR-0013) change, so escalated. Disabled meanwhile. Full trace in
DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Port 53 also got no PIA replies: this network transparently intercepts outbound
:53, so dig works but OpenVPN-over-53 is swallowed. An NTP/123 probe from the host
returned fine, proving arbitrary non-53 UDP works bidirectionally (so it's
port-specific: 1197 conflict + 53 DNS-intercept, not a UDP block). Move to 8080
(community-verified for PIA, non-DNS, non-1197).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Wiring the port via "${PIA_OPENVPN_PORT}" made Flux/kustomize emit an unquoted
53 (int) after substitution, which k8s rejects for an env value ("expected
string, got 53"). Same gotcha as the Longhorn numberOfReplicas fix. Hardcode the
quoted string; cluster-var removed with a NOTE explaining why.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator identified the block: the home network NATs its own OpenVPN server on
WAN:1197, colliding with the cluster's outbound PIA-on-1197 flow (port-specific,
not a general UDP block — UDP/53 worked in tcpdump). PIA accepts OpenVPN on
53/1194/1197/1198/8080/9201; chose 53 (empirically proven to traverse this net).
- cluster-vars: add PIA_OPENVPN_PORT="53"
- gluetun: OPENVPN_ENDPOINT_PORT=${PIA_OPENVPN_PORT}
- re-enable vpn-egress; DECISIONS-LOG resolution noted
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Phase-3 VPN config is complete + correct (OpenVPN-UDP, SE Stockholm), but the
OpenVPN UDP/1197 handshake to PIA never gets a reply. Traced with tcpdump: packets
leave pod→node→host→WiFi (SNAT 192.168.2.103) fine, but 0 replies return to the
host, while host UDP/53 works. The 192.168.2.x home router/ISP drops OpenVPN UDP —
upstream of everything in this repo. Reverted to disabled to avoid a crashloop
firing false KubePodCrashLooping alerts; one-line re-enable once the network is
fixed. Full trace in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gluetun's PIA OpenVPN provider rejects "Sweden"; the valid PF-capable Sweden
region identifier is "SE Stockholm" ("SE Streaming Optimized" lacks port
forwarding). Preserves the operator's Sweden choice.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gluetun cannot do PIA over WireGuard (upstream unsupported; live "provider name
is not valid for Wireguard"). Operator approved OpenVPN-UDP and ADR-0013 was
amended in the spec. This aligns the impl:
- platform/kustomization.yaml: re-enable `- vpn-egress` (Phase 3, OpenVPN-UDP)
- gluetun-gateway.yaml: clean stale WireGuard comments (header, image note,
VPN_TYPE/creds comments); config already VPN_TYPE=openvpn
- README.md: PIA-over-OpenVPN, status=enabled, steering iface wg0 -> tun0
- DECISIONS-LOG.md: disk/drive incident (k3s -> NVMe; failing Seagate) + Phase-3
OpenVPN resolution
Stale WireGuard comments inside the operator's encrypted pia-credentials.sops.yaml
header left untouched (editing breaks the SOPS MAC; cosmetic).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The deployed v3.40.0 rejected PIA+WireGuard ('private internet access is not valid for
Wireguard'). Verified v3.41.0 accepts the PIA+WireGuard provider/protocol (passes config
validation; ADR-0013's model holds — v3.40.0 was just too old). Bump the pin.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
PIA creds provisioned (SOPS) + wired. Enable vpn-egress in platform/kustomization.yaml
to deploy the shared Gluetun gateway (PIA over WireGuard, Sweden region, kill-switch).
Egress-class steering (vpn-class pods -> tunnel) still to be validated against the live
tunnel next. Gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
gluetun-gateway.yaml referenced ${PIA_REGION}/${CLUSTER_POD_CIDR} but they were never defined
in cluster-vars, and the README didn't say where to set them. Add both to
clusters/base/cluster-vars.yaml (ConfigMap data:, string values, Flux postBuild substitution).
CLUSTER_POD_CIDR is 10.0.0.0/8 (Cilium cluster-pool — pods get 10.x, confirmed live), NOT the
k3s 10.42.0.0/16 I'd assumed. README now names the file + format + per-env override + PF-region note.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Enabling monitoring deadlocked the platform Kustomization: alerts.yaml ships
PrometheusRule CRs whose monitoring.coreos.com CRDs are installed by the
kube-prometheus-stack HelmRelease in the SAME pass → 'no matches for kind
PrometheusRule' → atomic apply failure → NO monitoring HelmReleases created.
Same controllers/configs fix as cert-manager/kyverno: drop alerts.yaml from
platform/monitoring and apply it via platform-config (dependsOn platform), so the
monitoring CRDs exist first. Verified: platform has 0 PrometheusRule (keeps the 3
monitoring HRs); platform-config has the 8 alert rules; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Per ADR-0013: platform/vpn-egress/ = a single shared Gluetun gateway (PIA over WireGuard,
built-in kill-switch FIREWALL=on, native port forwarding), namespace (non-app, NET_ADMIN +
/dev/net/tun), control-API Service, and a pre-wired (commented) pia-credentials SOPS line.
Defined the PIA secret location (platform/vpn-egress, keys username/password → OPENVPN_USER/
OPENVPN_PASSWORD) and updated the SOPS template accordingly.
NOT wired into platform/kustomization.yaml (Phase 3, author-first) — deploy needs PIA creds +
validation of the egress-steering mechanism (Cilium Egress Gateway → the Gluetun node's wg0;
documented in README with options + a fail-closed rationale, deliberately NOT committed as an
untested CiliumEgressGatewayPolicy). WireGuard chosen (UDP, protocol != TCP); PIA supports it on
all regions and the secret stays username/password.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85
