Add platform/vpn-egress/egress-steering.yaml — a CiliumEgressGatewayPolicy
that steers homelab.keithjackson.dev/egress-class=vpn pods out a gateway
node's host tun0 split-tunnel interface, per ADR-0015 (node-level tunnel +
Cilium Egress Gateway). Honors all four ADR-0015 invariants: only vpn-class
pods are selected (default/internet + node stay on the local/WAN path),
RFC1918 + cluster ranges are excludedCIDRs so only public internet is
tunneled and tun0 is never a default route, and the host nftables kill-switch
(Ansible) fails vpn-class CLOSED while internet-class fails OPEN.
Interface contract shared verbatim with the parallel Ansible host-tunnel role:
node label homelab.keithjackson.dev/egress-gateway=true, interface tun0,
client label homelab.keithjackson.dev/egress-class=vpn.
Cilium Egress Gateway prerequisites confirmed enabled (egressGateway.enabled,
bpf.masquerade, kubeProxyReplacement all true, [parity]) — no HelmRelease change.
Intentionally NOT added to platform/vpn-egress/kustomization.yaml: the datapath
stays out of the runtime until the live tunnel exists and the egress-class /
vpn-killswitch conformance tests pass (README rule; DoD). Gluetun is retained
as the validated fallback until then. Serves R-NET-05/06/07.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Author the internet/vpn egress-class fixture pods for the egress-class + vpn-killswitch
conformance tests. Record the datapath finding: ADR-0013's preferred steering (Cilium
egress gateway -> gluetun tun0) is NOT viable because gluetun must run hostNetwork to
put tun0 on the node, and gluetun then redirects the NODE's default route through the
VPN (breaks k3s/etcd). Escalated: remaining viable path is option 2 (gateway-pod
routing). No steering datapath committed until the mechanism is chosen + validated.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
prometheus-...-0 was OOMKilled (exit 137) every ~4min during TSDB compaction/checkpoint
(261 restarts/37h). Raise limit to 2Gi + request to 512Mi; nodes have headroom.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Once the tunnel redirects the pod default route via tun0, the pod's response to the
kubelet's httpGet liveness probe egresses the tunnel and never returns → probe times
out → pod restarts ~every 2min despite the tunnel being up (PIA egress IP confirmed).
Switch to a local wget exec probe (stays on lo).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator provides a new .ovpn with an IP remote (gluetun's custom parser rejects
hostnames), so drop the resolve-remote initContainer/VPN_ENDPOINT_IP attempts and
mount the .ovpn directly (key pia.ovpn). Bump image to latest v3.41.3.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
gluetun's custom parser rejects the .ovpn's 'remote sweden.privacy.network' (needs
an IP). sweden.privacy.network LB pool = 158.173.241.{22,34,55,112,239}; pin .22
(verified to handshake) via VPN_ENDPOINT_IP/PORT while the .ovpn supplies cipher/CA.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator's kubectl create secret keyed the file by its name
(sweden-aes-128-cbc-udp-dns.ovpn); mount item now references that key -> pia.ovpn.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
PIA-provider server selection is flaky here (random dead IPs); the operator's .ovpn
uses sweden.privacy.network (load-balanced, reliable). Switch to custom mode
(VPN_SERVICE_PROVIDER=custom, OPENVPN_CUSTOM_CONFIG=/config/pia.ovpn), mount the
pia-custom-ovpn Secret, keep creds + kill-switch + tun-mtu 1300. Drops PIA-provider
server list/updater and native PF (Phase-6 concern). Pending the operator-provisioned
pia-custom-ovpn Secret (not merged until it exists).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
gluetun silently ignored PIA_ENCRYPTION_PRESET (renamed) and defaulted to strong
(RSA-4096) the whole time. The correct v3.x var is
PRIVATE_INTERNET_ACCESS_OPENVPN_ENCRYPTION_PRESET=normal → RSA-2048/AES-128-CBC/1198,
the smaller handshake that fits the 1450 pod MTU.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
RSA-4096 strong handshake wouldn't hold over the pod's 1450 MTU (Cilium VXLAN);
MTU tuning is exhausted (no --tls-mtu in OpenVPN 2.6, PIA rejects --fragment).
Operator downloaded the lower-encryption PIA config (port 1198, RSA-2048,
AES-128-CBC, SHA1) = gluetun's 'normal' preset. Smaller handshake fits 1450.
Dropped the strong-handshake band-aid flags; kept OPENVPN_MTU 1300 for the data channel.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
RSA-4096 strong-preset handshake is marginal over the 1450 pod MTU (Cilium VXLAN).
--mtu-disc no clears DF so oversized control packets fragment through instead of
being silently dropped. Last MTU-path lever (OpenVPN 2.6 has no --tls-mtu).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
OpenVPN 2.6.16 rejects --tls-mtu (not a real option) and PIA doesn't support
--fragment, so control-channel packets can't be shrunk. Keep --hand-window 120 to
give the RSA-4096 handshake more time over a lossy/marginal path, plus tun-mtu 1300.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
RSA-4096 strong-preset handshake is marginal over the pod's 1450 MTU (Cilium VXLAN).
Lower tun-mtu to 1300 and add --tls-mtu 1200 (smaller control packets) + --hand-window
120 (more time to complete the handshake over a lossy path). Empirical test of the
MTU-tuning path (operator's choice over Cilium native routing).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
PIA's PF gateway 10.31.128.1 sits inside the /8 cluster-pool, so
FIREWALL_OUTBOUND_SUBNETS=10.0.0.0/8 made gluetun route it around the tunnel and
port forwarding timed out (getSignature to 10.31.128.1:19999). /16 covers actual
pod allocations and excludes PIA's 10.31.x.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator's working config is sweden.privacy.network:1197 with RSA-4096, AES-256-CBC,
SHA256 = PIA's 'strong' preset. gluetun defaults PIA to 'normal' (AES-128-GCM), a
cipher mismatch against the 1197 (strong) endpoint, so the handshake never completed
(silent after UDP link). PIA_ENCRYPTION_PRESET=strong matches it and selects 1197.
Reverts the 8080 port red herring.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
UPDATER_PERIOD only refreshes AFTER the tunnel is ready (chicken-and-egg: can't
connect with stale servers), and gluetun's bundled PIA Sweden list is retired
46.246.x (current is 158.173.x). Add an `update-servers` initContainer running
`gluetun update -providers "private internet access"` into a shared /gluetun
emptyDir the main container reads, so it dials current servers. UPDATER_PERIOD
raised to 720h (docs-compliant) for ongoing in-pod refresh.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CORRECTION: the connection failures were gluetun dialing RETIRED PIA IPs (46.246.x
= dead → timeouts, which I mis-read as a network block). A fresh PIA .ovpn connects
from the same host, and PIA's current SE Stockholm servers (158.173.x) are reachable.
Cilium was independently cleared (default-allow, host-fw off, Hubble FORWARDED).
Fix: UPDATER_PERIOD=24h so gluetun fetches PIA's current servers; keep
OPENVPN_MTU/MSSFIX=1350 for the real Cilium-VXLAN 1450 pod MTU (data channel);
PIA default UDP port. Re-enable vpn-egress. Full correction in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Decisive diagnosis: OpenVPN's outbound init packets are 14 bytes with zero
replies (so MTU was never the blocker), pod UDP egress is healthy (NTP replies),
but TCP 501/443 to PIA IPs time out while Google/Cloudflare are OPEN — across many
PIA servers. This 192.168.2.x WiFi segment blackholes PIA's IP ranges (and
intercepts DNS). Not code-fixable. Reverted the port/MTU red-herring env to PIA
defaults; disabled vpn-egress pending an operator network fix (move to the
unfiltered 192.168.1.x LAN, or unblock PIA on the router). Full trace in
DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Actual root cause (operator-directed layered MTU check): Cilium runs VXLAN
(routing-mode tunnel) so pods get MTU 1450 (1500-50 encap) while node, hypervisor,
and the Google Fiber WAN are all 1500. OpenVPN's ~1500-byte handshake packets
exceed 1450 and are dropped (PMTUD broken). The ports were a red herring.
Fix: OPENVPN_MTU=1350 + OPENVPN_MSSFIX=1350 so encrypted packets (~1402) fit under
1450. Keeps OpenVPN + VXLAN unchanged. Re-enable vpn-egress. Full layered trace
(pod/node/host) in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
OpenVPN can't connect on this network (each port fails differently, so it's
OpenVPN-specific not a UDP block — NTP/123 works): 1197 = home OpenVPN-server
conflict; 53 = DNS interception; 8080 = TLS handshake fails across servers (MTU
on WiFi+triple-NAT). Recommend pivoting to WireGuard via thrnz/docker-wireguard-pia
(UDP 1337, MTU-robust, ADR-0013 fallback, operator's original preference) — a
spec-level (ADR-0013) change, so escalated. Disabled meanwhile. Full trace in
DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Port 53 also got no PIA replies: this network transparently intercepts outbound
:53, so dig works but OpenVPN-over-53 is swallowed. An NTP/123 probe from the host
returned fine, proving arbitrary non-53 UDP works bidirectionally (so it's
port-specific: 1197 conflict + 53 DNS-intercept, not a UDP block). Move to 8080
(community-verified for PIA, non-DNS, non-1197).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Wiring the port via "${PIA_OPENVPN_PORT}" made Flux/kustomize emit an unquoted
53 (int) after substitution, which k8s rejects for an env value ("expected
string, got 53"). Same gotcha as the Longhorn numberOfReplicas fix. Hardcode the
quoted string; cluster-var removed with a NOTE explaining why.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85
