The in-cluster Gluetun gateway is retired; VPN egress is now a node-level host
OpenVPN split-tunnel (Ansible role vpn_egress_gateway) steered by a Cilium
Egress Gateway policy with a host nftables kill-switch (ADR-0015). Credentials
moved host-side to the Ansible vault. Update docs/comments only:
- platform/vpn-egress/README.md: rewrite for ADR-0015 (live + conformance-green);
drop the resolved "option 1 not viable with Gluetun" finding to a brief history note.
- platform/kustomization.yaml: Phase 3 comment -> node tunnel + Cilium steering.
- platform/kyverno/.../require-egress-class-label.yaml: ADR-0013 -> ADR-0015 steering.
- schemas/servicedefinition.schema.json: egress=vpn description -> node tunnel + Cilium.
- clusters/base/cluster-vars.yaml: reframe PIA_REGION / CLUSTER_POD_CIDR WHY comments
to ADR-0015 (Cilium excludedCIDRs); both keys retained (still referenced/documented).
- platform/vpn-egress/namespace.yaml: no privileged gateway pod under ADR-0015.
- pia-credentials.sops.yaml / pia-custom-ovpn.sops.yaml: comment headers only note
they are now UNUSED (creds host-side), retained pending cleanup; payloads untouched.
No runtime manifest behavior, alert exprs, kustomization resources, or SOPS payloads
changed. No spec-repo edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator-approved. Removes the vestigial in-cluster gluetun Deployment/Service (the node-level
host tunnel steered by Cilium Egress Gateway is validated + conformance-green) and updates the
VPNTunnelDown alert to key on node tun0 presence (absent(node_network_receive_bytes_total{device=tun0}))
instead of the now-gone gluetun_vpn_up gauge; drops the obsolete pod-based EgressGatewayUnhealthy.
Unused pia-* SOPS secrets retained pending a separate cleanup.
gluetun retirement removes the gluetun_vpn_up gauge and the in-cluster gateway pod. Point
VPNTunnelDown at absent(node_network_receive_bytes_total{device=tun0}) — tun0 exists iff the
host OpenVPN tunnel is up (node_network_up unreliable: tun0 operstate=unknown). This also
fires if the gateway node/node-exporter is down (series goes stale). Remove the pod-readiness
EgressGatewayUnhealthy alert — there is no gateway pod under ADR-0015.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Prepared on a feature branch for operator review — NOT merged/pushed (retiring the running
fallback needs explicit approval). The node-level host OpenVPN tunnel steered by Cilium Egress
Gateway is validated end-to-end and gated by green conformance (egress-class + vpn-killswitch).
A Gluetun pod's tun0 was never steerable by Cilium egress gateway (the reason ADR-0015 moved the
tunnel to the node), so gluetun serves no steering role. The now-unused pia-* SOPS secrets are
retained pending a separate cleanup (creds are host-side in the Ansible vault).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
chainsaw v0.2.12 has no test-level `spec.finally` (finally exists only on a
step), and a step-level `finally` accepts only catch-style ops (script/command/
delete/wait), NOT `apply`/`assert`. Both tests failed to parse ("unknown field").
- egress-class: drop the redundant test-level `spec.finally` delete-pods block —
chainsaw auto-deletes applied fixtures on cleanup (skipDelete: false). The
valid step-level `finally` that restores the egress-class label is kept.
- vpn-killswitch: move the SAFETY-CRITICAL self-healing tunnel restart out of the
invalid `spec.finally` and into the STEP-level `finally` of the fault step
`when-force-tunnel-down-then-no-leak`, implemented as a `script` (finally can't
use apply/assert) that `kubectl apply`s the privileged node-pinned restart Job
(name `tunnel-restart`, distinct from `tunnel-stop`) and `kubectl wait`s for it
to complete — the equivalent of `assert succeeded:1`, blocking teardown until
the host confirms the tunnel is up. This finally runs even if the no-leak /
internet-unaffected asserts in the same step's `try` fail, so no path leaves the
gateway tunnel down. The recovery step is now poll-only (restart already
happened in the prior step's finally).
Validation with the pinned binary (~/.local/bin/chainsaw v0.2.12):
`chainsaw lint test --file ...` => "The document is valid" for both; `chainsaw
test ... --no-cluster` loads both past the parse stage (no "unknown field"), then
fails only at runtime for lack of a cluster (expected). yamllint + run.sh
--strict-coverage clean (RC=0). No live apply / fault injection, no spec edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Turn the vpn-killswitch scaffold into a runnable SAFETY-CRITICAL L3 test for
R-NET-06 under ADR-0015 (node-level host OpenVPN tunnel steered by Cilium Egress
Gateway with a host nftables kill-switch), replacing the obsolete in-cluster
gluetun design.
- fixtures/vpn-and-internet-pods.yaml: two egress-class probe pods + documented
RBAC/PodSecurity/namespace requirements for node-level fault injection.
- vpn-killswitch/chainsaw-test.yaml: GIVEN healthy tunnel (vpn egress != WAN);
WHEN the host `vpn-egress-tunnel` systemd unit is STOPPED via a privileged,
hostPID, node-pinned nsenter Job THEN vpn-class egress is blackholed (curl
fails, never returns the WAN IP = no leak) while internet-class is unaffected;
WHEN restarted THEN vpn egress recovers. spec-level `finally` ALWAYS runs an
idempotent start Job (self-healing) so a mid-test failure never leaves the
gateway tunnel down.
- incluster/rbac.yaml: grant the runner least-privilege for the delivered tests
(namespaces create/delete/patch for the PSA relabel, pods/services/configmaps/
pods-exec/pods-log, batch/jobs) and replace the gluetun-scale rule with the
ADR-0015 host-systemd mechanism note (privilege via PodSecurity, not RBAC).
Offline validation: python yaml parse (all docs), `run.sh --lint` (structural),
`run.sh --coverage-only --strict-coverage` (no tag/annotation drift; R-NET-05/06/
07 covered, R-NET-06 has its L3 test), yamllint. No live apply, no cluster run.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Session-limit interrupted the WS-TEST agent. egress-class test updated; vpn-killswitch
test + host fault-injection fixture still outstanding. Preserved for review/continuation.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The down-script previously flushed the kill-switch source set on tunnel-down, which
disarmed the 'ip saddr @vpn_egress_gateway_src oifname != tun0 drop' rule exactly when
the tunnel was down — leaving fail-closed to ride only on the stale SNAT source being
non-routable (could leak if the edge NATed it). Retain the set instead: the drop rule
stays armed so any vpn-class packet leaving the WAN while the tunnel is down is EXPLICITLY
dropped (edge-independent). Safe because the up-script flush+re-adds the source on every
reconnect. Validated behavior: WS-NET fault-injection on k3s-dev-3 (2026-08-24).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The CiliumEgressGatewayPolicy authored in 38fcfad was inadvertently removed from main when
feature/vpn-egress-gateway-role (which carried a git-rm of the duplicated file) was merged.
The kustomization now references it, so restore the file. Content identical to 38fcfad.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Host OpenVPN split-tunnel on k3s-dev-3 validated 2026-08-24: tun0 up, VPN egress
158.173.241.36, node default unchanged (WAN), scoped nft kill-switch active. Wire the
CiliumEgressGatewayPolicy into the runtime kustomization so egress-class=vpn pods steer
to the node tun0.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Author the vpn_egress_gateway role + playbook that stands up the NODE-level
PIA OpenVPN tunnel on egress-gateway node(s) — the host half of ADR-0015. The
CiliumEgressGatewayPolicy steering vpn-class pods is separate GitOps against the
same contract (group vpn_egress_gateway, node label
homelab.keithjackson.dev/egress-gateway=true, host iface tun0).
Honors all four ADR-0015 invariants:
- Split-tunnel (inv 1&2): OpenVPN route-noexec + pull-filter ignore
redirect-gateway => installs no routes; policy routing in a dedicated table
(51820) with `ip rule from <tun0-addr>` steers ONLY Cilium-SNAT'd vpn-class
traffic. Node default route stays on the WAN, tunnel up or down.
- Scoped nftables kill-switch (inv 3&4): own inet table, policy accept, drops
only tunnel-sourced (@vpn_egress_gateway_src set) packets not leaving tun0 =>
vpn-class fails closed (blackhole), node/internet-class fail open (WAN),
never matched.
PIA creds + the proven sweden-aes-128-cbc-udp-ip .ovpn come from Ansible vault
(host-side per ADR-0015/docs-07); vault keys documented, no secret values
committed. ansible-lint (production) + yamllint clean. Not deployed: green
conformance is gated on the live tunnel (operator host creds) — see role README.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add platform/vpn-egress/egress-steering.yaml — a CiliumEgressGatewayPolicy
that steers homelab.keithjackson.dev/egress-class=vpn pods out a gateway
node's host tun0 split-tunnel interface, per ADR-0015 (node-level tunnel +
Cilium Egress Gateway). Honors all four ADR-0015 invariants: only vpn-class
pods are selected (default/internet + node stay on the local/WAN path),
RFC1918 + cluster ranges are excludedCIDRs so only public internet is
tunneled and tun0 is never a default route, and the host nftables kill-switch
(Ansible) fails vpn-class CLOSED while internet-class fails OPEN.
Interface contract shared verbatim with the parallel Ansible host-tunnel role:
node label homelab.keithjackson.dev/egress-gateway=true, interface tun0,
client label homelab.keithjackson.dev/egress-class=vpn.
Cilium Egress Gateway prerequisites confirmed enabled (egressGateway.enabled,
bpf.masquerade, kubeProxyReplacement all true, [parity]) — no HelmRelease change.
Intentionally NOT added to platform/vpn-egress/kustomization.yaml: the datapath
stays out of the runtime until the live tunnel exists and the egress-class /
vpn-killswitch conformance tests pass (README rule; DoD). Gluetun is retained
as the validated fallback until then. Serves R-NET-05/06/07.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Author the internet/vpn egress-class fixture pods for the egress-class + vpn-killswitch
conformance tests. Record the datapath finding: ADR-0013's preferred steering (Cilium
egress gateway -> gluetun tun0) is NOT viable because gluetun must run hostNetwork to
put tun0 on the node, and gluetun then redirects the NODE's default route through the
VPN (breaks k3s/etcd). Escalated: remaining viable path is option 2 (gateway-pod
routing). No steering datapath committed until the mechanism is chosen + validated.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
prometheus-...-0 was OOMKilled (exit 137) every ~4min during TSDB compaction/checkpoint
(261 restarts/37h). Raise limit to 2Gi + request to 512Mi; nodes have headroom.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Once the tunnel redirects the pod default route via tun0, the pod's response to the
kubelet's httpGet liveness probe egresses the tunnel and never returns → probe times
out → pod restarts ~every 2min despite the tunnel being up (PIA egress IP confirmed).
Switch to a local wget exec probe (stays on lo).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator provides a new .ovpn with an IP remote (gluetun's custom parser rejects
hostnames), so drop the resolve-remote initContainer/VPN_ENDPOINT_IP attempts and
mount the .ovpn directly (key pia.ovpn). Bump image to latest v3.41.3.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
gluetun's custom parser rejects the .ovpn's 'remote sweden.privacy.network' (needs
an IP). sweden.privacy.network LB pool = 158.173.241.{22,34,55,112,239}; pin .22
(verified to handshake) via VPN_ENDPOINT_IP/PORT while the .ovpn supplies cipher/CA.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator's kubectl create secret keyed the file by its name
(sweden-aes-128-cbc-udp-dns.ovpn); mount item now references that key -> pia.ovpn.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
PIA-provider server selection is flaky here (random dead IPs); the operator's .ovpn
uses sweden.privacy.network (load-balanced, reliable). Switch to custom mode
(VPN_SERVICE_PROVIDER=custom, OPENVPN_CUSTOM_CONFIG=/config/pia.ovpn), mount the
pia-custom-ovpn Secret, keep creds + kill-switch + tun-mtu 1300. Drops PIA-provider
server list/updater and native PF (Phase-6 concern). Pending the operator-provisioned
pia-custom-ovpn Secret (not merged until it exists).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
gluetun silently ignored PIA_ENCRYPTION_PRESET (renamed) and defaulted to strong
(RSA-4096) the whole time. The correct v3.x var is
PRIVATE_INTERNET_ACCESS_OPENVPN_ENCRYPTION_PRESET=normal → RSA-2048/AES-128-CBC/1198,
the smaller handshake that fits the 1450 pod MTU.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
RSA-4096 strong handshake wouldn't hold over the pod's 1450 MTU (Cilium VXLAN);
MTU tuning is exhausted (no --tls-mtu in OpenVPN 2.6, PIA rejects --fragment).
Operator downloaded the lower-encryption PIA config (port 1198, RSA-2048,
AES-128-CBC, SHA1) = gluetun's 'normal' preset. Smaller handshake fits 1450.
Dropped the strong-handshake band-aid flags; kept OPENVPN_MTU 1300 for the data channel.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
RSA-4096 strong-preset handshake is marginal over the 1450 pod MTU (Cilium VXLAN).
--mtu-disc no clears DF so oversized control packets fragment through instead of
being silently dropped. Last MTU-path lever (OpenVPN 2.6 has no --tls-mtu).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
OpenVPN 2.6.16 rejects --tls-mtu (not a real option) and PIA doesn't support
--fragment, so control-channel packets can't be shrunk. Keep --hand-window 120 to
give the RSA-4096 handshake more time over a lossy/marginal path, plus tun-mtu 1300.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
RSA-4096 strong-preset handshake is marginal over the pod's 1450 MTU (Cilium VXLAN).
Lower tun-mtu to 1300 and add --tls-mtu 1200 (smaller control packets) + --hand-window
120 (more time to complete the handshake over a lossy path). Empirical test of the
MTU-tuning path (operator's choice over Cilium native routing).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 4.8