Files
home-network-k3s/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md
jakman85andClaude Opus 5.5 250fb54154 docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 22:46:00 -06:00

100 lines
7.7 KiB
Markdown

# ADR-0015 — VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch)
- Status: Accepted — tunnel placement, steering and the four invariants stand (Kill-switch and Resilience bullets, and the mechanism named in invariant 3, superseded by [ADR-0020](adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md))
- Date: 2026-08-23
- Serves: R-NET-05, R-NET-06, R-NET-07
- Supersedes (egress-steering + tunnel placement of): [ADR-0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md)
— which now stands only for the **provider/protocol** (PIA over OpenVPN-UDP + port forwarding). By
extension this also closes the steering question left open since [ADR-0008](adr-0008-vpn-egress-model.md).
## Context
ADR-0013 placed the VPN in a **single shared in-cluster Gluetun pod** and *deferred* the egress-steering
mechanism (how `egress-class=vpn` pods actually route through it), naming "Cilium Egress Gateway → the
gateway's tunnel interface" as the recommended-but-unvalidated option.
Implementation (2026-08) proved that option **non-viable**, and validated the tunnel itself:
- The in-cluster Gluetun tunnel **works** — PIA Stockholm egress confirmed, stable.
- But `CiliumEgressGatewayPolicy` SNATs matched pods out of a **node** interface; Gluetun-in-a-pod's `tun0`
lives in the *pod* netns, invisible to the node datapath. Running Gluetun `hostNetwork` to put `tun0` on
the node **breaks the node**: Gluetun owns its netns routing and on connect redirects `0.0.0.0/0` via the
tunnel — in host netns that hijacks the **node's own default route**, cutting k3s/etcd/kubelet off. Gluetun
has no "create `tun0` but don't touch the default route" mode.
- The accepted community alternative — the **pod-gateway** pattern (VXLAN overlay + admission-webhook-injected
client routing + gateway NAT, e.g. `angelnu/pod-gateway`) — works but is heavy, mutates client pods, and
side-steps Cilium's datapath.
The clean fix is to move the tunnel to where routing is fully controllable — the **node** — and use Cilium's
native egress gateway to steer.
## Decision
The `egress: vpn` contract is realized by a **node-level VPN tunnel steered by Cilium Egress Gateway, with a
host-level kill-switch**:
- **Tunnel — host service, not a pod.** One or two labeled **egress-gateway nodes** run a **host OpenVPN
systemd service** (PIA over OpenVPN-UDP — provider/protocol per ADR-0013, same `.ovpn`). It is **Ansible-
managed host infrastructure** (like k3s itself), not a Flux workload. It is configured **split-tunnel**:
it creates `tun0` but does **not** set the node default route (`route-noexec` + policy routing), so the
node's own traffic (control plane, image pulls, other pods) stays on the WAN. Only traffic Cilium points
at `tun0` uses the VPN.
- **Steering — Cilium Egress Gateway.** A `CiliumEgressGatewayPolicy` selects pods labeled
`homelab.keithjackson.dev/egress-class=vpn` and egresses them via the gateway node's `tun0` (SNAT to the
tunnel address). Native Cilium eBPF — **no client-pod changes, no sidecar, no overlay**. (`egressGateway`,
`bpf.masquerade`, `kubeProxyReplacement` are already enabled per ADR-0002.)
- **Kill-switch (fail-closed, R-NET-06) — host nftables.** An **nftables rule on the gateway node** DROPS the
steered (`egress-class=vpn`) traffic whenever `tun0` is absent/down — a host backstop **independent of**
Cilium egress-gateway fail-open edge cases. This is a *stronger* guarantee than either pod-based option.
- **Resilience.** Two egress-gateway nodes + **Cilium Egress Gateway HA** (gateway failover) provide the
active/standby the earlier ADRs asked for.
The user-facing contract is unchanged: services are labeled `egress-class=vpn` (Kyverno-enforced), have **no
independent egress path**, and the rules are **enforced outside the service** — now via a Cilium policy + host
nft rather than an in-cluster gateway pod.
### Invariants (NON-NEGOTIABLE — implementation must satisfy all four)
The two egress classes have **opposite fail-safe directions**; getting this wrong either leaks VPN traffic or
breaks the cluster. These are hard requirements, not preferences:
1. **Default is the local/private WAN path — for everything.** Every pod (regardless of namespace) AND the
nodes themselves egress via the normal private-network/WAN stack by default. Traffic is on the VPN **only**
if a service explicitly opts in with `egress-class=vpn`.
2. **The VPN tunnel is never a default route.** Not for the node, not for any pod. `tun0` carries **only** the
flows a `CiliumEgressGatewayPolicy` explicitly selects (`egress-class=vpn`). The host tunnel is split-tunnel
(`route-noexec`/policy routing) precisely to guarantee this.
3. **The kill-switch blackhole is scoped to vpn-class only.** When `tun0` is down, the nftables rule drops
**only** the `egress-class=vpn` (Cilium-egress-gateway-marked) traffic. It MUST NOT match or drop default/
`internet`-class traffic or node traffic. `internet`-class egress is unaffected whether the tunnel is up
or down.
4. **Opposite fail directions.** Default/`internet` traffic **fails open** (toward the WAN — never blackholed).
`egress-class=vpn` traffic **fails closed** (toward blackhole — never leaks to the WAN). A misconfiguration
or partial failure must degrade in these directions, never the reverse.
These are gated by the `egress-class` (default/internet shows WAN IP; vpn shows VPN IP) and `vpn-killswitch`
(tunnel down ⇒ vpn-class blackholed, internet-class unaffected) conformance tests.
## Consequences
- Steering is **Cilium-native and declarative** (a `CiliumEgressGatewayPolicy` in Git) and touches no app pods
— flipping a service's `egress-class` label re-routes it with no workload rearchitecting.
- **Two-layer ownership:** the tunnel is host infra (**Ansible** role on the gateway nodes); the steering
policy is **Flux/GitOps**. Consistent with how k3s is provisioned. This is the intended split, not a leak.
- **Secrets move host-side:** PIA credentials + `.ovpn` live on the gateway node(s) via **Ansible vault**, not
a cluster SOPS Secret — see docs/07. (SOPS still governs in-cluster secrets.)
- **Gluetun is dropped** as the gateway. Its validated custom-mode config + the working PIA `.ovpn` transfer
directly to the host OpenVPN service, so the connectivity work is retained; only the packaging changes.
- **Port forwarding** (PIA NAT-PMP for qBittorrent) must be re-solved host-side (a small pf/natpmp helper that
publishes the assigned port to the app) — Phase 6, as before.
- **SPOF** on a single gateway node unless run HA (two nodes + Cilium failover).
- Validated end-to-end by the **`egress-class`** (R-NET-05/07) and **`vpn-killswitch`** (R-NET-06) conformance
tests: vpn-class egress shows the PIA IP, internet-class shows the WAN IP, a git label flip re-routes, and
forcing `tun0` down drops vpn-class egress (no leak) while internet-class is unaffected.
## Alternatives considered
- **In-cluster Gluetun pod + Cilium Egress Gateway → its `tun0`** (ADR-0013's deferred option 1) — NON-VIABLE:
`hostNetwork` Gluetun hijacks the node default route.
- **pod-gateway pattern** (VXLAN overlay + admission webhook + gateway NAT) — the accepted *in-cluster*
alternative; works but heavier, mutates client pods, bypasses Cilium's datapath. Rejected for a Cilium
cluster in favor of the native node-level egress gateway.
- **Per-pod Gluetun sidecar** (ADR-0008 Pattern A) — rejected earlier: privilege colocation with the workload
and no declarative east-west policy.
- **Keeping the working in-cluster Gluetun pod for its OWN egress only** — retained as the fallback if the
node-level tunnel proves troublesome, but it cannot serve *other* pods, which is the whole `egress: vpn`
requirement.