The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
100 lines
7.7 KiB
Markdown
100 lines
7.7 KiB
Markdown
# ADR-0015 — VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch)
|
|
- Status: Accepted — tunnel placement, steering and the four invariants stand (Kill-switch and Resilience bullets, and the mechanism named in invariant 3, superseded by [ADR-0020](adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md))
|
|
- Date: 2026-08-23
|
|
- Serves: R-NET-05, R-NET-06, R-NET-07
|
|
- Supersedes (egress-steering + tunnel placement of): [ADR-0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md)
|
|
— which now stands only for the **provider/protocol** (PIA over OpenVPN-UDP + port forwarding). By
|
|
extension this also closes the steering question left open since [ADR-0008](adr-0008-vpn-egress-model.md).
|
|
|
|
## Context
|
|
ADR-0013 placed the VPN in a **single shared in-cluster Gluetun pod** and *deferred* the egress-steering
|
|
mechanism (how `egress-class=vpn` pods actually route through it), naming "Cilium Egress Gateway → the
|
|
gateway's tunnel interface" as the recommended-but-unvalidated option.
|
|
|
|
Implementation (2026-08) proved that option **non-viable**, and validated the tunnel itself:
|
|
- The in-cluster Gluetun tunnel **works** — PIA Stockholm egress confirmed, stable.
|
|
- But `CiliumEgressGatewayPolicy` SNATs matched pods out of a **node** interface; Gluetun-in-a-pod's `tun0`
|
|
lives in the *pod* netns, invisible to the node datapath. Running Gluetun `hostNetwork` to put `tun0` on
|
|
the node **breaks the node**: Gluetun owns its netns routing and on connect redirects `0.0.0.0/0` via the
|
|
tunnel — in host netns that hijacks the **node's own default route**, cutting k3s/etcd/kubelet off. Gluetun
|
|
has no "create `tun0` but don't touch the default route" mode.
|
|
- The accepted community alternative — the **pod-gateway** pattern (VXLAN overlay + admission-webhook-injected
|
|
client routing + gateway NAT, e.g. `angelnu/pod-gateway`) — works but is heavy, mutates client pods, and
|
|
side-steps Cilium's datapath.
|
|
|
|
The clean fix is to move the tunnel to where routing is fully controllable — the **node** — and use Cilium's
|
|
native egress gateway to steer.
|
|
|
|
## Decision
|
|
The `egress: vpn` contract is realized by a **node-level VPN tunnel steered by Cilium Egress Gateway, with a
|
|
host-level kill-switch**:
|
|
|
|
- **Tunnel — host service, not a pod.** One or two labeled **egress-gateway nodes** run a **host OpenVPN
|
|
systemd service** (PIA over OpenVPN-UDP — provider/protocol per ADR-0013, same `.ovpn`). It is **Ansible-
|
|
managed host infrastructure** (like k3s itself), not a Flux workload. It is configured **split-tunnel**:
|
|
it creates `tun0` but does **not** set the node default route (`route-noexec` + policy routing), so the
|
|
node's own traffic (control plane, image pulls, other pods) stays on the WAN. Only traffic Cilium points
|
|
at `tun0` uses the VPN.
|
|
- **Steering — Cilium Egress Gateway.** A `CiliumEgressGatewayPolicy` selects pods labeled
|
|
`homelab.keithjackson.dev/egress-class=vpn` and egresses them via the gateway node's `tun0` (SNAT to the
|
|
tunnel address). Native Cilium eBPF — **no client-pod changes, no sidecar, no overlay**. (`egressGateway`,
|
|
`bpf.masquerade`, `kubeProxyReplacement` are already enabled per ADR-0002.)
|
|
- **Kill-switch (fail-closed, R-NET-06) — host nftables.** An **nftables rule on the gateway node** DROPS the
|
|
steered (`egress-class=vpn`) traffic whenever `tun0` is absent/down — a host backstop **independent of**
|
|
Cilium egress-gateway fail-open edge cases. This is a *stronger* guarantee than either pod-based option.
|
|
- **Resilience.** Two egress-gateway nodes + **Cilium Egress Gateway HA** (gateway failover) provide the
|
|
active/standby the earlier ADRs asked for.
|
|
|
|
The user-facing contract is unchanged: services are labeled `egress-class=vpn` (Kyverno-enforced), have **no
|
|
independent egress path**, and the rules are **enforced outside the service** — now via a Cilium policy + host
|
|
nft rather than an in-cluster gateway pod.
|
|
|
|
### Invariants (NON-NEGOTIABLE — implementation must satisfy all four)
|
|
The two egress classes have **opposite fail-safe directions**; getting this wrong either leaks VPN traffic or
|
|
breaks the cluster. These are hard requirements, not preferences:
|
|
|
|
1. **Default is the local/private WAN path — for everything.** Every pod (regardless of namespace) AND the
|
|
nodes themselves egress via the normal private-network/WAN stack by default. Traffic is on the VPN **only**
|
|
if a service explicitly opts in with `egress-class=vpn`.
|
|
2. **The VPN tunnel is never a default route.** Not for the node, not for any pod. `tun0` carries **only** the
|
|
flows a `CiliumEgressGatewayPolicy` explicitly selects (`egress-class=vpn`). The host tunnel is split-tunnel
|
|
(`route-noexec`/policy routing) precisely to guarantee this.
|
|
3. **The kill-switch blackhole is scoped to vpn-class only.** When `tun0` is down, the nftables rule drops
|
|
**only** the `egress-class=vpn` (Cilium-egress-gateway-marked) traffic. It MUST NOT match or drop default/
|
|
`internet`-class traffic or node traffic. `internet`-class egress is unaffected whether the tunnel is up
|
|
or down.
|
|
4. **Opposite fail directions.** Default/`internet` traffic **fails open** (toward the WAN — never blackholed).
|
|
`egress-class=vpn` traffic **fails closed** (toward blackhole — never leaks to the WAN). A misconfiguration
|
|
or partial failure must degrade in these directions, never the reverse.
|
|
|
|
These are gated by the `egress-class` (default/internet shows WAN IP; vpn shows VPN IP) and `vpn-killswitch`
|
|
(tunnel down ⇒ vpn-class blackholed, internet-class unaffected) conformance tests.
|
|
|
|
## Consequences
|
|
- Steering is **Cilium-native and declarative** (a `CiliumEgressGatewayPolicy` in Git) and touches no app pods
|
|
— flipping a service's `egress-class` label re-routes it with no workload rearchitecting.
|
|
- **Two-layer ownership:** the tunnel is host infra (**Ansible** role on the gateway nodes); the steering
|
|
policy is **Flux/GitOps**. Consistent with how k3s is provisioned. This is the intended split, not a leak.
|
|
- **Secrets move host-side:** PIA credentials + `.ovpn` live on the gateway node(s) via **Ansible vault**, not
|
|
a cluster SOPS Secret — see docs/07. (SOPS still governs in-cluster secrets.)
|
|
- **Gluetun is dropped** as the gateway. Its validated custom-mode config + the working PIA `.ovpn` transfer
|
|
directly to the host OpenVPN service, so the connectivity work is retained; only the packaging changes.
|
|
- **Port forwarding** (PIA NAT-PMP for qBittorrent) must be re-solved host-side (a small pf/natpmp helper that
|
|
publishes the assigned port to the app) — Phase 6, as before.
|
|
- **SPOF** on a single gateway node unless run HA (two nodes + Cilium failover).
|
|
- Validated end-to-end by the **`egress-class`** (R-NET-05/07) and **`vpn-killswitch`** (R-NET-06) conformance
|
|
tests: vpn-class egress shows the PIA IP, internet-class shows the WAN IP, a git label flip re-routes, and
|
|
forcing `tun0` down drops vpn-class egress (no leak) while internet-class is unaffected.
|
|
|
|
## Alternatives considered
|
|
- **In-cluster Gluetun pod + Cilium Egress Gateway → its `tun0`** (ADR-0013's deferred option 1) — NON-VIABLE:
|
|
`hostNetwork` Gluetun hijacks the node default route.
|
|
- **pod-gateway pattern** (VXLAN overlay + admission webhook + gateway NAT) — the accepted *in-cluster*
|
|
alternative; works but heavier, mutates client pods, bypasses Cilium's datapath. Rejected for a Cilium
|
|
cluster in favor of the native node-level egress gateway.
|
|
- **Per-pod Gluetun sidecar** (ADR-0008 Pattern A) — rejected earlier: privilege colocation with the workload
|
|
and no declarative east-west policy.
|
|
- **Keeping the working in-cluster Gluetun pod for its OWN egress only** — retained as the fallback if the
|
|
node-level tunnel proves troublesome, but it cannot serve *other* pods, which is the whole `egress: vpn`
|
|
requirement.
|