docs(vpn-egress): reflect ADR-0015 node-tunnel egress, retire gluetun narrative
The in-cluster Gluetun gateway is retired; VPN egress is now a node-level host OpenVPN split-tunnel (Ansible role vpn_egress_gateway) steered by a Cilium Egress Gateway policy with a host nftables kill-switch (ADR-0015). Credentials moved host-side to the Ansible vault. Update docs/comments only: - platform/vpn-egress/README.md: rewrite for ADR-0015 (live + conformance-green); drop the resolved "option 1 not viable with Gluetun" finding to a brief history note. - platform/kustomization.yaml: Phase 3 comment -> node tunnel + Cilium steering. - platform/kyverno/.../require-egress-class-label.yaml: ADR-0013 -> ADR-0015 steering. - schemas/servicedefinition.schema.json: egress=vpn description -> node tunnel + Cilium. - clusters/base/cluster-vars.yaml: reframe PIA_REGION / CLUSTER_POD_CIDR WHY comments to ADR-0015 (Cilium excludedCIDRs); both keys retained (still referenced/documented). - platform/vpn-egress/namespace.yaml: no privileged gateway pod under ADR-0015. - pia-credentials.sops.yaml / pia-custom-ovpn.sops.yaml: comment headers only note they are now UNUSED (creds host-side), retained pending cleanup; payloads untouched. No runtime manifest behavior, alert exprs, kustomization resources, or SOPS payloads changed. No spec-repo edits. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
1 parent
9927b73a5a
commit
aafb4ca378
8 files changed
+88
-96
No files matched your search
@@ -1,70 +1,55 @@
|
||||
# platform/vpn-egress — shared VPN egress gateway (Phase 3, ADR-0013)
|
||||
# platform/vpn-egress — VPN egress steering (Phase 3, ADR-0015)
|
||||
|
||||
Single shared **Gluetun** gateway: **PIA over OpenVPN-UDP** (ADR-0013 amendment — Gluetun does not
|
||||
support PIA over WireGuard), built-in kill-switch, native port forwarding. `vpn`-class services
|
||||
(sonarr, radarr, jackett, qbittorrent) egress **only** through this tunnel; `internet`-class services
|
||||
go out the normal WAN. Serves **R-NET-05/06/07**.
|
||||
`vpn`-class services (sonarr, radarr, jackett, qbittorrent) egress **only** through a PIA VPN
|
||||
tunnel; `internet`-class services go out the normal WAN. Serves **R-NET-05/06/07**.
|
||||
|
||||
> **STATUS: enabled in `platform/kustomization.yaml` (Phase 3).** `pia-credentials` SOPS secret is
|
||||
> provisioned. Remaining work: validate the tunnel comes up + correct egress IP, then the
|
||||
> egress-steering mechanism below (the hard part) and the kill-switch conformance test.
|
||||
Egress is realized per **[ADR-0015](../../../home-network-k3s/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md)**
|
||||
(spec `docs/03`): a **node-level host OpenVPN split-tunnel** on a labeled egress-gateway node, steered
|
||||
by a **Cilium Egress Gateway** policy, with a **host nftables kill-switch**. There is **no in-cluster
|
||||
gateway pod** — the tunnel is host infrastructure, not a Flux workload.
|
||||
|
||||
## Pieces
|
||||
- `gluetun-gateway.yaml` — the Gluetun Deployment (`replicas: 1`, `Recreate`) + the `gluetun-control`
|
||||
Service (control API :8000 for forwarded-port lookup + tunnel-state health). Kill-switch = `FIREWALL=on`.
|
||||
- `pia-credentials` (SOPS) — PIA username/password only; Gluetun ships PIA's OpenVPN configs.
|
||||
> **STATUS: live + conformance-green.** The node tunnel was validated end-to-end; the `egress-class`
|
||||
> (R-NET-05/07) and `vpn-killswitch` (R-NET-06) conformance tests pass against the live cluster.
|
||||
|
||||
## Operator hand-off (when deploying Phase 3)
|
||||
1. Provision `pia-credentials` (see `bootstrap/sops/secret-templates/pia-credentials.yaml.example`,
|
||||
namespace `vpn-egress`, keys `username`/`password`) → `sops -e -i` → uncomment it in `kustomization.yaml`.
|
||||
2. cluster-vars (already defined in `clusters/base/cluster-vars.yaml`; a ConfigMap — values are plain
|
||||
strings under `data:`, injected into the `${...}` tokens by Flux `postBuild.substituteFrom`). Confirm
|
||||
or override:
|
||||
- `PIA_REGION` — a **port-forwarding-capable** PIA region (default `Sweden`; e.g. `Netherlands`,
|
||||
`CA Toronto`, `Switzerland`; NOT a US region — those lack PF). Gluetun `SERVER_REGIONS`.
|
||||
- `CLUSTER_POD_CIDR` — the **Cilium cluster-pool** pod CIDR (default `10.0.0.0/8`; pods get `10.x`,
|
||||
confirmed live — NOT the k3s `10.42.0.0/16`). Gluetun `FIREWALL_OUTBOUND_SUBNETS`.
|
||||
Per-env overrides go in `clusters/<env>/kustomization.yaml` as a cluster-vars JSON patch
|
||||
(`- op: replace, path: /data/PIA_REGION, value: "…"`). With the self-managing root, a `git push`
|
||||
auto-applies (no manual `kubectl apply -k`).
|
||||
3. Enable `- vpn-egress` in `platform/kustomization.yaml` (Phase 3).
|
||||
## How it works (ADR-0015)
|
||||
Three layers, two owners:
|
||||
|
||||
## Egress steering — the DESIGN DECISION (needs live validation)
|
||||
The hard part: routing `vpn`-class pods through this ONE shared tunnel. Cilium Egress Gateway
|
||||
(`CiliumEgressGatewayPolicy`) SNATs matched pods out of a **gateway node's interface** — it does not
|
||||
natively route through a *pod's* tunnel. Options considered:
|
||||
- **Tunnel — host OpenVPN (Ansible).** A labeled egress-gateway node runs a **host OpenVPN systemd
|
||||
service** (PIA over OpenVPN-UDP — provider/protocol still per ADR-0013). It is **split-tunnel**
|
||||
(`route-noexec` + policy routing): it creates `tun0` but never becomes the node default route, so the
|
||||
node's own traffic (control plane, image pulls, other pods) stays on the WAN. Managed by the Ansible
|
||||
role **`infra/ansible/roles/vpn_egress_gateway/`**, like k3s itself — not a cluster workload.
|
||||
- **Steering — Cilium Egress Gateway (GitOps).** `egress-steering.yaml` holds a
|
||||
`CiliumEgressGatewayPolicy` that selects pods labeled `homelab.keithjackson.dev/egress-class=vpn`
|
||||
(Kyverno-enforced) and SNATs them out the gateway node's `tun0`. Native Cilium eBPF — no client-pod
|
||||
changes, no sidecar, no overlay. `excludedCIDRs` keeps RFC1918 / cluster / LAN traffic on the local
|
||||
datapath; only public destinations leave via the tunnel.
|
||||
- **Kill-switch — host nftables (Ansible).** A host nftables rule on the gateway node DROPS the steered
|
||||
`vpn`-class traffic whenever `tun0` is down (fail-closed, R-NET-06), scoped to vpn-class only so
|
||||
`internet`-class and node traffic are unaffected.
|
||||
|
||||
1. **Cilium Egress Gateway → the Gluetun node's tunnel interface (recommended).** Pin Gluetun to a
|
||||
node; a `CiliumEgressGatewayPolicy` selects pods labeled `homelab.keithjackson.dev/egress-class=vpn`
|
||||
and egresses them via that node's tunnel interface (`tun0`, OpenVPN). If the tunnel drops, `tun0`
|
||||
is gone → traffic has no egress path → **fail-closed** (the kill-switch behavior R-NET-06 needs). This is the
|
||||
most Cilium-native fit but requires Gluetun's tunnel interface to be visible to the node datapath
|
||||
(hostNetwork Gluetun, or a shared netns) — the detail to VALIDATE.
|
||||
2. **Gateway-pod routing:** client pods default-route to the Gluetun pod IP (init-container +
|
||||
NET_ADMIN, or a Multus secondary interface); Gluetun forwards+NATs via the tunnel. Works without
|
||||
Cilium egress gateway but is more moving parts and pod-IP-fragile.
|
||||
3. **Per-pod Gluetun sidecar** — explicitly REJECTED by ADR-0013 (single shared gateway preferred).
|
||||
## Secrets — host-side now
|
||||
PIA credentials + the `.ovpn` live **on the gateway node via Ansible vault** (see spec `docs/07`), **not**
|
||||
a cluster SOPS Secret. SOPS still governs in-cluster secrets; VPN egress simply no longer needs one.
|
||||
|
||||
### FINDING (2026-08-23): option 1 is NOT viable with Gluetun — needs an architect decision
|
||||
With the tunnel now live (custom-mode Gluetun, PIA Stockholm egress), option 1 fails on inspection:
|
||||
`CiliumEgressGatewayPolicy interface: tun0` requires `tun0` on the **node**, which means running Gluetun
|
||||
`hostNetwork: true`. But Gluetun *owns its netns routing* — on connect its `[routing]` logic adds
|
||||
`0.0.0.0/0` via the tunnel. In hostNetwork that would **redirect the NODE's own default route through the
|
||||
VPN**, breaking k3s/etcd/kubelet on that node. Gluetun has no supported "create tun0 but don't touch the
|
||||
default route" mode, so option 1 can't expose `tun0` on the node safely. (The external-router-offload
|
||||
pattern some use — MikroTik/RouterOS policy routing per pod-subnet — is out of scope: ADR-0013 mandates an
|
||||
*in-cluster* gateway on a dumb SOHO edge.)
|
||||
The `pia-credentials.sops.yaml` and `pia-custom-ovpn.sops.yaml` secrets in this directory are **now
|
||||
UNUSED** — leftovers from the retired Gluetun pod, **retained pending a separate cleanup change**.
|
||||
|
||||
**→ Escalated.** The remaining viable path is **option 2 (gateway-pod routing)**: `egress-class=vpn` pods
|
||||
default-route to the Gluetun pod via an init-container (+NET_ADMIN), and Gluetun forwards+NATs out `tun0`.
|
||||
More moving parts, and it requires Gluetun to forward for *foreign* pods (not its native same-netns model),
|
||||
so it needs validation. Kill-switch (R-NET-06) then relies on Gluetun's `FIREWALL=on` dropping non-tunnel
|
||||
egress. Fixtures for the `egress-class`/`vpn-killswitch` conformance tests are authored; the
|
||||
`CiliumEgressGatewayPolicy` / steering datapath is intentionally NOT committed until the mechanism is chosen
|
||||
+ validated, to avoid shipping an untested (and potentially node-breaking) datapath.
|
||||
## Pieces in this directory
|
||||
- `namespace.yaml` — the `vpn-egress` namespace (now holds only the Cilium policy + the retained,
|
||||
unused secrets; no privileged gateway pod under ADR-0015).
|
||||
- `egress-steering.yaml` — the `CiliumEgressGatewayPolicy` (the steering datapath).
|
||||
- `pia-credentials.sops.yaml`, `pia-custom-ovpn.sops.yaml` — **unused**, retained pending cleanup.
|
||||
|
||||
## History (brief)
|
||||
ADR-0013 originally placed the VPN in a single shared **Gluetun pod** and deferred the steering
|
||||
mechanism. Implementation proved that a Gluetun-in-a-pod tunnel **cannot be a Cilium Egress Gateway
|
||||
target**: the policy SNATs out a *node* interface, but the pod's `tun0` lives in the pod netns, and
|
||||
running Gluetun `hostNetwork` to expose `tun0` on the node hijacks the node's default route (breaking
|
||||
k3s/etcd/kubelet). ADR-0015 resolved this by moving the tunnel to the **node** (host OpenVPN
|
||||
split-tunnel) and steering with Cilium's native egress gateway. See ADR-0015 for the full rationale.
|
||||
|
||||
## Port forwarding (qBittorrent) — follow-up
|
||||
PIA assigns a forwarded port that can change on reconnect. A small **port-sync helper**
|
||||
(`qbittorrent-natmap`) reads the current port from the `gluetun-control` API (:8000
|
||||
`/v1/openvpn/portforwarded`) and writes it into qBittorrent, keeping it hands-off (R-UPD). Authored
|
||||
alongside the qBittorrent app (Phase 6 apps / or here when qBittorrent lands).
|
||||
PIA assigns a forwarded port that can change on reconnect. Under ADR-0015 this is re-solved **host-side**
|
||||
(a small pf/natpmp helper on the gateway node that publishes the assigned port to the app) — Phase 6,
|
||||
alongside the qBittorrent app.
|
||||
Reference in new issue
Block a user