Home-Network/home-network-k3s
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ADR-0015: VPN egress steering — node-level tunnel + Cilium Egress Gateway

Browse Source 
The in-cluster Gluetun-pod gateway (ADR-0013) cannot be steered: a pod's tun0 is
invisible to the node datapath, and hostNetwork Gluetun hijacks the node's own
default route (breaks k3s/etcd). The tunnel itself works (PIA egress validated),
but not the steering.

New decision (ADR-0015): move the tunnel to a HOST OpenVPN systemd service on
egress-gateway node(s) (Ansible-managed, split-tunnel so it doesn't touch the node
default route), steer egress-class=vpn pods with a CiliumEgressGatewayPolicy (native
eBPF, no client-pod changes), and enforce the kill-switch with a host nftables rule
(fail-closed independent of Cilium). PIA/OpenVPN-UDP provider decision (ADR-0013)
stands; the .ovpn transfers over; PF re-solved host-side (Phase 6).

Amended for consistency: ADR-0013 (client+steering superseded), decisions/README,
docs/00, docs/01, docs/03, docs/07, agents/task-board.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-24 00:46:18 -05:00
1 parent 0ba06ec25b
commit 7598cff945
8 files changed
+130 -38
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/00-vision-and-scope.md
+3 -4
View File
@@ -208,10 +208,9 @@ implementation.
cluster**; the specific backing is **TBD later** and does not block the core platform. Plex's
metadata DB still lives on replicated Longhorn. The **off-cluster DR backup target** (for app
config, Nextcloud, Gitea, etc.) remains to be chosen at Phase 4. See [`04`](04-storage-and-state.md).
- **OQ-3 (VPN)** — **RESOLVED**: **PIA** via Gluetun, cluster-provisioned, **single shared
in-cluster tunnel** over **OpenVPN-UDP** (non-TCP), native port forwarding, rules enforced outside
the service (ADR-0013, supersedes ADR-0012's ExpressVPN choice). See
[`03`](03-networking-isolation-and-egress.md).
- **OQ-3 (VPN)** — **RESOLVED**: **PIA over OpenVPN-UDP** (non-TCP; ADR-0013). Egress is a **node-level
host tunnel steered by Cilium Egress Gateway** with a host nftables kill-switch (ADR-0015) — `vpn`-class
pods routed through it, rules enforced outside the service. See [`03`](03-networking-isolation-and-egress.md).
- **OQ-4 (TLS/DNS challenge access)** — **RESOLVED**: a **scoped Cloudflare API token**
(`Zone·DNS·Edit`, `keithjackson.dev`), SOPS-encrypted, drives cert-manager DNS-01 and ExternalDNS.
docs/01-architecture-overview.md
+2 -2
View File
@@ -15,7 +15,7 @@ choice here without a superseding ADR.**
| VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) |
| TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) |
| Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) |
| VPN egress | **Single shared in-cluster PIA/Gluetun gateway** (OpenVPN-UDP + native PF) + Cilium Egress Gateway | R-NET-05/06/07 | [ADR-0013](../decisions/adr-0013-vpn-provider-pia-gluetun-wireguard.md) (supersedes 0012/0008) | [03](03-networking-isolation-and-egress.md) |
| VPN egress | **Node-level PIA/OpenVPN tunnel** steered by **Cilium Egress Gateway** (host nftables kill-switch) | R-NET-05/06/07 | [ADR-0015](../decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md) (provider [0013](../decisions/adr-0013-vpn-provider-pia-gluetun-wireguard.md)) | [03](03-networking-isolation-and-egress.md) |
| Automated updates | **Flux image-update-automation** (images) + **Renovate** (charts) | R-UPD-01..03 | [ADR-0009](../decisions/adr-0009-automated-image-updates.md) | [11](11-automated-updates.md) |
| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (internal) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) |
| Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) |
@@ -46,7 +46,7 @@ choice here without a superseding ADR.**
│ Storage: Longhorn (replicated block) + bulk media store (see ADR-0003) │
├──────────────────────────────────────────────────────────────────────────┤
│ Network: Cilium (default-deny NetworkPolicy · Egress Gateway · Hubble) │
│ VPN egress (Gluetun sidecars / VPN gateway) · Multi-WAN routing │
│ VPN egress (node tunnel + Cilium egress GW) · Multi-WAN routing │
├──────────────────────────────────────────────────────────────────────────┤
│ Kubernetes: k3s (3 nodes, HA control plane, embedded etcd) │
├──────────────────────────────────────────────────────────────────────────┤
docs/03-networking-isolation-and-egress.md
+29 -26
View File
@@ -86,36 +86,39 @@ as multi-ISP routing but as **two egress paths selected by routing rules**:
ports) to the ingress VIP** for `public` services — a documented manual step ([09](09-dns-and-ingress.md)).
- The **service-facing contract is unchanged**: a service still just declares `egress: internet|vpn`.
## VPN egress model: single shared PIA/Gluetun gateway (R-NET-05/06 — ADR-0013, supersedes ADR-0012/0008)
## VPN egress model: node-level tunnel + Cilium Egress Gateway (R-NET-05/06 — ADR-0015; provider ADR-0013)
VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the design is a **single,
shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside
the service**:
the service**. The steering mechanism was settled by **ADR-0015** after the original in-cluster-Gluetun
approach proved un-steerable (a Gluetun *pod*'s `tun0` is invisible to the node datapath, and
`hostNetwork` Gluetun hijacks the node's own default route). The model:
- **One in-cluster VPN egress gateway** (an HA-capable Deployment) runs **Gluetun** configured for
**PIA over OpenVPN-UDP** ("non-TCP or better"; WireGuard is not supported for PIA in Gluetun —
ADR-0013 amendment). The cluster provisions and controls the tunnel
(nothing on the edge router). Gluetun provides the built-in **kill-switch/firewall**, health
checks, and auto-reconnect.
- **`vpn`-class pods are steered through the gateway** (Cilium Egress Gateway / routing); they have
**no independent egress path**. The kill-switch and allow rules are enforced at the **gateway +
namespace/policy level**, i.e. *outside* each service — the service is just labeled `egress: vpn`
and cannot leak on its own.
- **Kill-switch (fail-closed) is mandatory and enforced centrally:** if the tunnel is down, `vpn`
pods have **no route to the plain internet**; only the tunnel path exists. `internet`-class
services are unaffected.
- **Port forwarding for qBittorrent:** Gluetun **native PF** (`VPN_PORT_FORWARDING=on`) on a
**PF-enabled PIA region**; a **port-sync helper** (e.g. `qbittorrent-natmap`) writes the assigned
port into qBittorrent so it tracks reconnects — kept hands-off. Alert if PF acquisition fails.
- **Trade-off (documented):** one shared tunnel = one shared egress IP and a single failure domain
for `vpn` services → run the gateway **active/standby** for resilience; alert on tunnel state
([10](10-observability-and-alerting.md)).
- **Prerequisite (simple):** just the **PIA username/password**, stored SOPS-encrypted
([07](07-security-and-secrets.md)) — no manual `.ovpn` extraction. Validate end-to-end (tunnel +
kill-switch + PF) in dev/test.
- **Tunnel on the node, not in a pod.** One or two labeled **egress-gateway nodes** run a **host OpenVPN
systemd service** — **PIA over OpenVPN-UDP** (provider/protocol per ADR-0013; "non-TCP or better"),
**Ansible-managed host infra** (like k3s itself), NOT a Flux workload. It is **split-tunnel**: it
creates `tun0` but does **not** set the node's default route (`route-noexec` + policy routing), so the
node's own traffic stays on the WAN — only what Cilium points at `tun0` uses the VPN. The tunnel is
on the cluster nodes, not the edge router (dumb SOHO edge, per OQ-1).
- **`vpn`-class pods are steered by Cilium Egress Gateway.** A `CiliumEgressGatewayPolicy` selects pods
labeled `egress-class=vpn` and egresses them via the gateway node's `tun0` (native eBPF SNAT to the
tunnel IP). **No client-pod changes, no sidecar, no overlay.** They have **no independent egress path**;
the service is just labeled `egress: vpn` and cannot leak on its own.
- **Kill-switch (fail-closed) — host nftables.** An **nftables rule on the gateway node** DROPS steered
(`egress-class=vpn`) traffic whenever `tun0` is down — a host backstop independent of any Cilium
egress-gateway fail-open edge case. If the tunnel is down, `vpn` pods have **no route to the plain
internet**; `internet`-class services are unaffected.
- **Port forwarding for qBittorrent:** PIA NAT-PMP, re-solved **host-side** (a small pf/natpmp helper on
the gateway node that publishes the assigned forwarded port into qBittorrent) — kept hands-off. Alert
if PF acquisition fails.
- **Resilience:** two egress-gateway nodes + **Cilium Egress Gateway HA** (gateway failover) give the
active/standby resilience; alert on tunnel state ([10](10-observability-and-alerting.md)).
- **Prerequisite:** the **PIA username/password + `.ovpn`** on the gateway node(s) via **Ansible vault**
([07](07-security-and-secrets.md)) — the tunnel is host infra, so its creds are host-side (not a cluster
SOPS Secret). Validate end-to-end (tunnel + steering + kill-switch + PF) in dev/test.
**Kill-switch is an acceptance gate:** with the tunnel forcibly down, every `vpn` service must fail
closed (no packets to the plain internet). Tested, not assumed.
closed (no packets to the plain internet), while `internet`-class is unaffected. Tested, not assumed.
## Ingress (north-south) — summary
@@ -139,8 +142,8 @@ Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forward
## Resolved
- **OQ-1** — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All
segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope.
- **OQ-3** — **PIA** via Gluetun, single shared in-cluster tunnel over **OpenVPN-UDP**, native port
forwarding, rules enforced outside the service (ADR-0013, supersedes ADR-0012's ExpressVPN choice).
- **OQ-3** — **PIA over OpenVPN-UDP** (ADR-0013); egress is a **node-level host tunnel** steered by **Cilium
Egress Gateway** with a host nftables kill-switch (ADR-0015), rules enforced outside the service.
## Open questions
- Exact steering mechanism for `vpn` pods → gateway (Cilium Egress Gateway vs routing/SNAT) —
docs/07-security-and-secrets.md
+2 -1
View File
@@ -47,7 +47,8 @@ Defense in depth with declarative, auditable controls:
[AGENTS.md §5](../AGENTS.md)).
### Secret categories to manage
- VPN credentials (PIA username/password; OpenVPN-UDP) for the egress model (ADR-0013).
- VPN credentials (PIA username/password + `.ovpn`; OpenVPN-UDP) — held **host-side on the egress-gateway
node(s) via Ansible vault** (the tunnel is host infra, not a cluster pod; ADR-0015), NOT a cluster SOPS Secret.
- Keycloak admin credential and per-service **OIDC client secrets** ([15](15-identity-and-access.md)).
- DNS provider API token (Cloudflare) for ACME DNS-01 + ExternalDNS.
- SMTP credentials for Alertmanager email.