From 38fcfadbf01eb3ba539332ad7607755c16597486 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 24 Aug 2026 00:59:36 -0500 Subject: [PATCH] phase3(vpn): author Cilium Egress Gateway steering policy (ADR-0015) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add platform/vpn-egress/egress-steering.yaml — a CiliumEgressGatewayPolicy that steers homelab.keithjackson.dev/egress-class=vpn pods out a gateway node's host tun0 split-tunnel interface, per ADR-0015 (node-level tunnel + Cilium Egress Gateway). Honors all four ADR-0015 invariants: only vpn-class pods are selected (default/internet + node stay on the local/WAN path), RFC1918 + cluster ranges are excludedCIDRs so only public internet is tunneled and tun0 is never a default route, and the host nftables kill-switch (Ansible) fails vpn-class CLOSED while internet-class fails OPEN. Interface contract shared verbatim with the parallel Ansible host-tunnel role: node label homelab.keithjackson.dev/egress-gateway=true, interface tun0, client label homelab.keithjackson.dev/egress-class=vpn. Cilium Egress Gateway prerequisites confirmed enabled (egressGateway.enabled, bpf.masquerade, kubeProxyReplacement all true, [parity]) — no HelmRelease change. Intentionally NOT added to platform/vpn-egress/kustomization.yaml: the datapath stays out of the runtime until the live tunnel exists and the egress-class / vpn-killswitch conformance tests pass (README rule; DoD). Gluetun is retained as the validated fallback until then. Serves R-NET-05/06/07. Co-Authored-By: Claude Opus 4.8 --- platform/vpn-egress/egress-steering.yaml | 88 ++++++++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 platform/vpn-egress/egress-steering.yaml diff --git a/platform/vpn-egress/egress-steering.yaml b/platform/vpn-egress/egress-steering.yaml new file mode 100644 index 0000000..6112502 --- /dev/null +++ b/platform/vpn-egress/egress-steering.yaml @@ -0,0 +1,88 @@ +--- +# ============================================================================ +# Cilium Egress Gateway steering for `egress-class=vpn` pods — ADR-0015. +# Serves: R-NET-05, R-NET-06, R-NET-07. +# +# ########################################################################## +# ## NOT WIRED INTO THE RUNTIME YET — DO NOT ADD TO kustomization.yaml ## +# ## UNTIL THE HOST tun0 TUNNEL IS LIVE AND THIS DATAPATH IS VALIDATED. ## +# ########################################################################## +# +# This policy steers vpn-class pods out a gateway node's host `tun0` interface. It is INERT and +# potentially disruptive until the Ansible-managed host OpenVPN split-tunnel service actually exists +# on a node labeled `homelab.keithjackson.dev/egress-gateway=true`. If Flux shipped this before the +# tunnel is up, vpn-class pods would be SNAT'd to a non-existent interface — an unvalidated datapath. +# Per platform/vpn-egress/README.md: "the steering datapath is intentionally NOT committed [to the +# runtime kustomization] until the mechanism is chosen + validated." ADR-0015 chose the mechanism +# (node tunnel + Cilium Egress Gateway); this file IS that datapath, authored but held out of the +# resource list for the PM to enable after a live-tunnel validation (see TODO at the bottom). +# +# --------------------------------------------------------------------------- +# PREREQUISITES CONFIRMED (2026-08-24): +# * Cilium Egress Gateway is ENABLED per ADR-0002 — platform/cilium/helmrelease.yaml and +# platform/cilium/values.yaml both set (all [parity] with the Ansible bootstrap render): +# egressGateway.enabled: true +# bpf.masquerade: true (Egress Gateway REQUIRES eBPF masquerade) +# kubeProxyReplacement: true +# No Cilium HelmRelease change is needed for this policy. (If any of the three were false this +# would be an ESCALATION, not a silent HelmRelease edit — they are all true, so we proceed.) +# +# INTERFACE CONTRACT (shared verbatim with the parallel Ansible host-tunnel role — ADR-0015): +# * Gateway node label ...... homelab.keithjackson.dev/egress-gateway=true (applied by Ansible) +# * Host tunnel interface .... tun0 (split-tunnel OpenVPN) +# * Client selection ......... homelab.keithjackson.dev/egress-class=vpn (Kyverno-enforced) +# --------------------------------------------------------------------------- +apiVersion: cilium.io/v2 +kind: CiliumEgressGatewayPolicy +metadata: + name: vpn-egress-steering + annotations: + homelab.keithjackson.dev/requirements: R-NET-05,R-NET-06,R-NET-07 + homelab.keithjackson.dev/adr: ADR-0015 +spec: + # --- Client selection (INVARIANTS 1 & 2) -------------------------------- + # ONLY pods explicitly opted into the vpn class are matched. Default/internet-class pods and node + # traffic are never selected, so they stay on the local/private-WAN path (invariant 1: default is + # the local path). Because ONLY these flows are steered, tun0 never becomes a default route for the + # node or for any other pod (invariant 2). The selector is intentionally cluster-wide (no namespace + # scope) so the class label alone is the contract — every namespace's vpn-class pods route here. + selectors: + - podSelector: + matchLabels: + homelab.keithjackson.dev/egress-class: vpn + # --- What gets tunneled (INVARIANT 1: only PUBLIC internet leaves via the VPN) ------------------ + # destinationCIDRs is the full internet; excludedCIDRs carves out everything that must stay LOCAL: + # all RFC1918 private ranges (which also cover the Cilium cluster-pool pod CIDR 10.0.0.0/8, the k3s + # service/pod ranges, and the LAN 192.168.x/10.73.x node network). So intra-cluster east-west, + # pod->service, and LAN traffic are NOT SNAT'd out tun0 — only genuinely public destinations are. + # Excluded flows fall through to Cilium's normal local datapath (invariant 1). + destinationCIDRs: + - "0.0.0.0/0" + excludedCIDRs: + - "10.0.0.0/8" # RFC1918 + Cilium cluster-pool pod CIDR (CLUSTER_POD_CIDR) + k3s service net + - "172.16.0.0/12" # RFC1918 + - "192.168.0.0/16" # RFC1918 (home LAN / node network family) + # --- Where it egresses -------------------------------------------------- + # The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`. When + # tun0 is absent/down, Cilium has no egress interface here; the host nftables kill-switch (Ansible, + # ADR-0015) is the authoritative fail-closed backstop that DROPS this vpn-class traffic — scoped to + # vpn-class only, so default/internet-class is unaffected (invariants 3 & 4: vpn fails CLOSED, + # internet fails OPEN). + egressGateway: + nodeSelector: + matchLabels: + homelab.keithjackson.dev/egress-gateway: "true" + interface: tun0 +# --------------------------------------------------------------------------- +# TODO(PM — enable after live validation, then retire Gluetun): +# 1. Confirm the Ansible host-tunnel role has brought `tun0` up on a node labeled +# homelab.keithjackson.dev/egress-gateway=true, and the host nftables kill-switch is in place. +# 2. Add `- egress-steering.yaml` to platform/vpn-egress/kustomization.yaml so Flux ships it. +# 3. Run the `egress-class` conformance test: vpn-class pod shows the PIA IP, internet-class shows +# the WAN IP, and a git label flip re-routes (R-NET-05/07). +# 4. Run the `vpn-killswitch` conformance test: force tun0 down => vpn-class blackholed (no leak), +# internet-class unaffected (R-NET-06). +# 5. Only AFTER both pass green: retire the Gluetun Deployment (remove gluetun-gateway.yaml from +# kustomization.yaml) per ADR-0015 ("Gluetun is dropped as the gateway"). Do NOT retire it in +# the same change that enables this — keep the validated fallback until steering is proven. +# ---------------------------------------------------------------------------