Merge feature/vpn-egress-gateway-role into main (ADR-0015 host tunnel role, not yet deployed)

infra/ansible/inventory/group_vars/vpn_egress_gateway/README.md

@@ -0,0 +1,52 @@
+# group_vars for the `vpn_egress_gateway` node(s) — ADR-0015 host tunnel secrets
+
+The node-level VPN tunnel is **host infra**, so (per ADR-0015 / docs/07) its PIA credentials
+live **host-side in Ansible vault** here — NOT in a cluster SOPS Secret. Create an encrypted
+`vault.yml` in this directory (committed, encrypted; the passphrase file stays out of the repo,
+see `ansible.cfg` `vault_password_file`).
+
+## Required vault keys (operator provides the VALUES — none are committed in plaintext)
+
+```bash
+cd infra/ansible
+ansible-vault create inventory/group_vars/vpn_egress_gateway/vault.yml
+```
+
+`vault.yml` must define exactly these three keys (the role maps them to its
+`vpn_egress_gateway_*` vars in `roles/vpn_egress_gateway/defaults/main.yml`):
+
+| Vault key | What it is |
+|---|---|
+| `vault_vpn_egress_gateway_pia_username` | PIA account username (for the `.ovpn`'s `auth-user-pass`) |
+| `vault_vpn_egress_gateway_pia_password` | PIA account password |
+| `vault_vpn_egress_gateway_pia_ovpn`     | The **full text** of the proven PIA `.ovpn` — the same `sweden-aes-128-cbc-udp-ip.ovpn` (PIA over OpenVPN-UDP, AES-128-CBC/SHA1/RSA-2048, IP-pinned Stockholm remote `158.173.241.x:1198/udp`) currently held in the k8s `pia-custom-ovpn` Secret. Use a YAML block scalar so the inline `<ca>`/`<crl-verify>` PEM survives verbatim. |
+
+Example skeleton (values are placeholders — DO NOT commit unencrypted):
+
+```yaml
+vault_vpn_egress_gateway_pia_username: "pXXXXXXX"
+vault_vpn_egress_gateway_pia_password: "REDACTED"
+vault_vpn_egress_gateway_pia_ovpn: |
+  client
+  dev tun
+  proto udp
+  remote 158.173.241.36 1198
+  cipher aes-128-cbc
+  auth sha1
+  auth-user-pass
+  ...
+  <ca>
+  -----BEGIN CERTIFICATE-----
+  ...
+  -----END CERTIFICATE-----
+  </ca>
+```
+
+> To lift the exact `.ovpn` out of the existing cluster Secret:
+> `sops -d platform/vpn-egress/pia-custom-ovpn.sops.yaml` → the `sweden-aes-128-cbc-udp-ip.ovpn`
+> value is the text to paste under `vault_vpn_egress_gateway_pia_ovpn`.
+
+## Notes
+- `vault.yml` is skipped by the yamllint gate (`ci/yamllint.yaml` ignores `**/vault.yml`).
+- This is the Ansible-bootstrap-layer secret (host credential), distinct from the platform's
+  in-cluster SOPS + age secrets (ADR-0007) — different layer, different tool, by design (ADR-0015).

infra/ansible/inventory/hosts.ini

@@ -20,3 +20,10 @@ k3s_secondary
 # (sudo) password comes from the vaulted group_vars/hypervisor/vault.yml.
 [hypervisor]
 localhost ansible_connection=local
+
+# VPN egress-gateway node(s) — run the node-level split-tunnel OpenVPN + kill-switch
+# (ADR-0015, playbooks/vpn.egress.gateway.yml). Initially just k3s-dev-3 (single gateway;
+# add a second host here for Cilium Egress Gateway HA). Members are already defined above
+# with their ansible_host under k3s_secondary; this group only tags the egress role target.
+[vpn_egress_gateway]
+k3s-dev-3

infra/ansible/playbooks/vpn.egress.gateway.yml

@@ -0,0 +1,18 @@
+---
+# Stand up the node-level VPN egress tunnel on the designated egress-gateway node(s)
+# (ADR-0015). Host infra — the split-tunnel OpenVPN service + policy routing + scoped
+# nftables kill-switch. The CiliumEgressGatewayPolicy that steers vpn-class pods to this
+# node's tun0 is authored separately as GitOps (platform/), against the same contract:
+#   * inventory group `vpn_egress_gateway`  (this play's target)
+#   * k8s node label   homelab.keithjackson.dev/egress-gateway=true  (set by the role)
+#   * host tunnel iface tun0                 (split-tunnel; never the node default route)
+#
+# Secrets come from Ansible vault (inventory/group_vars/vpn_egress_gateway/vault.yml):
+#   vault_vpn_egress_gateway_pia_username, vault_vpn_egress_gateway_pia_password, vault_vpn_egress_gateway_pia_ovpn
+# Run:  ansible-playbook playbooks/vpn.egress.gateway.yml
+- name: Configure the node-level VPN egress gateway
+  hosts: vpn_egress_gateway
+  become: true
+  gather_facts: true          # package module + service facts need os facts
+  roles:
+    - role: vpn_egress_gateway

infra/ansible/roles/vpn_egress_gateway/README.md

@@ -0,0 +1,101 @@
+# role: vpn_egress_gateway (ADR-0015)
+
+Stands up the **node-level VPN tunnel** on the egress-gateway node(s): a host **OpenVPN
+systemd service** (PIA over OpenVPN-UDP), **split-tunnel** via policy routing, and a
+**scoped nftables kill-switch**. This is the host half of ADR-0015; the
+`CiliumEgressGatewayPolicy` that steers `egress-class=vpn` pods to this node's `tun0` is
+separate GitOps authored against the shared **interface contract** below.
+
+> **CONFORMANCE IS GATED ON A LIVE TUNNEL.** This role cannot be proven green without the
+> operator's PIA host credentials (see "What requires a live tunnel"). Authoring + lint pass
+> in CI; the `egress-class` / `vpn-killswitch` conformance tests (R-NET-05/06/07) go green only
+> once the tunnel is live in dev/test.
+
+## Interface contract (shared with the Cilium-policy agent)
+- **Inventory group** `vpn_egress_gateway` — the egress-gateway node(s) (initially `k3s-dev-3`).
+- **k8s node label** `homelab.keithjackson.dev/egress-gateway=true` — set by this role; the
+  `CiliumEgressGatewayPolicy` selects the gateway node by it.
+- **Host tunnel iface** `tun0` — the Cilium policy SNATs vpn-class pods out of it.
+- **Optional fwmark** `0x1c8` — a contract *hook* (see kill-switch layering) the Cilium side
+  MAY stamp on vpn-class packets for a tunnel-down-independent host backstop.
+
+## How the four ADR-0015 invariants are met
+
+### 1 & 2 — split-tunnel; tun0 is never a default route
+`templates/vpn-egress.conf.j2` includes the operator's proven `.ovpn` and layers on
+`route-noexec` + `pull-filter ignore "redirect-gateway"`, so **OpenVPN installs no routes and
+never touches the main routing table**. The node's own default route stays on the WAN whether
+the tunnel is up or down — k3s/etcd/kubelet are unaffected.
+
+Steering is done with **policy routing** in a dedicated table (`51820`, named `vpn_egress`):
+- The OpenVPN **up-script** adds `default dev tun0` to table `51820` and an ip-rule
+  **`from <tun0-addr> lookup 51820`** (priority 5000). Only packets whose source is the tunnel
+  address — i.e. the vpn-class pods Cilium has SNAT'd to `tun0` — consult that table. Nothing
+  else does, so nothing else is re-routed.
+- The **down-script** removes the ip-rule and the tunnel default (leaving the blackhole).
+
+### 3 & 4 — kill-switch scoped to vpn-class, opposite fail directions
+Two independent layers, both scoped so they can only ever affect vpn-class traffic:
+
+1. **Routing blackhole (fail-closed fallback).** Table `51820` always carries a permanent
+   `blackhole default` (installed by `vpn-egress-killswitch.service`, metric 4000). When the
+   tunnel default is withdrawn (flap / partial teardown) while the ip-rule still steers, packets
+   hit the blackhole instead of falling through to the WAN.
+2. **nftables backstop (`templates/vpn-egress-killswitch.nft.j2`).** Own `inet
+   vpn_egress_gateway_killswitch` table, base chain **policy `accept`** (so it can only *drop* the
+   narrow matches, never break anything). In `postrouting`:
+   - `oifname tun0 ip saddr @vpn_egress_gateway_src accept` — tunnel up: let steered traffic out.
+   - `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` — **fail closed:** a tunnel-sourced packet
+     trying to leave any other interface is dropped (no WAN leak).
+   - `meta mark 0x1c8 oifname != tun0 drop` — the optional fwmark backstop.
+
+`@vpn_egress_gateway_src` holds the **live tun0 address** (added by the up-script, flushed by the
+down-script). Node traffic and `internet`-class traffic are never in the set, never carry the
+mark, and are never targeted — so they always hit `policy accept` and **fail open toward the
+WAN** (invariant 4), tunnel up or down. The opposite directions are structural, not incidental.
+
+## Why the node default route is provably unaffected (design-level verification)
+- OpenVPN runs with `route-noexec` and `pull-filter ignore "redirect-gateway"` → it adds **zero**
+  routes and cannot install a `0.0.0.0/0` via `tun0`. (This is exactly the failure mode that made
+  `hostNetwork` gluetun non-viable in ADR-0015; here OpenVPN is explicitly told not to route.)
+- All tunnel routing lives in **table 51820 only**; the up/down scripts touch that table and
+  ip-rules, never `table main` / `table default`.
+- The ip-rule matches **`from <tun0-addr>`** — an address the node itself never sources traffic
+  from — so node-originated and internet-class packets never enter the tunnel table.
+- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src` / the fwmark.
+- Live check (post-deploy): `ip route show table main default` still points at the WAN NIC, and
+  `curl ipinfo` from the node shows the WAN IP, with the tunnel both up and forced down.
+
+## Kill-switch layering & ESCALATION (honest limit)
+- The nft `@vpn_egress_gateway_src` rule and the routing blackhole robustly cover the case where the
+  tunnel is up but a vpn-class packet is (mis)routed toward the WAN, and flaps.
+- The **tunnel fully-down** case relies on **Cilium Egress Gateway itself failing closed** when
+  its egress interface `tun0` is absent (Cilium 1.16 drops egress-gateway-marked traffic with no
+  valid egress path rather than falling back to the node's default route) **plus** the routing
+  blackhole. That is the ADR's expected behavior.
+- For a host backstop that is **fully independent of Cilium even in a fail-open regression** when
+  the tunnel is down (no tun0 address to key on), the host needs a stable marker on vpn-class
+  packets. The only contract-available marker is the **`0x1c8` fwmark hook** — which the
+  *Cilium-policy* side must stamp. **ESCALATION for the PM:** either (a) confirm the Cilium
+  Egress Gateway 1.16 fail-closed + routing-blackhole guarantee is accepted as sufficient for
+  R-NET-06, or (b) add the shared fwmark to the interface contract so the CiliumEgressGatewayPolicy
+  marks vpn-class traffic and this role's fwmark rule becomes the tunnel-down-independent backstop.
+  This is flagged rather than worked around — no spec was weakened to reach done.
+
+## What requires a LIVE tunnel to validate (cannot be tested without operator creds)
+1. The PIA `.ovpn` + username/password actually authenticate and `tun0` comes up as a host iface.
+2. `egress-class`: a vpn-class pod's public IP is the PIA Stockholm IP; internet-class shows the WAN IP.
+3. `vpn-killswitch`: forcing `tun0`/the tunnel service down blackholes vpn-class egress (no leak)
+   while internet-class is unaffected — the R-NET-06 fault-injection gate.
+4. That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped
+   rules actually see the packets (eBPF datapath interaction — see escalation).
+5. MTU/PMTU across `pod → tun0 → PIA` (in-pod gluetun needed 1300 for Cilium VXLAN; on the node
+   the tunnel rides the physical WAN, so `vpn_egress_gateway_tun_mtu` defaults empty — verify live).
+6. Port forwarding (PIA NAT-PMP) for qBittorrent — ADR-0015 Phase-6 follow-up, not in this role.
+
+## Files
+- `defaults/main.yml` — all `vpn_egress_gateway_*` vars; the three vault-backed credential vars.
+- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, node label.
+- `handlers/main.yml` — daemon-reload, kill-switch reload, tunnel restart.
+- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, both systemd units.
+- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys: `inventory/group_vars/vpn_egress_gateway/README.md`.

infra/ansible/roles/vpn_egress_gateway/defaults/main.yml

@@ -0,0 +1,66 @@
+---
+# vpn_egress_gateway role defaults (ADR-0015).
+# Node-level split-tunnel host OpenVPN + policy routing + scoped nftables kill-switch.
+# All vars are role-name prefixed. NO secret VALUES live here — the three credential
+# vars below reference operator-provided Ansible-vault keys (see the role README and
+# inventory/group_vars/vpn_egress_gateway/README.md). They are intentionally NOT
+# defaulted so a missing vault fails loud rather than silently deploying a broken tunnel.
+
+# --- packages -------------------------------------------------------------------
+vpn_egress_gateway_packages:
+  - openvpn
+  - nftables
+
+# --- credentials / config (from Ansible vault — operator provides the values) ----
+# The PIA .ovpn PROVEN in Phase 3 (ADR-0013/0015): sweden-aes-128-cbc-udp-ip.ovpn —
+# PIA over OpenVPN-UDP, AES-128-CBC / SHA1 / RSA-2048, IP-pinned PIA Stockholm remote
+# (158.173.241.x:1198/udp), auth-user-pass. Same file that drove the working gluetun pod.
+vpn_egress_gateway_pia_username: "{{ vault_vpn_egress_gateway_pia_username }}"
+vpn_egress_gateway_pia_password: "{{ vault_vpn_egress_gateway_pia_password }}"
+# The FULL text of the PIA .ovpn (inline <ca>/<crl-verify>, cipher/auth, remote line).
+vpn_egress_gateway_pia_ovpn: "{{ vault_vpn_egress_gateway_pia_ovpn }}"
+
+# --- on-disk layout -------------------------------------------------------------
+vpn_egress_gateway_config_dir: /etc/openvpn/client
+vpn_egress_gateway_scripts_dir: /etc/openvpn/scripts
+vpn_egress_gateway_ovpn_path: "{{ vpn_egress_gateway_config_dir }}/pia.ovpn"
+vpn_egress_gateway_credentials_path: "{{ vpn_egress_gateway_config_dir }}/pia-credentials.txt"
+vpn_egress_gateway_wrapper_conf: "{{ vpn_egress_gateway_config_dir }}/vpn-egress.conf"
+
+# --- tunnel interface + split-tunnel policy routing (invariants 1 & 2) ----------
+# The host tunnel interface Cilium Egress Gateway steers vpn-class pods out of.
+vpn_egress_gateway_tun_iface: tun0
+# Dedicated policy-routing table for the tunnel (NEVER the main table — the node's
+# own default route stays on the WAN). Numeric id + a name added to rt_tables.
+vpn_egress_gateway_route_table_id: 51820
+vpn_egress_gateway_route_table_name: vpn_egress
+# ip-rule priority for the `from <tun0-ip> lookup vpn_egress` rule (well below main).
+vpn_egress_gateway_rule_priority: 5000
+# Route metrics inside the dedicated table: the live tunnel default is preferred;
+# a permanent blackhole is the fail-closed fallback during flaps / partial teardown.
+vpn_egress_gateway_tun_route_metric: 50
+vpn_egress_gateway_blackhole_metric: 4000
+
+# Optional tun MTU override. Empty => honor whatever the .ovpn/PIA negotiates. On the
+# NODE the tunnel rides the physical WAN (1500), so unlike the in-pod gluetun case
+# (Cilium VXLAN 1450 → needed 1300) no shrink is required by default.
+vpn_egress_gateway_tun_mtu: ""
+
+# --- nftables kill-switch (invariants 3 & 4 — scoped to vpn-class ONLY) ----------
+vpn_egress_gateway_nft_table: vpn_egress_gateway_killswitch
+# Named set holding the LIVE tun0 source address (maintained by the up/down scripts).
+# The kill-switch matches ONLY packets SNAT'd to this address by Cilium Egress Gateway
+# — i.e. exactly the egress-class=vpn traffic. Node / internet-class traffic is never
+# in this set and is never matched (fails OPEN toward the WAN).
+vpn_egress_gateway_nft_src_set: vpn_egress_gateway_src
+# Contract-hook fwmark: if the CiliumEgressGatewayPolicy side stamps vpn-class packets
+# with this mark, the kill-switch also drops marked packets not leaving via tun0 — an
+# extra, tunnel-down-independent backstop. See README "Kill-switch layering / escalation".
+vpn_egress_gateway_fwmark: "0x1c8"
+
+# --- kubernetes node label (steering contract) ----------------------------------
+# The label the parallel CiliumEgressGatewayPolicy selects the gateway node by.
+vpn_egress_gateway_node_label_key: homelab.keithjackson.dev/egress-gateway
+vpn_egress_gateway_node_label_value: "true"
+# k3s ships its kubeconfig here; the gateway node is a k3s server so `k3s kubectl` works.
+vpn_egress_gateway_kubeconfig: /etc/rancher/k3s/k3s.yaml

infra/ansible/roles/vpn_egress_gateway/handlers/main.yml

@@ -0,0 +1,14 @@
+---
+- name: Reload systemd
+  ansible.builtin.systemd_service:
+    daemon_reload: true
+
+- name: Reload vpn-egress kill-switch
+  ansible.builtin.systemd_service:
+    name: vpn-egress-killswitch.service
+    state: restarted
+
+- name: Restart vpn-egress tunnel
+  ansible.builtin.systemd_service:
+    name: vpn-egress-tunnel.service
+    state: restarted

infra/ansible/roles/vpn_egress_gateway/tasks/main.yml

@@ -0,0 +1,139 @@
+---
+# vpn_egress_gateway — node-level split-tunnel PIA OpenVPN + policy routing + scoped
+# nftables kill-switch on the egress-gateway node(s) (ADR-0015). Host infra, Ansible-
+# managed like k3s itself; the CiliumEgressGatewayPolicy that steers pods is separate GitOps.
+
+- name: Install OpenVPN and nftables
+  ansible.builtin.package:
+    name: "{{ vpn_egress_gateway_packages }}"
+    state: present
+
+- name: Register the dedicated policy-routing table name
+  ansible.builtin.lineinfile:
+    path: /etc/iproute2/rt_tables
+    line: "{{ vpn_egress_gateway_route_table_id }} {{ vpn_egress_gateway_route_table_name }}"
+    regexp: "^{{ vpn_egress_gateway_route_table_id }}\\s"
+    state: present
+    create: true
+    mode: "0644"
+
+- name: Ensure the OpenVPN config and scripts directories exist
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0750"
+  loop:
+    - "{{ vpn_egress_gateway_config_dir }}"
+    - "{{ vpn_egress_gateway_scripts_dir }}"
+
+- name: Deploy the operator's proven PIA .ovpn (from Ansible vault)
+  ansible.builtin.copy:
+    content: "{{ vpn_egress_gateway_pia_ovpn }}"
+    dest: "{{ vpn_egress_gateway_ovpn_path }}"
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+  notify: Restart vpn-egress tunnel
+
+- name: Deploy the PIA auth-user-pass credentials (from Ansible vault)
+  ansible.builtin.template:
+    src: pia-credentials.txt.j2
+    dest: "{{ vpn_egress_gateway_credentials_path }}"
+    owner: root
+    group: root
+    mode: "0600"
+  no_log: true
+  notify: Restart vpn-egress tunnel
+
+- name: Deploy the split-tunnel OpenVPN wrapper config
+  ansible.builtin.template:
+    src: vpn-egress.conf.j2
+    dest: "{{ vpn_egress_gateway_wrapper_conf }}"
+    owner: root
+    group: root
+    mode: "0600"
+  notify: Restart vpn-egress tunnel
+
+- name: Deploy the split-tunnel up/down policy-routing scripts
+  ansible.builtin.template:
+    src: "{{ item }}.j2"
+    dest: "{{ vpn_egress_gateway_scripts_dir }}/{{ item }}"
+    owner: root
+    group: root
+    mode: "0750"
+  loop:
+    - vpn-egress-up.sh
+    - vpn-egress-down.sh
+  notify: Restart vpn-egress tunnel
+
+- name: Deploy the scoped nftables kill-switch ruleset
+  ansible.builtin.template:
+    src: vpn-egress-killswitch.nft.j2
+    dest: "{{ vpn_egress_gateway_config_dir }}/vpn-egress-killswitch.nft"
+    owner: root
+    group: root
+    mode: "0600"
+  notify: Reload vpn-egress kill-switch
+
+- name: Deploy the kill-switch systemd unit
+  ansible.builtin.template:
+    src: vpn-egress-killswitch.service.j2
+    dest: /etc/systemd/system/vpn-egress-killswitch.service
+    owner: root
+    group: root
+    mode: "0644"
+  notify:
+    - Reload systemd
+    - Reload vpn-egress kill-switch
+
+- name: Deploy the tunnel systemd unit
+  ansible.builtin.template:
+    src: vpn-egress-tunnel.service.j2
+    dest: /etc/systemd/system/vpn-egress-tunnel.service
+    owner: root
+    group: root
+    mode: "0644"
+  notify:
+    - Reload systemd
+    - Restart vpn-egress tunnel
+
+- name: Flush handlers so units/routing exist before enable+start
+  ansible.builtin.meta: flush_handlers
+
+- name: Enable and start the kill-switch (must be up before the tunnel)
+  ansible.builtin.systemd_service:
+    name: vpn-egress-killswitch.service
+    enabled: true
+    state: started
+
+- name: Enable and start the split-tunnel VPN service
+  ansible.builtin.systemd_service:
+    name: vpn-egress-tunnel.service
+    enabled: true
+    state: started
+
+# --- steering contract: label the k8s node the Cilium Egress Gateway policy selects ----
+- name: Read the current egress-gateway node label
+  ansible.builtin.command:
+    cmd: >-
+      k3s kubectl get node {{ inventory_hostname }}
+      -o jsonpath={.metadata.labels.{{ vpn_egress_gateway_node_label_key | replace('.', '\\.') | replace('/', '\\/') }}}
+  environment:
+    KUBECONFIG: "{{ vpn_egress_gateway_kubeconfig }}"
+  register: vpn_egress_gateway_label_current
+  changed_when: false
+  failed_when: false
+
+- name: Label this node as the VPN egress gateway
+  ansible.builtin.command:
+    cmd: >-
+      k3s kubectl label node {{ inventory_hostname }}
+      {{ vpn_egress_gateway_node_label_key }}={{ vpn_egress_gateway_node_label_value }} --overwrite
+  environment:
+    KUBECONFIG: "{{ vpn_egress_gateway_kubeconfig }}"
+  register: vpn_egress_gateway_label_apply
+  changed_when: "'not labeled' not in vpn_egress_gateway_label_apply.stdout"
+  when: vpn_egress_gateway_label_current.stdout | default('') != vpn_egress_gateway_node_label_value

infra/ansible/roles/vpn_egress_gateway/templates/pia-credentials.txt.j2

@@ -0,0 +1,2 @@
+{{ vpn_egress_gateway_pia_username }}
+{{ vpn_egress_gateway_pia_password }}

infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# {{ ansible_managed }}
+# OpenVPN --down hook (with `down-pre`, runs BEFORE tun0 is destroyed). Tears down the
+# split-tunnel steering so vpn-class traffic FAILS CLOSED, never leaks to the WAN
+# (ADR-0015 invariants 3 & 4).
+#
+# Order matters: (1) empty the kill-switch source set and drop the ip-rule so no packet
+# can be routed toward a vanishing tun0, then (2) remove the tunnel default from the
+# dedicated table. The permanent `blackhole default` in that table is LEFT in place as
+# the fail-closed fallback. The node's main-table WAN default is never touched here.
+set -uo pipefail
+
+TABLE_ID="{{ vpn_egress_gateway_route_table_id }}"
+RULE_PRIO="{{ vpn_egress_gateway_rule_priority }}"
+NFT_TABLE="{{ vpn_egress_gateway_nft_table }}"
+NFT_SET="{{ vpn_egress_gateway_nft_src_set }}"
+IFACE="${dev:-{{ vpn_egress_gateway_tun_iface }}}"
+SRC="${ifconfig_local:-}"
+
+# 1) Stop scoping/steering to the (now dead) tunnel source.
+nft flush set inet "${NFT_TABLE}" "${NFT_SET}" 2>/dev/null || true
+if [ -n "${SRC}" ]; then
+  while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done
+fi
+
+# 2) Remove the tunnel default from the dedicated table; keep the blackhole fallback.
+ip route del default dev "${IFACE}" table "${TABLE_ID}" 2>/dev/null || true
+
+echo "vpn-egress-down: steering removed; vpn-class fails closed (blackhole retained)"

infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2

@@ -0,0 +1,40 @@
+#!/usr/sbin/nft -f
+# {{ ansible_managed }}
+# ADR-0015 kill-switch (invariants 3 & 4) — SCOPED TO vpn-class ONLY, fails CLOSED.
+#
+# Own dedicated table so re-loading it never disturbs Cilium's or the node's own rules.
+# The base chains are policy ACCEPT: this table can ONLY ever drop the narrow vpn-class
+# traffic matched below; node traffic and internet-class traffic are untouched and keep
+# egressing the WAN (fail OPEN — invariant 4). It matches on:
+#   * @{{ vpn_egress_gateway_nft_src_set }} — the LIVE tun0 address Cilium Egress Gateway SNATs
+#     vpn-class pods to (populated by the OpenVPN up-script; emptied by the down-script).
+#     This IS the egress-class=vpn traffic and nothing else.
+#   * meta mark {{ vpn_egress_gateway_fwmark }} — optional contract-hook: a tunnel-DOWN-independent
+#     backstop, active only if the CiliumEgressGatewayPolicy side stamps this fwmark.
+
+table inet {{ vpn_egress_gateway_nft_table }} {
+    # Live tunnel source address(es). Empty whenever the tunnel is down => the two
+    # saddr rules below match nothing => vpn-class has no path at all (fail closed).
+    set {{ vpn_egress_gateway_nft_src_set }} {
+        type ipv4_addr
+    }
+
+    chain postrouting {
+        # oifname is resolved at postrouting; SNAT/masquerade has already applied here,
+        # so the Cilium-SNAT'd vpn-class packet's saddr is the tunnel address.
+        type filter hook postrouting priority 100; policy accept;
+
+        # Tunnel UP: let the steered vpn-class traffic out tun0.
+        oifname "{{ vpn_egress_gateway_tun_iface }}" ip saddr @{{ vpn_egress_gateway_nft_src_set }} accept
+
+        # FAIL CLOSED: a vpn-class (tunnel-sourced) packet trying to leave ANY other
+        # interface (i.e. the WAN) is dropped — never leaks. Scoped by the set, so it
+        # can never match node or internet-class traffic.
+        ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != "{{ vpn_egress_gateway_tun_iface }}" drop
+
+        # Tunnel-down-independent backstop (contract hook, see role README): if vpn-class
+        # packets carry the agreed fwmark, drop any not leaving via tun0 even when the set
+        # is empty (tunnel fully down). Harmless if the Cilium side does not set the mark.
+        meta mark {{ vpn_egress_gateway_fwmark }} oifname != "{{ vpn_egress_gateway_tun_iface }}" drop
+    }
+}

infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.service.j2

@@ -0,0 +1,25 @@
+# {{ ansible_managed }}
+# Installs the ADR-0015 kill-switch state that must exist INDEPENDENTLY of (and BEFORE)
+# the tunnel: the scoped nftables table/set and the permanent fail-closed blackhole route
+# in the dedicated policy-routing table. Ordered before the tunnel so the nft set exists
+# when the OpenVPN up-script registers the live tunnel address.
+[Unit]
+Description=VPN egress gateway kill-switch (nftables + fail-closed blackhole) — ADR-0015
+Documentation=file://{{ vpn_egress_gateway_config_dir }}/README
+After=network-pre.target nftables.service
+Wants=network-pre.target
+Before=vpn-egress-tunnel.service
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+# Load the scoped kill-switch table (idempotent: delete-then-add).
+ExecStart=/usr/sbin/nft -f {{ vpn_egress_gateway_config_dir }}/vpn-egress-killswitch.nft
+# Ensure the dedicated table has a permanent blackhole default (fail closed) even before
+# the tunnel first connects. The tunnel up-script adds the preferred `default dev tun0`.
+ExecStart=/sbin/ip route replace blackhole default table {{ vpn_egress_gateway_route_table_id }} metric {{ vpn_egress_gateway_blackhole_metric }}
+# Tear down cleanly: remove the whole scoped table (never touches other nft rules).
+ExecStop=/usr/sbin/nft delete table inet {{ vpn_egress_gateway_nft_table }}
+
+[Install]
+WantedBy=multi-user.target

infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-tunnel.service.j2

@@ -0,0 +1,25 @@
+# {{ ansible_managed }}
+# Node-level host OpenVPN tunnel (ADR-0015): PIA over OpenVPN-UDP, split-tunnel. Creates
+# {{ vpn_egress_gateway_tun_iface }} but — via the wrapper's route-noexec / pull-filter — NEVER sets
+# the node default route. The node (k3s/etcd/kubelet) keeps working whether this is up or down.
+[Unit]
+Description=VPN egress gateway tunnel (PIA OpenVPN-UDP, split-tunnel) — ADR-0015
+Documentation=file://{{ vpn_egress_gateway_config_dir }}/README
+After=network-online.target vpn-egress-killswitch.service
+Wants=network-online.target
+Requires=vpn-egress-killswitch.service
+
+[Service]
+Type=notify
+# openvpn signals readiness to systemd once the tunnel is up.
+ExecStart=/usr/sbin/openvpn --suppress-timestamps --nobind --config {{ vpn_egress_gateway_wrapper_conf }}
+# The up/down scripts need CAP_NET_ADMIN to manage tun0, ip rules/routes and the nft set.
+CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_SETPCAP CAP_DAC_OVERRIDE
+AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW
+Restart=on-failure
+RestartSec=5
+# Not the node default route, so a tunnel crash must NOT cascade to the node — just retry.
+KillMode=process
+
+[Install]
+WantedBy=multi-user.target

infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# {{ ansible_managed }}
+# OpenVPN --up hook. Runs as root AFTER tun0 is created, with the tunnel's dynamic
+# address available in $ifconfig_local (and $dev = the interface, forced to tun0).
+#
+# Installs the SPLIT-TUNNEL policy routing (ADR-0015 invariant 2): only traffic whose
+# source is the tunnel address (i.e. the vpn-class pods Cilium Egress Gateway SNATs to
+# tun0) is routed via the dedicated table out tun0. The node's own default route in the
+# MAIN table is never touched.
+set -euo pipefail
+
+TABLE_ID="{{ vpn_egress_gateway_route_table_id }}"
+RULE_PRIO="{{ vpn_egress_gateway_rule_priority }}"
+TUN_METRIC="{{ vpn_egress_gateway_tun_route_metric }}"
+BH_METRIC="{{ vpn_egress_gateway_blackhole_metric }}"
+NFT_TABLE="{{ vpn_egress_gateway_nft_table }}"
+NFT_SET="{{ vpn_egress_gateway_nft_src_set }}"
+IFACE="${dev:-{{ vpn_egress_gateway_tun_iface }}}"
+SRC="${ifconfig_local:-}"
+
+if [ -z "${SRC}" ]; then
+  echo "vpn-egress-up: ifconfig_local not set; refusing to install partial routing" >&2
+  exit 1
+fi
+
+# Preferred default via the tunnel inside the DEDICATED table (not the main table).
+ip route replace default dev "${IFACE}" table "${TABLE_ID}" metric "${TUN_METRIC}"
+# Permanent fail-closed fallback: if the tunnel default is ever withdrawn while the
+# ip-rule still steers traffic here (flap / partial teardown), packets blackhole
+# instead of falling through to the main table's WAN default (invariant 4: fail closed).
+ip route replace blackhole default table "${TABLE_ID}" metric "${BH_METRIC}"
+
+# Steer ONLY tunnel-sourced (Cilium-SNAT'd vpn-class) packets into the dedicated table.
+# Idempotent: delete any stale copy of this exact rule first.
+while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done
+ip rule add from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}"
+
+# Register the live tunnel source with the kill-switch set so the nft backstop can scope
+# to exactly this traffic. Flush any stale members first (address changes on reconnect).
+nft flush set inet "${NFT_TABLE}" "${NFT_SET}" 2>/dev/null || true
+nft add element inet "${NFT_TABLE}" "${NFT_SET}" "{ ${SRC} }"
+
+echo "vpn-egress-up: tun=${IFACE} src=${SRC} table=${TABLE_ID} — split-tunnel routing active"

infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress.conf.j2

@@ -0,0 +1,45 @@
+# {{ ansible_managed }}
+# Split-tunnel wrapper for the node-level PIA egress tunnel (ADR-0015, invariants 1 & 2).
+# It INCLUDES the operator's proven PIA .ovpn and layers the split-tunnel directives on top,
+# so the same file that drove the working gluetun pod is reused verbatim (only the packaging
+# changes: pod -> host systemd service).
+
+# Pull in the operator's PIA .ovpn (inline <ca>/<crl-verify>, cipher aes-128-cbc, auth sha1,
+# remote 158.173.241.x 1198 udp, auth-user-pass). Directives AFTER this include augment/override it.
+config {{ vpn_egress_gateway_ovpn_path }}
+
+# Force a deterministic interface name so the Cilium Egress Gateway policy + kill-switch can
+# reference it by name. (The .ovpn's own `dev tun` is overridden here.)
+dev {{ vpn_egress_gateway_tun_iface }}
+dev-type tun
+
+# --- SPLIT-TUNNEL: create tun0 but NEVER become the node default route (invariant 2) ---------
+# route-noexec: OpenVPN installs NO routes itself (no `redirect-gateway`, no pushed routes).
+route-noexec
+# Belt-and-suspenders: even if the server pushes redirect-gateway, drop it before it is applied.
+pull-filter ignore "redirect-gateway"
+# Do not let the VPN rewrite the NODE's DNS — the node/kubelet keep the local resolver.
+pull-filter ignore "dhcp-option DNS"
+pull-filter ignore "dhcp-option DOMAIN"
+
+# Credentials for the .ovpn's `auth-user-pass` (rendered from Ansible vault, mode 0600).
+auth-user-pass {{ vpn_egress_gateway_credentials_path }}
+auth-nocache
+
+# The dedicated policy-routing table + ip-rule + nft set membership are installed by the
+# up/down scripts (they receive tun0's dynamic address + gateway from OpenVPN's env).
+script-security 2
+up {{ vpn_egress_gateway_scripts_dir }}/vpn-egress-up.sh
+down {{ vpn_egress_gateway_scripts_dir }}/vpn-egress-down.sh
+# Run the down script BEFORE tearing the tunnel down so it can clean up while tun0 still exists.
+down-pre
+persist-tun
+{% if vpn_egress_gateway_tun_mtu | length > 0 %}
+tun-mtu {{ vpn_egress_gateway_tun_mtu }}
+{% endif %}
+
+# Keep the tunnel resilient; systemd also restarts the unit on hard failure.
+keepalive 10 60
+resolv-retry infinite
+nobind
+verb 3

platform/vpn-egress/egress-steering.yaml

@@ -1,88 +0,0 @@
----
-# ============================================================================
-# Cilium Egress Gateway steering for `egress-class=vpn` pods — ADR-0015.
-# Serves: R-NET-05, R-NET-06, R-NET-07.
-#
-# ##########################################################################
-# ##  NOT WIRED INTO THE RUNTIME YET — DO NOT ADD TO kustomization.yaml   ##
-# ##  UNTIL THE HOST tun0 TUNNEL IS LIVE AND THIS DATAPATH IS VALIDATED.  ##
-# ##########################################################################
-#
-# This policy steers vpn-class pods out a gateway node's host `tun0` interface. It is INERT and
-# potentially disruptive until the Ansible-managed host OpenVPN split-tunnel service actually exists
-# on a node labeled `homelab.keithjackson.dev/egress-gateway=true`. If Flux shipped this before the
-# tunnel is up, vpn-class pods would be SNAT'd to a non-existent interface — an unvalidated datapath.
-# Per platform/vpn-egress/README.md: "the steering datapath is intentionally NOT committed [to the
-# runtime kustomization] until the mechanism is chosen + validated." ADR-0015 chose the mechanism
-# (node tunnel + Cilium Egress Gateway); this file IS that datapath, authored but held out of the
-# resource list for the PM to enable after a live-tunnel validation (see TODO at the bottom).
-#
-# ---------------------------------------------------------------------------
-# PREREQUISITES CONFIRMED (2026-08-24):
-#   * Cilium Egress Gateway is ENABLED per ADR-0002 — platform/cilium/helmrelease.yaml and
-#     platform/cilium/values.yaml both set (all [parity] with the Ansible bootstrap render):
-#         egressGateway.enabled: true
-#         bpf.masquerade:        true   (Egress Gateway REQUIRES eBPF masquerade)
-#         kubeProxyReplacement:  true
-#     No Cilium HelmRelease change is needed for this policy. (If any of the three were false this
-#     would be an ESCALATION, not a silent HelmRelease edit — they are all true, so we proceed.)
-#
-# INTERFACE CONTRACT (shared verbatim with the parallel Ansible host-tunnel role — ADR-0015):
-#   * Gateway node label ...... homelab.keithjackson.dev/egress-gateway=true   (applied by Ansible)
-#   * Host tunnel interface .... tun0                                          (split-tunnel OpenVPN)
-#   * Client selection ......... homelab.keithjackson.dev/egress-class=vpn     (Kyverno-enforced)
-# ---------------------------------------------------------------------------
-apiVersion: cilium.io/v2
-kind: CiliumEgressGatewayPolicy
-metadata:
-  name: vpn-egress-steering
-  annotations:
-    homelab.keithjackson.dev/requirements: R-NET-05,R-NET-06,R-NET-07
-    homelab.keithjackson.dev/adr: ADR-0015
-spec:
-  # --- Client selection (INVARIANTS 1 & 2) --------------------------------
-  # ONLY pods explicitly opted into the vpn class are matched. Default/internet-class pods and node
-  # traffic are never selected, so they stay on the local/private-WAN path (invariant 1: default is
-  # the local path). Because ONLY these flows are steered, tun0 never becomes a default route for the
-  # node or for any other pod (invariant 2). The selector is intentionally cluster-wide (no namespace
-  # scope) so the class label alone is the contract — every namespace's vpn-class pods route here.
-  selectors:
-    - podSelector:
-        matchLabels:
-          homelab.keithjackson.dev/egress-class: vpn
-  # --- What gets tunneled (INVARIANT 1: only PUBLIC internet leaves via the VPN) ------------------
-  # destinationCIDRs is the full internet; excludedCIDRs carves out everything that must stay LOCAL:
-  # all RFC1918 private ranges (which also cover the Cilium cluster-pool pod CIDR 10.0.0.0/8, the k3s
-  # service/pod ranges, and the LAN 192.168.x/10.73.x node network). So intra-cluster east-west,
-  # pod->service, and LAN traffic are NOT SNAT'd out tun0 — only genuinely public destinations are.
-  # Excluded flows fall through to Cilium's normal local datapath (invariant 1).
-  destinationCIDRs:
-    - "0.0.0.0/0"
-  excludedCIDRs:
-    - "10.0.0.0/8"       # RFC1918 + Cilium cluster-pool pod CIDR (CLUSTER_POD_CIDR) + k3s service net
-    - "172.16.0.0/12"    # RFC1918
-    - "192.168.0.0/16"   # RFC1918 (home LAN / node network family)
-  # --- Where it egresses --------------------------------------------------
-  # The gateway node (Ansible-labeled) SNATs matched traffic out its host split-tunnel `tun0`. When
-  # tun0 is absent/down, Cilium has no egress interface here; the host nftables kill-switch (Ansible,
-  # ADR-0015) is the authoritative fail-closed backstop that DROPS this vpn-class traffic — scoped to
-  # vpn-class only, so default/internet-class is unaffected (invariants 3 & 4: vpn fails CLOSED,
-  # internet fails OPEN).
-  egressGateway:
-    nodeSelector:
-      matchLabels:
-        homelab.keithjackson.dev/egress-gateway: "true"
-    interface: tun0
-# ---------------------------------------------------------------------------
-# TODO(PM — enable after live validation, then retire Gluetun):
-#   1. Confirm the Ansible host-tunnel role has brought `tun0` up on a node labeled
-#      homelab.keithjackson.dev/egress-gateway=true, and the host nftables kill-switch is in place.
-#   2. Add `- egress-steering.yaml` to platform/vpn-egress/kustomization.yaml so Flux ships it.
-#   3. Run the `egress-class` conformance test: vpn-class pod shows the PIA IP, internet-class shows
-#      the WAN IP, and a git label flip re-routes (R-NET-05/07).
-#   4. Run the `vpn-killswitch` conformance test: force tun0 down => vpn-class blackholed (no leak),
-#      internet-class unaffected (R-NET-06).
-#   5. Only AFTER both pass green: retire the Gluetun Deployment (remove gluetun-gateway.yaml from
-#      kustomization.yaml) per ADR-0015 ("Gluetun is dropped as the gateway"). Do NOT retire it in
-#      the same change that enables this — keep the validated fallback until steering is proven.
-# ---------------------------------------------------------------------------