From 2c109e2a316c3a812ec42000b062a452cbc74cbc Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 24 Aug 2026 01:09:19 -0500 Subject: [PATCH] feat(vpn): node-level split-tunnel egress gateway Ansible role (ADR-0015) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Author the vpn_egress_gateway role + playbook that stands up the NODE-level PIA OpenVPN tunnel on egress-gateway node(s) — the host half of ADR-0015. The CiliumEgressGatewayPolicy steering vpn-class pods is separate GitOps against the same contract (group vpn_egress_gateway, node label homelab.keithjackson.dev/egress-gateway=true, host iface tun0). Honors all four ADR-0015 invariants: - Split-tunnel (inv 1&2): OpenVPN route-noexec + pull-filter ignore redirect-gateway => installs no routes; policy routing in a dedicated table (51820) with `ip rule from ` steers ONLY Cilium-SNAT'd vpn-class traffic. Node default route stays on the WAN, tunnel up or down. - Scoped nftables kill-switch (inv 3&4): own inet table, policy accept, drops only tunnel-sourced (@vpn_egress_gateway_src set) packets not leaving tun0 => vpn-class fails closed (blackhole), node/internet-class fail open (WAN), never matched. PIA creds + the proven sweden-aes-128-cbc-udp-ip .ovpn come from Ansible vault (host-side per ADR-0015/docs-07); vault keys documented, no secret values committed. ansible-lint (production) + yamllint clean. Not deployed: green conformance is gated on the live tunnel (operator host creds) — see role README. Co-Authored-By: Claude Opus 4.8 --- .../group_vars/vpn_egress_gateway/README.md | 52 +++++++ infra/ansible/inventory/hosts.ini | 7 + .../ansible/playbooks/vpn.egress.gateway.yml | 18 +++ .../roles/vpn_egress_gateway/README.md | 101 +++++++++++++ .../vpn_egress_gateway/defaults/main.yml | 66 +++++++++ .../vpn_egress_gateway/handlers/main.yml | 14 ++ .../roles/vpn_egress_gateway/tasks/main.yml | 139 ++++++++++++++++++ .../templates/pia-credentials.txt.j2 | 2 + .../templates/vpn-egress-down.sh.j2 | 29 ++++ .../templates/vpn-egress-killswitch.nft.j2 | 40 +++++ .../vpn-egress-killswitch.service.j2 | 25 ++++ .../templates/vpn-egress-tunnel.service.j2 | 25 ++++ .../templates/vpn-egress-up.sh.j2 | 43 ++++++ .../templates/vpn-egress.conf.j2 | 45 ++++++ 14 files changed, 606 insertions(+) create mode 100644 infra/ansible/inventory/group_vars/vpn_egress_gateway/README.md create mode 100644 infra/ansible/playbooks/vpn.egress.gateway.yml create mode 100644 infra/ansible/roles/vpn_egress_gateway/README.md create mode 100644 infra/ansible/roles/vpn_egress_gateway/defaults/main.yml create mode 100644 infra/ansible/roles/vpn_egress_gateway/handlers/main.yml create mode 100644 infra/ansible/roles/vpn_egress_gateway/tasks/main.yml create mode 100644 infra/ansible/roles/vpn_egress_gateway/templates/pia-credentials.txt.j2 create mode 100644 infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 create mode 100644 infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 create mode 100644 infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.service.j2 create mode 100644 infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-tunnel.service.j2 create mode 100644 infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 create mode 100644 infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress.conf.j2 diff --git a/infra/ansible/inventory/group_vars/vpn_egress_gateway/README.md b/infra/ansible/inventory/group_vars/vpn_egress_gateway/README.md new file mode 100644 index 0000000..5307074 --- /dev/null +++ b/infra/ansible/inventory/group_vars/vpn_egress_gateway/README.md @@ -0,0 +1,52 @@ +# group_vars for the `vpn_egress_gateway` node(s) — ADR-0015 host tunnel secrets + +The node-level VPN tunnel is **host infra**, so (per ADR-0015 / docs/07) its PIA credentials +live **host-side in Ansible vault** here — NOT in a cluster SOPS Secret. Create an encrypted +`vault.yml` in this directory (committed, encrypted; the passphrase file stays out of the repo, +see `ansible.cfg` `vault_password_file`). + +## Required vault keys (operator provides the VALUES — none are committed in plaintext) + +```bash +cd infra/ansible +ansible-vault create inventory/group_vars/vpn_egress_gateway/vault.yml +``` + +`vault.yml` must define exactly these three keys (the role maps them to its +`vpn_egress_gateway_*` vars in `roles/vpn_egress_gateway/defaults/main.yml`): + +| Vault key | What it is | +|---|---| +| `vault_vpn_egress_gateway_pia_username` | PIA account username (for the `.ovpn`'s `auth-user-pass`) | +| `vault_vpn_egress_gateway_pia_password` | PIA account password | +| `vault_vpn_egress_gateway_pia_ovpn` | The **full text** of the proven PIA `.ovpn` — the same `sweden-aes-128-cbc-udp-ip.ovpn` (PIA over OpenVPN-UDP, AES-128-CBC/SHA1/RSA-2048, IP-pinned Stockholm remote `158.173.241.x:1198/udp`) currently held in the k8s `pia-custom-ovpn` Secret. Use a YAML block scalar so the inline ``/`` PEM survives verbatim. | + +Example skeleton (values are placeholders — DO NOT commit unencrypted): + +```yaml +vault_vpn_egress_gateway_pia_username: "pXXXXXXX" +vault_vpn_egress_gateway_pia_password: "REDACTED" +vault_vpn_egress_gateway_pia_ovpn: | + client + dev tun + proto udp + remote 158.173.241.36 1198 + cipher aes-128-cbc + auth sha1 + auth-user-pass + ... + + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- + +``` + +> To lift the exact `.ovpn` out of the existing cluster Secret: +> `sops -d platform/vpn-egress/pia-custom-ovpn.sops.yaml` → the `sweden-aes-128-cbc-udp-ip.ovpn` +> value is the text to paste under `vault_vpn_egress_gateway_pia_ovpn`. + +## Notes +- `vault.yml` is skipped by the yamllint gate (`ci/yamllint.yaml` ignores `**/vault.yml`). +- This is the Ansible-bootstrap-layer secret (host credential), distinct from the platform's + in-cluster SOPS + age secrets (ADR-0007) — different layer, different tool, by design (ADR-0015). diff --git a/infra/ansible/inventory/hosts.ini b/infra/ansible/inventory/hosts.ini index 915c37d..7503991 100644 --- a/infra/ansible/inventory/hosts.ini +++ b/infra/ansible/inventory/hosts.ini @@ -20,3 +20,10 @@ k3s_secondary # (sudo) password comes from the vaulted group_vars/hypervisor/vault.yml. [hypervisor] localhost ansible_connection=local + +# VPN egress-gateway node(s) — run the node-level split-tunnel OpenVPN + kill-switch +# (ADR-0015, playbooks/vpn.egress.gateway.yml). Initially just k3s-dev-3 (single gateway; +# add a second host here for Cilium Egress Gateway HA). Members are already defined above +# with their ansible_host under k3s_secondary; this group only tags the egress role target. +[vpn_egress_gateway] +k3s-dev-3 diff --git a/infra/ansible/playbooks/vpn.egress.gateway.yml b/infra/ansible/playbooks/vpn.egress.gateway.yml new file mode 100644 index 0000000..92d7a37 --- /dev/null +++ b/infra/ansible/playbooks/vpn.egress.gateway.yml @@ -0,0 +1,18 @@ +--- +# Stand up the node-level VPN egress tunnel on the designated egress-gateway node(s) +# (ADR-0015). Host infra — the split-tunnel OpenVPN service + policy routing + scoped +# nftables kill-switch. The CiliumEgressGatewayPolicy that steers vpn-class pods to this +# node's tun0 is authored separately as GitOps (platform/), against the same contract: +# * inventory group `vpn_egress_gateway` (this play's target) +# * k8s node label homelab.keithjackson.dev/egress-gateway=true (set by the role) +# * host tunnel iface tun0 (split-tunnel; never the node default route) +# +# Secrets come from Ansible vault (inventory/group_vars/vpn_egress_gateway/vault.yml): +# vault_vpn_egress_gateway_pia_username, vault_vpn_egress_gateway_pia_password, vault_vpn_egress_gateway_pia_ovpn +# Run: ansible-playbook playbooks/vpn.egress.gateway.yml +- name: Configure the node-level VPN egress gateway + hosts: vpn_egress_gateway + become: true + gather_facts: true # package module + service facts need os facts + roles: + - role: vpn_egress_gateway diff --git a/infra/ansible/roles/vpn_egress_gateway/README.md b/infra/ansible/roles/vpn_egress_gateway/README.md new file mode 100644 index 0000000..2d3aece --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/README.md @@ -0,0 +1,101 @@ +# role: vpn_egress_gateway (ADR-0015) + +Stands up the **node-level VPN tunnel** on the egress-gateway node(s): a host **OpenVPN +systemd service** (PIA over OpenVPN-UDP), **split-tunnel** via policy routing, and a +**scoped nftables kill-switch**. This is the host half of ADR-0015; the +`CiliumEgressGatewayPolicy` that steers `egress-class=vpn` pods to this node's `tun0` is +separate GitOps authored against the shared **interface contract** below. + +> **CONFORMANCE IS GATED ON A LIVE TUNNEL.** This role cannot be proven green without the +> operator's PIA host credentials (see "What requires a live tunnel"). Authoring + lint pass +> in CI; the `egress-class` / `vpn-killswitch` conformance tests (R-NET-05/06/07) go green only +> once the tunnel is live in dev/test. + +## Interface contract (shared with the Cilium-policy agent) +- **Inventory group** `vpn_egress_gateway` — the egress-gateway node(s) (initially `k3s-dev-3`). +- **k8s node label** `homelab.keithjackson.dev/egress-gateway=true` — set by this role; the + `CiliumEgressGatewayPolicy` selects the gateway node by it. +- **Host tunnel iface** `tun0` — the Cilium policy SNATs vpn-class pods out of it. +- **Optional fwmark** `0x1c8` — a contract *hook* (see kill-switch layering) the Cilium side + MAY stamp on vpn-class packets for a tunnel-down-independent host backstop. + +## How the four ADR-0015 invariants are met + +### 1 & 2 — split-tunnel; tun0 is never a default route +`templates/vpn-egress.conf.j2` includes the operator's proven `.ovpn` and layers on +`route-noexec` + `pull-filter ignore "redirect-gateway"`, so **OpenVPN installs no routes and +never touches the main routing table**. The node's own default route stays on the WAN whether +the tunnel is up or down — k3s/etcd/kubelet are unaffected. + +Steering is done with **policy routing** in a dedicated table (`51820`, named `vpn_egress`): +- The OpenVPN **up-script** adds `default dev tun0` to table `51820` and an ip-rule + **`from lookup 51820`** (priority 5000). Only packets whose source is the tunnel + address — i.e. the vpn-class pods Cilium has SNAT'd to `tun0` — consult that table. Nothing + else does, so nothing else is re-routed. +- The **down-script** removes the ip-rule and the tunnel default (leaving the blackhole). + +### 3 & 4 — kill-switch scoped to vpn-class, opposite fail directions +Two independent layers, both scoped so they can only ever affect vpn-class traffic: + +1. **Routing blackhole (fail-closed fallback).** Table `51820` always carries a permanent + `blackhole default` (installed by `vpn-egress-killswitch.service`, metric 4000). When the + tunnel default is withdrawn (flap / partial teardown) while the ip-rule still steers, packets + hit the blackhole instead of falling through to the WAN. +2. **nftables backstop (`templates/vpn-egress-killswitch.nft.j2`).** Own `inet + vpn_egress_gateway_killswitch` table, base chain **policy `accept`** (so it can only *drop* the + narrow matches, never break anything). In `postrouting`: + - `oifname tun0 ip saddr @vpn_egress_gateway_src accept` — tunnel up: let steered traffic out. + - `ip saddr @vpn_egress_gateway_src oifname != tun0 drop` — **fail closed:** a tunnel-sourced packet + trying to leave any other interface is dropped (no WAN leak). + - `meta mark 0x1c8 oifname != tun0 drop` — the optional fwmark backstop. + +`@vpn_egress_gateway_src` holds the **live tun0 address** (added by the up-script, flushed by the +down-script). Node traffic and `internet`-class traffic are never in the set, never carry the +mark, and are never targeted — so they always hit `policy accept` and **fail open toward the +WAN** (invariant 4), tunnel up or down. The opposite directions are structural, not incidental. + +## Why the node default route is provably unaffected (design-level verification) +- OpenVPN runs with `route-noexec` and `pull-filter ignore "redirect-gateway"` → it adds **zero** + routes and cannot install a `0.0.0.0/0` via `tun0`. (This is exactly the failure mode that made + `hostNetwork` gluetun non-viable in ADR-0015; here OpenVPN is explicitly told not to route.) +- All tunnel routing lives in **table 51820 only**; the up/down scripts touch that table and + ip-rules, never `table main` / `table default`. +- The ip-rule matches **`from `** — an address the node itself never sources traffic + from — so node-originated and internet-class packets never enter the tunnel table. +- The nft table is **policy accept** and matches only `@vpn_egress_gateway_src` / the fwmark. +- Live check (post-deploy): `ip route show table main default` still points at the WAN NIC, and + `curl ipinfo` from the node shows the WAN IP, with the tunnel both up and forced down. + +## Kill-switch layering & ESCALATION (honest limit) +- The nft `@vpn_egress_gateway_src` rule and the routing blackhole robustly cover the case where the + tunnel is up but a vpn-class packet is (mis)routed toward the WAN, and flaps. +- The **tunnel fully-down** case relies on **Cilium Egress Gateway itself failing closed** when + its egress interface `tun0` is absent (Cilium 1.16 drops egress-gateway-marked traffic with no + valid egress path rather than falling back to the node's default route) **plus** the routing + blackhole. That is the ADR's expected behavior. +- For a host backstop that is **fully independent of Cilium even in a fail-open regression** when + the tunnel is down (no tun0 address to key on), the host needs a stable marker on vpn-class + packets. The only contract-available marker is the **`0x1c8` fwmark hook** — which the + *Cilium-policy* side must stamp. **ESCALATION for the PM:** either (a) confirm the Cilium + Egress Gateway 1.16 fail-closed + routing-blackhole guarantee is accepted as sufficient for + R-NET-06, or (b) add the shared fwmark to the interface contract so the CiliumEgressGatewayPolicy + marks vpn-class traffic and this role's fwmark rule becomes the tunnel-down-independent backstop. + This is flagged rather than worked around — no spec was weakened to reach done. + +## What requires a LIVE tunnel to validate (cannot be tested without operator creds) +1. The PIA `.ovpn` + username/password actually authenticate and `tun0` comes up as a host iface. +2. `egress-class`: a vpn-class pod's public IP is the PIA Stockholm IP; internet-class shows the WAN IP. +3. `vpn-killswitch`: forcing `tun0`/the tunnel service down blackholes vpn-class egress (no leak) + while internet-class is unaffected — the R-NET-06 fault-injection gate. +4. That Cilium Egress Gateway SNAT to `tun0` traverses host `postrouting` so the nft set-scoped + rules actually see the packets (eBPF datapath interaction — see escalation). +5. MTU/PMTU across `pod → tun0 → PIA` (in-pod gluetun needed 1300 for Cilium VXLAN; on the node + the tunnel rides the physical WAN, so `vpn_egress_gateway_tun_mtu` defaults empty — verify live). +6. Port forwarding (PIA NAT-PMP) for qBittorrent — ADR-0015 Phase-6 follow-up, not in this role. + +## Files +- `defaults/main.yml` — all `vpn_egress_gateway_*` vars; the three vault-backed credential vars. +- `tasks/main.yml` — packages, rt_tables, config/scripts/units, enable+start, node label. +- `handlers/main.yml` — daemon-reload, kill-switch reload, tunnel restart. +- `templates/` — wrapper `.conf`, credentials, up/down scripts, nft ruleset, both systemd units. +- Playbook: `playbooks/vpn.egress.gateway.yml`. Vault keys: `inventory/group_vars/vpn_egress_gateway/README.md`. diff --git a/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml b/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml new file mode 100644 index 0000000..8e82ef0 --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/defaults/main.yml @@ -0,0 +1,66 @@ +--- +# vpn_egress_gateway role defaults (ADR-0015). +# Node-level split-tunnel host OpenVPN + policy routing + scoped nftables kill-switch. +# All vars are role-name prefixed. NO secret VALUES live here — the three credential +# vars below reference operator-provided Ansible-vault keys (see the role README and +# inventory/group_vars/vpn_egress_gateway/README.md). They are intentionally NOT +# defaulted so a missing vault fails loud rather than silently deploying a broken tunnel. + +# --- packages ------------------------------------------------------------------- +vpn_egress_gateway_packages: + - openvpn + - nftables + +# --- credentials / config (from Ansible vault — operator provides the values) ---- +# The PIA .ovpn PROVEN in Phase 3 (ADR-0013/0015): sweden-aes-128-cbc-udp-ip.ovpn — +# PIA over OpenVPN-UDP, AES-128-CBC / SHA1 / RSA-2048, IP-pinned PIA Stockholm remote +# (158.173.241.x:1198/udp), auth-user-pass. Same file that drove the working gluetun pod. +vpn_egress_gateway_pia_username: "{{ vault_vpn_egress_gateway_pia_username }}" +vpn_egress_gateway_pia_password: "{{ vault_vpn_egress_gateway_pia_password }}" +# The FULL text of the PIA .ovpn (inline /, cipher/auth, remote line). +vpn_egress_gateway_pia_ovpn: "{{ vault_vpn_egress_gateway_pia_ovpn }}" + +# --- on-disk layout ------------------------------------------------------------- +vpn_egress_gateway_config_dir: /etc/openvpn/client +vpn_egress_gateway_scripts_dir: /etc/openvpn/scripts +vpn_egress_gateway_ovpn_path: "{{ vpn_egress_gateway_config_dir }}/pia.ovpn" +vpn_egress_gateway_credentials_path: "{{ vpn_egress_gateway_config_dir }}/pia-credentials.txt" +vpn_egress_gateway_wrapper_conf: "{{ vpn_egress_gateway_config_dir }}/vpn-egress.conf" + +# --- tunnel interface + split-tunnel policy routing (invariants 1 & 2) ---------- +# The host tunnel interface Cilium Egress Gateway steers vpn-class pods out of. +vpn_egress_gateway_tun_iface: tun0 +# Dedicated policy-routing table for the tunnel (NEVER the main table — the node's +# own default route stays on the WAN). Numeric id + a name added to rt_tables. +vpn_egress_gateway_route_table_id: 51820 +vpn_egress_gateway_route_table_name: vpn_egress +# ip-rule priority for the `from lookup vpn_egress` rule (well below main). +vpn_egress_gateway_rule_priority: 5000 +# Route metrics inside the dedicated table: the live tunnel default is preferred; +# a permanent blackhole is the fail-closed fallback during flaps / partial teardown. +vpn_egress_gateway_tun_route_metric: 50 +vpn_egress_gateway_blackhole_metric: 4000 + +# Optional tun MTU override. Empty => honor whatever the .ovpn/PIA negotiates. On the +# NODE the tunnel rides the physical WAN (1500), so unlike the in-pod gluetun case +# (Cilium VXLAN 1450 → needed 1300) no shrink is required by default. +vpn_egress_gateway_tun_mtu: "" + +# --- nftables kill-switch (invariants 3 & 4 — scoped to vpn-class ONLY) ---------- +vpn_egress_gateway_nft_table: vpn_egress_gateway_killswitch +# Named set holding the LIVE tun0 source address (maintained by the up/down scripts). +# The kill-switch matches ONLY packets SNAT'd to this address by Cilium Egress Gateway +# — i.e. exactly the egress-class=vpn traffic. Node / internet-class traffic is never +# in this set and is never matched (fails OPEN toward the WAN). +vpn_egress_gateway_nft_src_set: vpn_egress_gateway_src +# Contract-hook fwmark: if the CiliumEgressGatewayPolicy side stamps vpn-class packets +# with this mark, the kill-switch also drops marked packets not leaving via tun0 — an +# extra, tunnel-down-independent backstop. See README "Kill-switch layering / escalation". +vpn_egress_gateway_fwmark: "0x1c8" + +# --- kubernetes node label (steering contract) ---------------------------------- +# The label the parallel CiliumEgressGatewayPolicy selects the gateway node by. +vpn_egress_gateway_node_label_key: homelab.keithjackson.dev/egress-gateway +vpn_egress_gateway_node_label_value: "true" +# k3s ships its kubeconfig here; the gateway node is a k3s server so `k3s kubectl` works. +vpn_egress_gateway_kubeconfig: /etc/rancher/k3s/k3s.yaml diff --git a/infra/ansible/roles/vpn_egress_gateway/handlers/main.yml b/infra/ansible/roles/vpn_egress_gateway/handlers/main.yml new file mode 100644 index 0000000..bad35d8 --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/handlers/main.yml @@ -0,0 +1,14 @@ +--- +- name: Reload systemd + ansible.builtin.systemd_service: + daemon_reload: true + +- name: Reload vpn-egress kill-switch + ansible.builtin.systemd_service: + name: vpn-egress-killswitch.service + state: restarted + +- name: Restart vpn-egress tunnel + ansible.builtin.systemd_service: + name: vpn-egress-tunnel.service + state: restarted diff --git a/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml b/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml new file mode 100644 index 0000000..206786d --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/tasks/main.yml @@ -0,0 +1,139 @@ +--- +# vpn_egress_gateway — node-level split-tunnel PIA OpenVPN + policy routing + scoped +# nftables kill-switch on the egress-gateway node(s) (ADR-0015). Host infra, Ansible- +# managed like k3s itself; the CiliumEgressGatewayPolicy that steers pods is separate GitOps. + +- name: Install OpenVPN and nftables + ansible.builtin.package: + name: "{{ vpn_egress_gateway_packages }}" + state: present + +- name: Register the dedicated policy-routing table name + ansible.builtin.lineinfile: + path: /etc/iproute2/rt_tables + line: "{{ vpn_egress_gateway_route_table_id }} {{ vpn_egress_gateway_route_table_name }}" + regexp: "^{{ vpn_egress_gateway_route_table_id }}\\s" + state: present + create: true + mode: "0644" + +- name: Ensure the OpenVPN config and scripts directories exist + ansible.builtin.file: + path: "{{ item }}" + state: directory + owner: root + group: root + mode: "0750" + loop: + - "{{ vpn_egress_gateway_config_dir }}" + - "{{ vpn_egress_gateway_scripts_dir }}" + +- name: Deploy the operator's proven PIA .ovpn (from Ansible vault) + ansible.builtin.copy: + content: "{{ vpn_egress_gateway_pia_ovpn }}" + dest: "{{ vpn_egress_gateway_ovpn_path }}" + owner: root + group: root + mode: "0600" + no_log: true + notify: Restart vpn-egress tunnel + +- name: Deploy the PIA auth-user-pass credentials (from Ansible vault) + ansible.builtin.template: + src: pia-credentials.txt.j2 + dest: "{{ vpn_egress_gateway_credentials_path }}" + owner: root + group: root + mode: "0600" + no_log: true + notify: Restart vpn-egress tunnel + +- name: Deploy the split-tunnel OpenVPN wrapper config + ansible.builtin.template: + src: vpn-egress.conf.j2 + dest: "{{ vpn_egress_gateway_wrapper_conf }}" + owner: root + group: root + mode: "0600" + notify: Restart vpn-egress tunnel + +- name: Deploy the split-tunnel up/down policy-routing scripts + ansible.builtin.template: + src: "{{ item }}.j2" + dest: "{{ vpn_egress_gateway_scripts_dir }}/{{ item }}" + owner: root + group: root + mode: "0750" + loop: + - vpn-egress-up.sh + - vpn-egress-down.sh + notify: Restart vpn-egress tunnel + +- name: Deploy the scoped nftables kill-switch ruleset + ansible.builtin.template: + src: vpn-egress-killswitch.nft.j2 + dest: "{{ vpn_egress_gateway_config_dir }}/vpn-egress-killswitch.nft" + owner: root + group: root + mode: "0600" + notify: Reload vpn-egress kill-switch + +- name: Deploy the kill-switch systemd unit + ansible.builtin.template: + src: vpn-egress-killswitch.service.j2 + dest: /etc/systemd/system/vpn-egress-killswitch.service + owner: root + group: root + mode: "0644" + notify: + - Reload systemd + - Reload vpn-egress kill-switch + +- name: Deploy the tunnel systemd unit + ansible.builtin.template: + src: vpn-egress-tunnel.service.j2 + dest: /etc/systemd/system/vpn-egress-tunnel.service + owner: root + group: root + mode: "0644" + notify: + - Reload systemd + - Restart vpn-egress tunnel + +- name: Flush handlers so units/routing exist before enable+start + ansible.builtin.meta: flush_handlers + +- name: Enable and start the kill-switch (must be up before the tunnel) + ansible.builtin.systemd_service: + name: vpn-egress-killswitch.service + enabled: true + state: started + +- name: Enable and start the split-tunnel VPN service + ansible.builtin.systemd_service: + name: vpn-egress-tunnel.service + enabled: true + state: started + +# --- steering contract: label the k8s node the Cilium Egress Gateway policy selects ---- +- name: Read the current egress-gateway node label + ansible.builtin.command: + cmd: >- + k3s kubectl get node {{ inventory_hostname }} + -o jsonpath={.metadata.labels.{{ vpn_egress_gateway_node_label_key | replace('.', '\\.') | replace('/', '\\/') }}} + environment: + KUBECONFIG: "{{ vpn_egress_gateway_kubeconfig }}" + register: vpn_egress_gateway_label_current + changed_when: false + failed_when: false + +- name: Label this node as the VPN egress gateway + ansible.builtin.command: + cmd: >- + k3s kubectl label node {{ inventory_hostname }} + {{ vpn_egress_gateway_node_label_key }}={{ vpn_egress_gateway_node_label_value }} --overwrite + environment: + KUBECONFIG: "{{ vpn_egress_gateway_kubeconfig }}" + register: vpn_egress_gateway_label_apply + changed_when: "'not labeled' not in vpn_egress_gateway_label_apply.stdout" + when: vpn_egress_gateway_label_current.stdout | default('') != vpn_egress_gateway_node_label_value diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/pia-credentials.txt.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/pia-credentials.txt.j2 new file mode 100644 index 0000000..7d3d736 --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/pia-credentials.txt.j2 @@ -0,0 +1,2 @@ +{{ vpn_egress_gateway_pia_username }} +{{ vpn_egress_gateway_pia_password }} diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 new file mode 100644 index 0000000..5701318 --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-down.sh.j2 @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# {{ ansible_managed }} +# OpenVPN --down hook (with `down-pre`, runs BEFORE tun0 is destroyed). Tears down the +# split-tunnel steering so vpn-class traffic FAILS CLOSED, never leaks to the WAN +# (ADR-0015 invariants 3 & 4). +# +# Order matters: (1) empty the kill-switch source set and drop the ip-rule so no packet +# can be routed toward a vanishing tun0, then (2) remove the tunnel default from the +# dedicated table. The permanent `blackhole default` in that table is LEFT in place as +# the fail-closed fallback. The node's main-table WAN default is never touched here. +set -uo pipefail + +TABLE_ID="{{ vpn_egress_gateway_route_table_id }}" +RULE_PRIO="{{ vpn_egress_gateway_rule_priority }}" +NFT_TABLE="{{ vpn_egress_gateway_nft_table }}" +NFT_SET="{{ vpn_egress_gateway_nft_src_set }}" +IFACE="${dev:-{{ vpn_egress_gateway_tun_iface }}}" +SRC="${ifconfig_local:-}" + +# 1) Stop scoping/steering to the (now dead) tunnel source. +nft flush set inet "${NFT_TABLE}" "${NFT_SET}" 2>/dev/null || true +if [ -n "${SRC}" ]; then + while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done +fi + +# 2) Remove the tunnel default from the dedicated table; keep the blackhole fallback. +ip route del default dev "${IFACE}" table "${TABLE_ID}" 2>/dev/null || true + +echo "vpn-egress-down: steering removed; vpn-class fails closed (blackhole retained)" diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 new file mode 100644 index 0000000..134c6ad --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.nft.j2 @@ -0,0 +1,40 @@ +#!/usr/sbin/nft -f +# {{ ansible_managed }} +# ADR-0015 kill-switch (invariants 3 & 4) — SCOPED TO vpn-class ONLY, fails CLOSED. +# +# Own dedicated table so re-loading it never disturbs Cilium's or the node's own rules. +# The base chains are policy ACCEPT: this table can ONLY ever drop the narrow vpn-class +# traffic matched below; node traffic and internet-class traffic are untouched and keep +# egressing the WAN (fail OPEN — invariant 4). It matches on: +# * @{{ vpn_egress_gateway_nft_src_set }} — the LIVE tun0 address Cilium Egress Gateway SNATs +# vpn-class pods to (populated by the OpenVPN up-script; emptied by the down-script). +# This IS the egress-class=vpn traffic and nothing else. +# * meta mark {{ vpn_egress_gateway_fwmark }} — optional contract-hook: a tunnel-DOWN-independent +# backstop, active only if the CiliumEgressGatewayPolicy side stamps this fwmark. + +table inet {{ vpn_egress_gateway_nft_table }} { + # Live tunnel source address(es). Empty whenever the tunnel is down => the two + # saddr rules below match nothing => vpn-class has no path at all (fail closed). + set {{ vpn_egress_gateway_nft_src_set }} { + type ipv4_addr + } + + chain postrouting { + # oifname is resolved at postrouting; SNAT/masquerade has already applied here, + # so the Cilium-SNAT'd vpn-class packet's saddr is the tunnel address. + type filter hook postrouting priority 100; policy accept; + + # Tunnel UP: let the steered vpn-class traffic out tun0. + oifname "{{ vpn_egress_gateway_tun_iface }}" ip saddr @{{ vpn_egress_gateway_nft_src_set }} accept + + # FAIL CLOSED: a vpn-class (tunnel-sourced) packet trying to leave ANY other + # interface (i.e. the WAN) is dropped — never leaks. Scoped by the set, so it + # can never match node or internet-class traffic. + ip saddr @{{ vpn_egress_gateway_nft_src_set }} oifname != "{{ vpn_egress_gateway_tun_iface }}" drop + + # Tunnel-down-independent backstop (contract hook, see role README): if vpn-class + # packets carry the agreed fwmark, drop any not leaving via tun0 even when the set + # is empty (tunnel fully down). Harmless if the Cilium side does not set the mark. + meta mark {{ vpn_egress_gateway_fwmark }} oifname != "{{ vpn_egress_gateway_tun_iface }}" drop + } +} diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.service.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.service.j2 new file mode 100644 index 0000000..25bfca5 --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-killswitch.service.j2 @@ -0,0 +1,25 @@ +# {{ ansible_managed }} +# Installs the ADR-0015 kill-switch state that must exist INDEPENDENTLY of (and BEFORE) +# the tunnel: the scoped nftables table/set and the permanent fail-closed blackhole route +# in the dedicated policy-routing table. Ordered before the tunnel so the nft set exists +# when the OpenVPN up-script registers the live tunnel address. +[Unit] +Description=VPN egress gateway kill-switch (nftables + fail-closed blackhole) — ADR-0015 +Documentation=file://{{ vpn_egress_gateway_config_dir }}/README +After=network-pre.target nftables.service +Wants=network-pre.target +Before=vpn-egress-tunnel.service + +[Service] +Type=oneshot +RemainAfterExit=yes +# Load the scoped kill-switch table (idempotent: delete-then-add). +ExecStart=/usr/sbin/nft -f {{ vpn_egress_gateway_config_dir }}/vpn-egress-killswitch.nft +# Ensure the dedicated table has a permanent blackhole default (fail closed) even before +# the tunnel first connects. The tunnel up-script adds the preferred `default dev tun0`. +ExecStart=/sbin/ip route replace blackhole default table {{ vpn_egress_gateway_route_table_id }} metric {{ vpn_egress_gateway_blackhole_metric }} +# Tear down cleanly: remove the whole scoped table (never touches other nft rules). +ExecStop=/usr/sbin/nft delete table inet {{ vpn_egress_gateway_nft_table }} + +[Install] +WantedBy=multi-user.target diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-tunnel.service.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-tunnel.service.j2 new file mode 100644 index 0000000..c58c4a2 --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-tunnel.service.j2 @@ -0,0 +1,25 @@ +# {{ ansible_managed }} +# Node-level host OpenVPN tunnel (ADR-0015): PIA over OpenVPN-UDP, split-tunnel. Creates +# {{ vpn_egress_gateway_tun_iface }} but — via the wrapper's route-noexec / pull-filter — NEVER sets +# the node default route. The node (k3s/etcd/kubelet) keeps working whether this is up or down. +[Unit] +Description=VPN egress gateway tunnel (PIA OpenVPN-UDP, split-tunnel) — ADR-0015 +Documentation=file://{{ vpn_egress_gateway_config_dir }}/README +After=network-online.target vpn-egress-killswitch.service +Wants=network-online.target +Requires=vpn-egress-killswitch.service + +[Service] +Type=notify +# openvpn signals readiness to systemd once the tunnel is up. +ExecStart=/usr/sbin/openvpn --suppress-timestamps --nobind --config {{ vpn_egress_gateway_wrapper_conf }} +# The up/down scripts need CAP_NET_ADMIN to manage tun0, ip rules/routes and the nft set. +CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW CAP_SETPCAP CAP_DAC_OVERRIDE +AmbientCapabilities=CAP_NET_ADMIN CAP_NET_RAW +Restart=on-failure +RestartSec=5 +# Not the node default route, so a tunnel crash must NOT cascade to the node — just retry. +KillMode=process + +[Install] +WantedBy=multi-user.target diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 new file mode 100644 index 0000000..d108dbc --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2 @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# {{ ansible_managed }} +# OpenVPN --up hook. Runs as root AFTER tun0 is created, with the tunnel's dynamic +# address available in $ifconfig_local (and $dev = the interface, forced to tun0). +# +# Installs the SPLIT-TUNNEL policy routing (ADR-0015 invariant 2): only traffic whose +# source is the tunnel address (i.e. the vpn-class pods Cilium Egress Gateway SNATs to +# tun0) is routed via the dedicated table out tun0. The node's own default route in the +# MAIN table is never touched. +set -euo pipefail + +TABLE_ID="{{ vpn_egress_gateway_route_table_id }}" +RULE_PRIO="{{ vpn_egress_gateway_rule_priority }}" +TUN_METRIC="{{ vpn_egress_gateway_tun_route_metric }}" +BH_METRIC="{{ vpn_egress_gateway_blackhole_metric }}" +NFT_TABLE="{{ vpn_egress_gateway_nft_table }}" +NFT_SET="{{ vpn_egress_gateway_nft_src_set }}" +IFACE="${dev:-{{ vpn_egress_gateway_tun_iface }}}" +SRC="${ifconfig_local:-}" + +if [ -z "${SRC}" ]; then + echo "vpn-egress-up: ifconfig_local not set; refusing to install partial routing" >&2 + exit 1 +fi + +# Preferred default via the tunnel inside the DEDICATED table (not the main table). +ip route replace default dev "${IFACE}" table "${TABLE_ID}" metric "${TUN_METRIC}" +# Permanent fail-closed fallback: if the tunnel default is ever withdrawn while the +# ip-rule still steers traffic here (flap / partial teardown), packets blackhole +# instead of falling through to the main table's WAN default (invariant 4: fail closed). +ip route replace blackhole default table "${TABLE_ID}" metric "${BH_METRIC}" + +# Steer ONLY tunnel-sourced (Cilium-SNAT'd vpn-class) packets into the dedicated table. +# Idempotent: delete any stale copy of this exact rule first. +while ip rule del from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" 2>/dev/null; do :; done +ip rule add from "${SRC}" lookup "${TABLE_ID}" priority "${RULE_PRIO}" + +# Register the live tunnel source with the kill-switch set so the nft backstop can scope +# to exactly this traffic. Flush any stale members first (address changes on reconnect). +nft flush set inet "${NFT_TABLE}" "${NFT_SET}" 2>/dev/null || true +nft add element inet "${NFT_TABLE}" "${NFT_SET}" "{ ${SRC} }" + +echo "vpn-egress-up: tun=${IFACE} src=${SRC} table=${TABLE_ID} — split-tunnel routing active" diff --git a/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress.conf.j2 b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress.conf.j2 new file mode 100644 index 0000000..f8bed99 --- /dev/null +++ b/infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress.conf.j2 @@ -0,0 +1,45 @@ +# {{ ansible_managed }} +# Split-tunnel wrapper for the node-level PIA egress tunnel (ADR-0015, invariants 1 & 2). +# It INCLUDES the operator's proven PIA .ovpn and layers the split-tunnel directives on top, +# so the same file that drove the working gluetun pod is reused verbatim (only the packaging +# changes: pod -> host systemd service). + +# Pull in the operator's PIA .ovpn (inline /, cipher aes-128-cbc, auth sha1, +# remote 158.173.241.x 1198 udp, auth-user-pass). Directives AFTER this include augment/override it. +config {{ vpn_egress_gateway_ovpn_path }} + +# Force a deterministic interface name so the Cilium Egress Gateway policy + kill-switch can +# reference it by name. (The .ovpn's own `dev tun` is overridden here.) +dev {{ vpn_egress_gateway_tun_iface }} +dev-type tun + +# --- SPLIT-TUNNEL: create tun0 but NEVER become the node default route (invariant 2) --------- +# route-noexec: OpenVPN installs NO routes itself (no `redirect-gateway`, no pushed routes). +route-noexec +# Belt-and-suspenders: even if the server pushes redirect-gateway, drop it before it is applied. +pull-filter ignore "redirect-gateway" +# Do not let the VPN rewrite the NODE's DNS — the node/kubelet keep the local resolver. +pull-filter ignore "dhcp-option DNS" +pull-filter ignore "dhcp-option DOMAIN" + +# Credentials for the .ovpn's `auth-user-pass` (rendered from Ansible vault, mode 0600). +auth-user-pass {{ vpn_egress_gateway_credentials_path }} +auth-nocache + +# The dedicated policy-routing table + ip-rule + nft set membership are installed by the +# up/down scripts (they receive tun0's dynamic address + gateway from OpenVPN's env). +script-security 2 +up {{ vpn_egress_gateway_scripts_dir }}/vpn-egress-up.sh +down {{ vpn_egress_gateway_scripts_dir }}/vpn-egress-down.sh +# Run the down script BEFORE tearing the tunnel down so it can clean up while tun0 still exists. +down-pre +persist-tun +{% if vpn_egress_gateway_tun_mtu | length > 0 %} +tun-mtu {{ vpn_egress_gateway_tun_mtu }} +{% endif %} + +# Keep the tunnel resilient; systemd also restarts the unit on hard failure. +keepalive 10 60 +resolv-retry infinite +nobind +verb 3