jakman85andClaude Opus 5
04606d48b2
Four places in the tree told the reader that the host nftables table is the authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress README, egress-steering.yaml, the role README, and the vpn-killswitch test's own GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway datapath bpf_redirect()s those skbs to the egress device without invoking netfilter POSTROUTING. Corrected with the measurement in each place rather than softened, because a guard the repo trusts and that does not fire is worse than no guard. What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops the traffic when it cannot use its cached egress interface, verified by tcpdump on both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out either). The nft table keeps a narrower, now-proven job: node-originated traffic sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on k3s-prod-3 when such a packet was routed at the WAN). Also recorded: * egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the gateway node is what tells Cilium it changed (the egress-uplink node label). A future reader's instinct will be to have the node write to that object; the comment says not to, because it is Flux-managed, and points at the label and `cilium-dbg bpf egress list` as the debugging entry points instead. * egress-steering.yaml — why the new NFS media mounts are not affected by pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns, so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server too, but that is the second reason, not the first. * vpn-killswitch test — its recovery step depends on the re-steer path, and could not have passed before it existed. Its no-leak assertion was always right; only the explanation was wrong. Also states the prerequisite plainly: the test cannot run at all without a labelled gateway node. * platform README — prod is live on k3s-prod-3 with the before/after egress IPs, and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway restored egress rather than moving it off the WAN. * role README — the standing "add the shared fwmark to the interface contract" escalation is closed as not implementable on Cilium OSS, rather than left open for someone to attempt. DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 / docs-03 amendment for the architect (specs are immutable to agents) and the five things left for the operator — chief among them that a silent gateway outage still has no alert, which is what let the dev gateway sit dead for 80 minutes. Co-Authored-By: Claude Opus 5 <[email protected]>
191 lines
12 KiB
Markdown
191 lines
12 KiB
Markdown
# platform/vpn-egress — VPN egress steering (Phase 3, ADR-0015)
|
|
|
|
`vpn`-class services (sonarr, radarr, jackett, qbittorrent) egress **only** through a PIA VPN
|
|
tunnel; `internet`-class services go out the normal WAN. Serves **R-NET-05/06/07**.
|
|
|
|
Egress is realized per **[ADR-0015](../../../home-network-k3s/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md)**
|
|
(spec `docs/03`): a **node-level host OpenVPN split-tunnel** on a labeled egress-gateway node, steered
|
|
by a **Cilium Egress Gateway** policy, with a **host nftables kill-switch**. There is **no in-cluster
|
|
gateway pod** — the tunnel is host infrastructure, not a Flux workload.
|
|
|
|
> **STATUS: live on BOTH clusters** — `k3s-dev-3` and, since 2026-09-12, `k3s-prod-3`.
|
|
> The `egress-class` (R-NET-05/07) and `vpn-killswitch` (R-NET-06) tests pass on dev.
|
|
> **Caveat on that "conformance-green":** the `vpn-killswitch` test stops and then restarts the
|
|
> gateway tunnel, and until 2026-09-12 a restarted tunnel never recovered (Cilium kept a stale
|
|
> egress ifindex — see below), so the test's recovery step could not have passed on the
|
|
> implementation as it stood. Whatever produced the earlier green result, re-run it.
|
|
|
|
## How it works (ADR-0015)
|
|
Three layers, two owners:
|
|
|
|
- **Tunnel — host OpenVPN (Ansible).** A labeled egress-gateway node runs a **host OpenVPN systemd
|
|
service** (PIA over OpenVPN-UDP — provider/protocol still per ADR-0013). It is **split-tunnel**
|
|
(`route-noexec` + policy routing): it creates `tun0` but never becomes the node default route, so the
|
|
node's own traffic (control plane, image pulls, other pods) stays on the WAN. Managed by the Ansible
|
|
role **`infra/ansible/roles/vpn_egress_gateway/`**, like k3s itself — not a cluster workload.
|
|
- **Steering — Cilium Egress Gateway (GitOps).** `egress-steering.yaml` holds a
|
|
`CiliumEgressGatewayPolicy` that selects pods labeled `homelab.keithjackson.dev/egress-class=vpn`
|
|
(Kyverno-enforced) and SNATs them out the gateway node's `tun0`. Native Cilium eBPF — no client-pod
|
|
changes, no sidecar, no overlay. `excludedCIDRs` keeps RFC1918 / cluster / LAN traffic on the local
|
|
datapath; only public destinations leave via the tunnel.
|
|
- **Kill-switch — Cilium's own drop, NOT host nftables (corrected 2026-09-12).** vpn-class fails
|
|
closed because Cilium drops the traffic when it cannot use its cached egress interface. The host
|
|
nftables table is retained for a narrower job. See "The kill-switch does not do what this file
|
|
used to claim" below — that section is the measurement, not an opinion.
|
|
|
|
## LIVE ON PROD since 2026-09-12 — `k3s-prod-3`
|
|
|
|
| | dev | prod |
|
|
|---|---|---|
|
|
| gateway node | `k3s-dev-3` | `k3s-prod-3` |
|
|
| inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) |
|
|
| vpn-class egress IP | `158.173.241.36` (PIA Stockholm) | `158.173.241.36` (PIA Stockholm) |
|
|
| internet-class egress IP | home WAN | `136.51.115.252` (home WAN) |
|
|
|
|
Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three
|
|
nodes — all egress the PIA address; plex and nextcloud (internet-class) still egress the home
|
|
WAN. **Before the gateway existed, prod's vpn-class pods had no internet at all**: the CEGP was
|
|
reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress list` showed
|
|
`Gateway IP: Not Found` for their `0.0.0.0/0` entry and the traffic was dropped. Arming the
|
|
gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also
|
|
means nothing had been leaking in the meantime.
|
|
|
|
Both environments currently share one PIA account and, as it happens, one public egress IP.
|
|
That is inside the account's connection limit, but it means the two environments are not
|
|
distinguishable from the outside; a separate prod PIA account is the clean fix if that ever
|
|
matters (see `infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`).
|
|
|
|
## The kill-switch does not do what this file used to claim (measured 2026-09-12)
|
|
|
|
This README, `egress-steering.yaml`, and the role all used to say the **host nftables
|
|
kill-switch** is the authoritative fail-closed backstop for `vpn`-class traffic. **It cannot be.**
|
|
With `bpf.masquerade: true` — which Egress Gateway requires — the SNAT and the egress happen in
|
|
eBPF, which `bpf_redirect()`s the skb straight to the egress device. netfilter POSTROUTING is
|
|
never invoked for those packets.
|
|
|
|
A counters-only nft table installed alongside the real one on the gateway node, while a vpn-class
|
|
pod egressed:
|
|
|
|
| counter | packets |
|
|
|---|---|
|
|
| `forward`, saddr = pod CIDR | **0** |
|
|
| `forward`, saddr = PIA tunnel range | **0** |
|
|
| `postrouting`, oif tun0, saddr = tunnel range | **0** |
|
|
| `postrouting`, oif WAN, saddr = pod CIDR | **0** |
|
|
| `postrouting`, oif WAN, any (control: the hook IS live) | 45166 |
|
|
| `postrouting`, oif tun0, saddr = tunnel (control: a node-originated curl) | 16 |
|
|
|
|
**What actually fails closed is Cilium itself.** When it cannot use its cached egress
|
|
interface/IP it drops the packet. Verified with tcpdump on *both* the WAN and tun0, during a
|
|
tunnel stop and during a tunnel restart: **0 packets left either device.** There is no leak — the
|
|
guarantee is real — but the mechanism is Cilium's, and the existing `vpn-killswitch` conformance
|
|
test passes because of it, not because of the nft rules.
|
|
|
|
The nft table is **kept**, with a `counter` on every rule, for the narrower thing it genuinely
|
|
does cover: **node-originated** traffic sourced from a tunnel address. That was proven on
|
|
`k3s-prod-3` in both directions — a source-bound ping with the tunnel ip-rule present got 3/3
|
|
replies and moved `killswitch_accept`; with the ip-rule removed (so the routing table sent it out
|
|
`eno1`) `killswitch_drop` moved **0 -> 3 packets / 252 bytes** and the ping got nothing back.
|
|
If `killswitch_drop` ever starts counting in normal operation, the datapath has changed — read
|
|
that as news, not reassurance.
|
|
|
|
Also removed: the `meta mark 0x1c8 oifname != tun0 drop` "contract hook". Nothing stamps that
|
|
mark (the only Cilium mark on the gateway node is `0x200/0xf00`, the L7-proxy rule to table
|
|
2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is
|
|
a hazard to whatever component picks that value next. The role's `vpn_egress_gateway_fwmark`
|
|
variable is gone with it, and the "add the fwmark to the interface contract" escalation in the
|
|
role README is **closed as not-implementable** rather than left open.
|
|
|
|
## Cilium never re-resolves `tun0` — the gateway dies on every reconnect without help
|
|
|
|
Cilium resolves `interface: tun0` once into a cached (egress IP, egress ifindex) and does not
|
|
revisit it. Every tunnel restart re-creates tun0 with a **new ifindex**, so the cached one goes
|
|
stale and vpn-class egress is silently blackholed. Found the hard way: the dev gateway had been
|
|
dead for ~80 minutes when this work started, with no alert and nothing in the ledger.
|
|
|
|
`egress-gateway-reconciliation-trigger-interval: 1s` does **not** help; both clusters carry that
|
|
setting and dev still did not self-heal 20s+ after the tunnel was back. It coalesces
|
|
reconciliations that k8s object events already triggered — netlink link events are not among them.
|
|
|
|
The Ansible role now publishes `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on
|
|
the gateway node on every connect, plus a 2-minute convergence timer. That node update makes
|
|
Cilium re-process the policy and re-resolve the interface (verified: cached ifindex 1986 -> 1987
|
|
within 10s, egress recovered immediately). A node label rather than an annotation on this
|
|
Flux-managed policy, so GitOps state is never written to from a node.
|
|
|
|
**Debugging a dead gateway starts here:**
|
|
|
|
```bash
|
|
kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink
|
|
kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
|
|
cilium-dbg bpf egress list # Egress Ifindex must match the node's live `ip link show tun0`
|
|
```
|
|
|
|
If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
|
|
gateway node is the next stop.
|
|
|
|
## Resilience — there is NO gateway failover (spec correction, escalated)
|
|
|
|
**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)
|
|
provide the active/standby the earlier ADRs asked for", and spec `docs/03` repeats it. That feature
|
|
does not exist in Cilium OSS.** The supported topology is **exactly one gateway node**.
|
|
|
|
Verified against the CRD actually deployed on this cluster
|
|
(`kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`, apiVersion `cilium.io/v2`,
|
|
Cilium **v1.20.1**) — the field's own description:
|
|
|
|
> **`egressGateway`** — "EgressGateway is the gateway node responsible for SNATing traffic.
|
|
> **In case multiple nodes are a match for the given set of labels, the first node in lexical
|
|
> ordering based on their name will be selected.**"
|
|
|
|
Consequences, stated plainly:
|
|
|
|
- **One gateway node is the supported topology.** `infra/ansible/inventory/hosts.ini`'s
|
|
`[vpn_egress_gateway]` group is singleton by design.
|
|
- **A second labelled node does NOT provide failover.** It is not a standby; it carries no traffic
|
|
and takes over nothing when the first node dies. Gateway-node loss = `vpn`-class egress down
|
|
(fail-closed, per R-NET-06 — no leak, but no service).
|
|
- **Lexical ordering decides who wins if two are ever labelled.** Labelling `k3s-prod-1` while
|
|
`k3s-prod-3` is the live gateway would **silently** move the role to `k3s-prod-1` — because `1`
|
|
sorts before `3` — taking the tunnel and the public egress IP with it, with no event and no alert.
|
|
- **The plural `egressGateways` field (Cilium 1.20) is not a workaround.** It *shards* endpoints
|
|
across entries by CiliumEndpoint UID rather than running active/standby: losing one entry's node
|
|
blackholes that entry's share and re-shards the remainder (breaking live connections), and each
|
|
shard leaves from a *different* public IP. Upstream confirmation:
|
|
[cilium/cilium#30157](https://github.com/cilium/cilium/issues/30157) ("egress gateway doesn't
|
|
support failover between nodes") and
|
|
[cilium/cilium#18230](https://github.com/cilium/cilium/issues/18230) ("egress gateway does not
|
|
tolerate single node failure").
|
|
|
|
**Governance:** ADR-0015 and `docs/03` are specifications and are immutable to agents
|
|
(spec `AGENTS.md` §11, R-TEST-06, enforced by `ci/spec-immutability.sh`) — they are **not** edited
|
|
here. The correction is raised as an **escalation with a drafted ADR-0016 supersede** in
|
|
`DECISIONS-LOG.md` (entry of 2026-09-06) for the operator/architect to accept. Until then this file
|
|
and `egress-steering.yaml` are the accurate statement of what the implementation can actually do.
|
|
|
|
## Secrets — host-side now
|
|
PIA credentials + the `.ovpn` live **on the gateway node via Ansible vault** (see spec `docs/07`), **not**
|
|
a cluster SOPS Secret. SOPS still governs in-cluster secrets; VPN egress simply no longer needs one.
|
|
|
|
The `pia-credentials.sops.yaml` and `pia-custom-ovpn.sops.yaml` secrets in this directory are **now
|
|
UNUSED** — leftovers from the retired Gluetun pod, **retained pending a separate cleanup change**.
|
|
|
|
## Pieces in this directory
|
|
- `namespace.yaml` — the `vpn-egress` namespace (now holds only the Cilium policy + the retained,
|
|
unused secrets; no privileged gateway pod under ADR-0015).
|
|
- `egress-steering.yaml` — the `CiliumEgressGatewayPolicy` (the steering datapath).
|
|
- `pia-credentials.sops.yaml`, `pia-custom-ovpn.sops.yaml` — **unused**, retained pending cleanup.
|
|
|
|
## History (brief)
|
|
ADR-0013 originally placed the VPN in a single shared **Gluetun pod** and deferred the steering
|
|
mechanism. Implementation proved that a Gluetun-in-a-pod tunnel **cannot be a Cilium Egress Gateway
|
|
target**: the policy SNATs out a *node* interface, but the pod's `tun0` lives in the pod netns, and
|
|
running Gluetun `hostNetwork` to expose `tun0` on the node hijacks the node's default route (breaking
|
|
k3s/etcd/kubelet). ADR-0015 resolved this by moving the tunnel to the **node** (host OpenVPN
|
|
split-tunnel) and steering with Cilium's native egress gateway. See ADR-0015 for the full rationale.
|
|
|
|
## Port forwarding (qBittorrent) — follow-up
|
|
PIA assigns a forwarded port that can change on reconnect. Under ADR-0015 this is re-solved **host-side**
|
|
(a small pf/natpmp helper on the gateway node that publishes the assigned port to the app) — Phase 6,
|
|
alongside the qBittorrent app.
|