jakman85andClaude Opus 5
16c6234696
`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first resolution, so after a reconnect the BPF egress map points at a destroyed interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat in exactly that state for ~80 minutes with nothing alerting. Two rules close it: * VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was re-created more times in the last hour than the gateway node's `egress-uplink` label changed, i.e. Cilium was never told about a reconnect. * VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the watchdog: a gateway node whose uplink label never reaches kube-state-metrics is a cluster where the rule above is inert. Both gated on a gateway node existing, reusing `VPNTunnelDown`'s `kube_node_labels` guard — a cluster with no gateway cannot be made to fire them. The direct label-vs-map comparison is NOT available and nothing was invented to pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS publishes no series carrying egress-map contents at all — the map is readable only via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the live ifindex) against a number inside a label value, so the level-triggered equality test is not expressible; the same invariant is counted over a window instead. Both limits, and the textfile-collector/black-box-probe routes that would lift them, are written down in the rule comments and the DECISIONS-LOG. `egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same kustomization, because without it the first rule has nothing to compare. Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr extracted from `kubectl kustomize platform-config`: empty when healthy; empty when evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier today. Nothing on prod was disturbed to produce that. Co-Authored-By: Claude Opus 5 <[email protected]>
249 lines
16 KiB
Markdown
249 lines
16 KiB
Markdown
# platform/vpn-egress — VPN egress steering (Phase 3, ADR-0015)
|
|
|
|
`vpn`-class services (sonarr, radarr, jackett, qbittorrent) egress **only** through a PIA VPN
|
|
tunnel; `internet`-class services go out the normal WAN. Serves **R-NET-05/06/07**.
|
|
|
|
Egress is realized per **[ADR-0015](../../../home-network-k3s/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md)**
|
|
(spec `docs/03`): a **node-level host OpenVPN split-tunnel** on a labeled egress-gateway node, steered
|
|
by a **Cilium Egress Gateway** policy, with a **host nftables kill-switch**. There is **no in-cluster
|
|
gateway pod** — the tunnel is host infrastructure, not a Flux workload.
|
|
|
|
> **STATUS: live on BOTH clusters** — `k3s-dev-3` and, since 2026-09-12, `k3s-prod-3`.
|
|
> The `egress-class` (R-NET-05/07) and `vpn-killswitch` (R-NET-06) tests pass on dev.
|
|
> **Caveat on that "conformance-green":** the `vpn-killswitch` test stops and then restarts the
|
|
> gateway tunnel, and until 2026-09-12 a restarted tunnel never recovered (Cilium kept a stale
|
|
> egress ifindex — see below), so the test's recovery step could not have passed on the
|
|
> implementation as it stood. Whatever produced the earlier green result, re-run it.
|
|
|
|
## How it works (ADR-0015)
|
|
Three layers, two owners:
|
|
|
|
- **Tunnel — host OpenVPN (Ansible).** A labeled egress-gateway node runs a **host OpenVPN systemd
|
|
service** (PIA over OpenVPN-UDP — provider/protocol still per ADR-0013). It is **split-tunnel**
|
|
(`route-noexec` + policy routing): it creates `tun0` but never becomes the node default route, so the
|
|
node's own traffic (control plane, image pulls, other pods) stays on the WAN. Managed by the Ansible
|
|
role **`infra/ansible/roles/vpn_egress_gateway/`**, like k3s itself — not a cluster workload.
|
|
- **Steering — Cilium Egress Gateway (GitOps).** `egress-steering.yaml` holds a
|
|
`CiliumEgressGatewayPolicy` that selects pods labeled `homelab.keithjackson.dev/egress-class=vpn`
|
|
(Kyverno-enforced) and SNATs them out the gateway node's `tun0`. Native Cilium eBPF — no client-pod
|
|
changes, no sidecar, no overlay. `excludedCIDRs` keeps RFC1918 / cluster / LAN traffic on the local
|
|
datapath; only public destinations leave via the tunnel.
|
|
- **Kill-switch — Cilium's own drop, NOT host nftables (corrected 2026-09-12).** vpn-class fails
|
|
closed because Cilium drops the traffic when it cannot use its cached egress interface. The host
|
|
nftables table is retained for a narrower job. See "The kill-switch does not do what this file
|
|
used to claim" below — that section is the measurement, not an opinion.
|
|
|
|
## LIVE ON PROD since 2026-09-12 — `k3s-prod-3`
|
|
|
|
| | dev | prod |
|
|
|---|---|---|
|
|
| gateway node | `k3s-dev-3` | `k3s-prod-3` |
|
|
| inventory group | `vpn_egress_gateway` | `vpn_egress_gateway_prod` (separate on purpose) |
|
|
| PIA server | `158.173.241.112` (the .ovpn's pin) | `158.173.241.135` (group_vars override — see below) |
|
|
| vpn-class egress IP | `158.173.241.36` | `158.173.241.11` |
|
|
| internet-class egress IP | home WAN | `136.51.115.252` (home WAN) |
|
|
|
|
Prod's four vpn-class workloads — jackett, qbittorrent, radarr, sonarr, spread across all three
|
|
nodes — all egress the PIA address; plex and nextcloud (internet-class) still egress the home
|
|
WAN. **Before the gateway existed, prod's vpn-class pods had no internet at all**: the CEGP was
|
|
reconciled with no node matching its nodeSelector, so `cilium-dbg bpf egress list` showed
|
|
`Gateway IP: Not Found` for their `0.0.0.0/0` entry and the traffic was dropped. Arming the
|
|
gateway *restored* their egress rather than moving it from the WAN to the tunnel — which also
|
|
means nothing had been leaking in the meantime.
|
|
|
|
### One PIA account is fine. One pinned PIA *server* is not.
|
|
|
|
Standing prod up broke dev, and the reason is not the connection limit. The vaulted `.ovpn`
|
|
pins **one** server, so both gateways dialled it — and PIA handed **both sessions the same
|
|
tunnel address**. Verbatim from both journals:
|
|
|
|
```
|
|
[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198
|
|
PUSH_REPLY, ... ifconfig 10.72.192.7 255.255.192.0 ...
|
|
```
|
|
|
|
Each reconnect steals `10.72.192.7` on that server; the loser receives nothing, trips
|
|
`--ping-restart` after 60 s, reconnects and steals it back. Measured cycle: dev restarting at
|
|
22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC with prod interleaved about a minute
|
|
off — every ~2 minutes, indefinitely. Neither environment had reliable vpn-class egress; prod's
|
|
first probes looked green only because they landed inside an up-window.
|
|
|
|
Dev recovered the minute prod left `.112`: its last `Inactivity timeout` is 23:05:06 UTC, and at
|
|
23:12:03 that same session was still up — seven minutes unbroken against a two-minute cycle — with
|
|
a source-bound ping returning 4/4 at 146 ms and vpn-class pod egress back to `158.173.241.36`.
|
|
|
|
**Fix:** `vpn_egress_gateway_pia_remote` in `group_vars/vpn_egress_gateway_prod/vars.yml`
|
|
rewrites just the `remote` line, pointing prod at `158.173.241.135` — another member of the
|
|
same PIA Stockholm LB pool (`sweden.privacy.network` resolves to
|
|
`158.173.241.{34,55,112,135,239}`; every server in it takes the same credentials and CA). Dev
|
|
keeps `.112`.
|
|
|
|
**So: one gateway per PIA server.** A third gateway, or a rebuild that re-pins a server, must
|
|
pick an unused address from the pool. A separate prod PIA account would also solve it — see
|
|
`infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md`.
|
|
|
|
That switch also produced the cleanest possible live proof of the re-steer path, for free: prod's
|
|
tunnel address changed from `10.72.192.7` (ifindex 128) to `10.150.192.86` (ifindex 129) — a real
|
|
new PIA address on a real reconnect, the exact case Cilium does not follow. The `egress-uplink`
|
|
label moved to `10.150.192.86-129` on its own and all four vpn-class workloads kept egressing,
|
|
now via `158.173.241.11`. No manual step, no CEGP edit.
|
|
|
|
## The kill-switch does not do what this file used to claim (measured 2026-09-12)
|
|
|
|
This README, `egress-steering.yaml`, and the role all used to say the **host nftables
|
|
kill-switch** is the authoritative fail-closed backstop for `vpn`-class traffic. **It cannot be.**
|
|
With `bpf.masquerade: true` — which Egress Gateway requires — the SNAT and the egress happen in
|
|
eBPF, which `bpf_redirect()`s the skb straight to the egress device. netfilter POSTROUTING is
|
|
never invoked for those packets.
|
|
|
|
A counters-only nft table installed alongside the real one on the gateway node, while a vpn-class
|
|
pod egressed:
|
|
|
|
| counter | packets |
|
|
|---|---|
|
|
| `forward`, saddr = pod CIDR | **0** |
|
|
| `forward`, saddr = PIA tunnel range | **0** |
|
|
| `postrouting`, oif tun0, saddr = tunnel range | **0** |
|
|
| `postrouting`, oif WAN, saddr = pod CIDR | **0** |
|
|
| `postrouting`, oif WAN, any (control: the hook IS live) | 45166 |
|
|
| `postrouting`, oif tun0, saddr = tunnel (control: a node-originated curl) | 16 |
|
|
|
|
**What actually fails closed is Cilium itself.** When it cannot use its cached egress
|
|
interface/IP it drops the packet. Verified with tcpdump on *both* the WAN and tun0, during a
|
|
tunnel stop and during a tunnel restart: **0 packets left either device.** There is no leak — the
|
|
guarantee is real — but the mechanism is Cilium's, and the existing `vpn-killswitch` conformance
|
|
test passes because of it, not because of the nft rules.
|
|
|
|
The nft table is **kept**, with a `counter` on every rule, for the narrower thing it genuinely
|
|
does cover: **node-originated** traffic sourced from a tunnel address. That was proven on
|
|
`k3s-prod-3` in both directions — a source-bound ping with the tunnel ip-rule present got 3/3
|
|
replies and moved `killswitch_accept`; with the ip-rule removed (so the routing table sent it out
|
|
`eno1`) `killswitch_drop` moved **0 -> 3 packets / 252 bytes** and the ping got nothing back.
|
|
If `killswitch_drop` ever starts counting in normal operation, the datapath has changed — read
|
|
that as news, not reassurance.
|
|
|
|
Also removed: the `meta mark 0x1c8 oifname != tun0 drop` "contract hook". Nothing stamps that
|
|
mark (the only Cilium mark on the gateway node is `0x200/0xf00`, the L7-proxy rule to table
|
|
2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is
|
|
a hazard to whatever component picks that value next. The role's `vpn_egress_gateway_fwmark`
|
|
variable is gone with it, and the "add the fwmark to the interface contract" escalation in the
|
|
role README is **closed as not-implementable** rather than left open.
|
|
|
|
## Cilium never re-resolves `tun0` — the gateway dies on every reconnect without help
|
|
|
|
Cilium resolves `interface: tun0` once into a cached (egress IP, egress ifindex) and does not
|
|
revisit it. Every tunnel restart re-creates tun0 with a **new ifindex**, so the cached one goes
|
|
stale and vpn-class egress is silently blackholed. Found the hard way: the dev gateway had been
|
|
dead for ~80 minutes when this work started, with no alert and nothing in the ledger.
|
|
|
|
`egress-gateway-reconciliation-trigger-interval: 1s` does **not** help; both clusters carry that
|
|
setting and dev still did not self-heal 20s+ after the tunnel was back. It coalesces
|
|
reconciliations that k8s object events already triggered — netlink link events are not among them.
|
|
|
|
The Ansible role now publishes `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on
|
|
the gateway node on every connect, plus a 2-minute convergence timer. That node update makes
|
|
Cilium re-process the policy and re-resolve the interface (verified: cached ifindex 1986 -> 1987
|
|
within 10s, egress recovered immediately). A node label rather than an annotation on this
|
|
Flux-managed policy, so GitOps state is never written to from a node.
|
|
|
|
**Debugging a dead gateway starts here:**
|
|
|
|
```bash
|
|
kubectl get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink
|
|
kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
|
|
cilium-dbg bpf egress list # Egress Ifindex must match the node's live `ip link show tun0`
|
|
```
|
|
|
|
If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
|
|
gateway node is the next stop.
|
|
|
|
### And now something watches it (2026-09-12)
|
|
|
|
The self-heal closed the datapath hole; it did not close the monitoring one, and a mechanism that
|
|
silently stops working is the same outage again. Two rules in `platform/monitoring/alerts.yaml`:
|
|
|
|
- **`VPNEgressSteeringStale`** (critical, `for: 15m`) — fires when the live tun0 was re-created
|
|
more times in the last hour than the `egress-uplink` label changed, i.e. Cilium was never told
|
|
about a reconnect. This is the case `VPNTunnelDown` cannot see: tun0 present, bytes flowing,
|
|
steering pointed at a destroyed ifindex.
|
|
- **`VPNEgressUplinkLabelMissing`** (warning, `for: 30m`) — fires when a node carries
|
|
`egress-gateway=true` but no `egress-uplink` label reaches kube-state-metrics, which is the state
|
|
in which the rule above would be inert. The watchdog for the watchdog.
|
|
|
|
Both are gated on a gateway node existing (the same `kube_node_labels` guard `VPNTunnelDown` uses),
|
|
so a cluster with no gateway cannot be made to fire them. `egress-uplink` had to be added to
|
|
`kube-state-metrics.metricLabelsAllowlist` in `platform/monitoring/kube-prometheus-stack.yaml` for
|
|
the label to exist as a metric at all — the two changes must ship together.
|
|
|
|
**What still is not watched:** "the label was written and Cilium still ignored it." Cilium OSS
|
|
exposes no metric carrying the egress map's contents (the map is readable only through
|
|
`cilium-dbg bpf egress list`), and the cilium-agent is not scraped on either cluster in the first
|
|
place — no `cilium_*` or `hubble_*` series exist. Closing that last gap means either exporting the
|
|
map's ifindex from the re-steer script through a node-exporter textfile collector, or a black-box
|
|
probe from a vpn-class pod. Neither exists yet.
|
|
|
|
## Resilience — there is NO gateway failover (spec correction, escalated)
|
|
|
|
**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)
|
|
provide the active/standby the earlier ADRs asked for", and spec `docs/03` repeats it. That feature
|
|
does not exist in Cilium OSS.** The supported topology is **exactly one gateway node**.
|
|
|
|
Verified against the CRD actually deployed on this cluster
|
|
(`kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`, apiVersion `cilium.io/v2`,
|
|
Cilium **v1.20.1**) — the field's own description:
|
|
|
|
> **`egressGateway`** — "EgressGateway is the gateway node responsible for SNATing traffic.
|
|
> **In case multiple nodes are a match for the given set of labels, the first node in lexical
|
|
> ordering based on their name will be selected.**"
|
|
|
|
Consequences, stated plainly:
|
|
|
|
- **One gateway node is the supported topology.** `infra/ansible/inventory/hosts.ini`'s
|
|
`[vpn_egress_gateway]` group is singleton by design.
|
|
- **A second labelled node does NOT provide failover.** It is not a standby; it carries no traffic
|
|
and takes over nothing when the first node dies. Gateway-node loss = `vpn`-class egress down
|
|
(fail-closed, per R-NET-06 — no leak, but no service).
|
|
- **Lexical ordering decides who wins if two are ever labelled.** Labelling `k3s-prod-1` while
|
|
`k3s-prod-3` is the live gateway would **silently** move the role to `k3s-prod-1` — because `1`
|
|
sorts before `3` — taking the tunnel and the public egress IP with it, with no event and no alert.
|
|
- **The plural `egressGateways` field (Cilium 1.20) is not a workaround.** It *shards* endpoints
|
|
across entries by CiliumEndpoint UID rather than running active/standby: losing one entry's node
|
|
blackholes that entry's share and re-shards the remainder (breaking live connections), and each
|
|
shard leaves from a *different* public IP. Upstream confirmation:
|
|
[cilium/cilium#30157](https://github.com/cilium/cilium/issues/30157) ("egress gateway doesn't
|
|
support failover between nodes") and
|
|
[cilium/cilium#18230](https://github.com/cilium/cilium/issues/18230) ("egress gateway does not
|
|
tolerate single node failure").
|
|
|
|
**Governance:** ADR-0015 and `docs/03` are specifications and are immutable to agents
|
|
(spec `AGENTS.md` §11, R-TEST-06, enforced by `ci/spec-immutability.sh`) — they are **not** edited
|
|
here. The correction is raised as an **escalation with a drafted ADR-0016 supersede** in
|
|
`DECISIONS-LOG.md` (entry of 2026-09-06) for the operator/architect to accept. Until then this file
|
|
and `egress-steering.yaml` are the accurate statement of what the implementation can actually do.
|
|
|
|
## Secrets — host-side now
|
|
PIA credentials + the `.ovpn` live **on the gateway node via Ansible vault** (see spec `docs/07`), **not**
|
|
a cluster SOPS Secret. SOPS still governs in-cluster secrets; VPN egress simply no longer needs one.
|
|
|
|
The `pia-credentials.sops.yaml` and `pia-custom-ovpn.sops.yaml` secrets in this directory are **now
|
|
UNUSED** — leftovers from the retired Gluetun pod, **retained pending a separate cleanup change**.
|
|
|
|
## Pieces in this directory
|
|
- `namespace.yaml` — the `vpn-egress` namespace (now holds only the Cilium policy + the retained,
|
|
unused secrets; no privileged gateway pod under ADR-0015).
|
|
- `egress-steering.yaml` — the `CiliumEgressGatewayPolicy` (the steering datapath).
|
|
- `pia-credentials.sops.yaml`, `pia-custom-ovpn.sops.yaml` — **unused**, retained pending cleanup.
|
|
|
|
## History (brief)
|
|
ADR-0013 originally placed the VPN in a single shared **Gluetun pod** and deferred the steering
|
|
mechanism. Implementation proved that a Gluetun-in-a-pod tunnel **cannot be a Cilium Egress Gateway
|
|
target**: the policy SNATs out a *node* interface, but the pod's `tun0` lives in the pod netns, and
|
|
running Gluetun `hostNetwork` to expose `tun0` on the node hijacks the node's default route (breaking
|
|
k3s/etcd/kubelet). ADR-0015 resolved this by moving the tunnel to the **node** (host OpenVPN
|
|
split-tunnel) and steering with Cilium's native egress gateway. See ADR-0015 for the full rationale.
|
|
|
|
## Port forwarding (qBittorrent) — follow-up
|
|
PIA assigns a forwarded port that can change on reconnect. Under ADR-0015 this is re-solved **host-side**
|
|
(a small pf/natpmp helper on the gateway node that publishes the assigned port to the app) — Phase 6,
|
|
alongside the qBittorrent app.
|