# vpn-down — The egress VPN tunnel is down **Alert:** `VPNTunnelDown` (critical, 5m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.networking` **Expression:** ``` absent(node_network_receive_bytes_total{device="tun0"}) and on() ( (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0) or (count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0) ) ``` ## What this means here There is **no in-cluster egress-gateway pod** on this platform (ADR-0015). The tunnel is a **host systemd service** — `vpn-egress-tunnel`, OpenVPN to PIA — running on one node, and steering is Cilium eBPF via a `CiliumEgressGatewayPolicy` (`vpn-egress-steering`). OpenVPN creates and destroys `tun0` with the connection, so the `tun0` byte-counter series exists **if and only if** the tunnel is up. The rule keys on that presence rather than `node_network_up`, because `tun0` reports `operstate=unknown` and `node_network_up` is therefore 0 even when the tunnel is fine. Live state, measured 2026-09-13: **the tunnel is up** (`node_network_receive_bytes_total{device="tun0"}` = 1 series) and the gateway node is labelled (`kube_node_labels{...egress_gateway="true"}` = 1 series), so this alert is armed and quiet. The gateway is **`k3s-prod-3`** (192.168.2.12, `eno1`). **The two facts that decide your response:** 1. **Cilium OSS has no egress-gateway failover.** Gateway-node loss is an outage, not a degradation. The ADR-0016 supersede drafted on 2026-09-06 is still open. Worse, Cilium OSS picks *"the first node in lexical ordering"* among nodeSelector matches — so labelling `k3s-prod-1` would **silently steal** the gateway role and the public egress IP from `k3s-prod-3`. `[vpn_egress_gateway_prod]` is a separate inventory group holding only `k3s-prod-3` for this reason; run the role as `-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3`, with `--list-hosts` first, every time. 2. **The `absent()` was originally evaluated cluster-wide and became a permanent false alarm.** On a cluster with no gateway node it could never clear, and from 2026-09-05T21:23Z it fired continuously into `email-critical` — unbounded noise on the one channel that must stay trustworthy. The two guards OR'd above are the fix: the alert arms itself the moment a node is labelled, and stays armed through a kube-state-metrics outage. Before that, the very first version keyed on `absent(gluetun_vpn_up)`, a metric gluetun's control API never exported at all — a pre-existing perma-false-alarm. **If this alert is firing, verify the tunnel by hand before believing it.** ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Confirm on the host, not from Prometheus: ``` ssh k3s-prod-3 'systemctl status vpn-egress-tunnel --no-pager | head -20; echo; ip -br addr show tun0; echo; ip link show tun0 | head -2' ``` Which node is the gateway, and is it labelled? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink; echo; \ kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io -o yaml | grep -A6 "nodeSelector\|egressIP\|interface"' ``` **The check that catches the silent failure** (see cause 1 below) — does Cilium's datapath point at the interface that actually exists? ``` ssh k3s-prod-3 'ip -o link | awk -F: "{print \$1, \$2}" | grep tun0' infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg bpf egress list' ``` If the `Egress Ifindex` in that map is not the live `tun0` ifindex, egress is broken **even though this alert is quiet**. Is traffic actually leaving through the tunnel? ``` ssh k3s-prod-3 'curl -s --interface tun0 --max-time 10 https://ipinfo.io/ip; echo; nft list table inet vpn_killswitch 2>/dev/null | grep -E "counter|killswitch_" | head' ``` ## Likely causes, most likely first ### 1. The tunnel is UP but Cilium is steering at a dead interface (the silent one) This is the richest failure on the platform and it does **not** set off this alert. **Cilium resolves a `CiliumEgressGatewayPolicy`'s `interface: tun0` exactly once**, caches (egress IP, ifindex), and never re-resolves. On `k3s-dev-3`, `cilium-dbg bpf egress list` showed `Egress Ifindex 3` — **there is no ifindex 3 on that node**; the live `tun0` was **1780**. Egress was dead for about 80 minutes with nothing alerting and the conformance ledger still reading green. `egress-gateway-reconciliation-trigger-interval: 1s` does **not** cover it (netlink link events are not Kubernetes object events). Measured behaviour, so you can pick a lever that works: | action | result | |---|---| | annotate the CEGP | map → 1780 in 8s, PIA IP returns | | `systemctl restart vpn-egress-tunnel` | tun0 1780→1985, **map stays 1780**, still broken | | the exact `vpn-killswitch` stop/30s/start sequence | still broken 20s+ later | | set **any** unrelated label on the gateway **node** | recovers within 10s | The durable fix is in `roles/vpn_egress_gateway`: the OpenVPN up-script publishes a node label `homelab.keithjackson.dev/egress-uplink=-` on every connect (via `systemctl start --no-block`, so it can never block the tunnel), plus a 2-minute convergence timer for boot. A node label, not a CEGP annotation — a node writing to GitOps state is a drift fight. **Nothing monitors this coupling yet**; that gap is exactly why dev sat dead for 80 minutes. ### 2. PIA handed two gateways the same tunnel address Standing rule: **one PIA server per gateway.** The vaulted `.ovpn` pins one server, so when prod came up green, dev went down: both gateways dialled `158.173.241.112:1198` and PIA gave both the tunnel address `10.72.192.7`. Each reconnect steals it — a measured ~2-minute mutual-eviction cycle (dev restarting at 22:57:05, 22:59:03, 23:01:05, 23:03:05, 23:05:06 UTC). Fix: the role var `vpn_egress_gateway_pia_remote` ("` `") rewrites only the `.ovpn` `remote` line; prod is pinned to **`158.173.241.135`** (pool `158.173.241.{34,55,112,135,239}`). *"A single green probe is not a stability check."* ### 3. PIA "unreachable" — suspect a stale server list first The 2026-08-22/23 chain went port collision → UDP/53 → 8080 → "MTU" (Cilium VXLAN pod MTU 1450 is real but irrelevant to a **14-byte** handshake) → "the network blackholes PIA" (**wrong**) before the actual cause: **gluetun's stale bundled server list**, dialling retired `46.246.x`/`212.112.x` addresses when live SE Stockholm is `158.173.241.x`. Lesson recorded verbatim: when a client "can't reach" the provider, suspect the stale server list **first**. ### 4. The gateway node is down `absent()` also fires when the gateway node or its node-exporter is gone, so this alert covers gateway-node-down by design. Go to `/runbooks/node-down`. Remember: **there is no failover** — while `k3s-prod-3` is down, VPN-classed egress is an outage. ### 5. Nothing is wrong and the alert is lying Given the two false-alarm post-mortems above, always confirm `ip -br addr show tun0` on the host before acting. ## What the kill-switch does and does not do Proven with counters on `k3s-prod-3`, because the intuitive reading is wrong: with **`bpf.masquerade: true`** (which Egress Gateway requires) SNAT and egress happen in eBPF and `bpf_redirect()` **skips netfilter POSTROUTING**. Measured: `forward` saddr=pod-CIDR **0**, `postrouting` oif WAN saddr=pod-CIDR **0**. **What actually fails closed is Cilium's own drop** — tcpdump during tunnel stop and restart showed **0 packets leaving either device**. The nft table is kept as defence-in-depth for node-originated tunnel-sourced traffic and was proven both ways: `ping -I 10.72.192.7` with the ip-rule → 3/3 at 144 ms with `killswitch_accept` advancing; rule removed → `killswitch_drop` 0→3 packets / 252 bytes, 100% loss. Every rule carries a `counter`, so: **if `killswitch_drop` ever counts in normal operation, the datapath has changed.** A reported "leak window" was **measured as no WAN leak** — a stale egress IP follows the cached ifindex, not a routing lookup. Note the `.nft` file uses `table` + `flush table`, which clears rules but leaves named-set elements intact. ## When to escalate or stop - **Escalate immediately if the gateway node is down.** No failover exists; this is an outage with no in-cluster remedy. - **Never label a second node `egress-gateway=true`.** Lexical ordering means `k3s-prod-1` would silently take the role and the public egress IP. - **Do not restart the tunnel expecting Cilium to follow** — measured above, it does not. If the map is stale, touching a node label is the lever that works (operator action). - **Do not remove or "simplify" the nft kill-switch** because the counters read zero. Zero on the pod path is the expected, measured result of eBPF masquerade; the table guards a different path. - A leak concern is a security matter (docs/07) — escalate rather than experimenting. --- Source of truth: `runbooks/vpn-down.md`. Index: `/runbooks/` Records: `DECISIONS-LOG.md` "2026-09-12 — VPN egress on PROD (k3s-prod-3), and three things the repo believed that are not true"; the 2026-08-22/23 gluetun chain; commit `155df8e` (R-NET-06 re-proven on prod, recovery ≤3s).