# vpn-egress-stale — VPN egress steering did not follow the tunnel

**Alerts:** `VPNEgressSteeringStale` (critical, 15m) and `VPNEgressUplinkLabelMissing` (warning, 30m)
**File:** `platform/monitoring/alerts.yaml`
**Related:** `/runbooks/vpn-down` is for the tunnel being DOWN. This page is for the tunnel being **up**
while the pods that must use it have no egress at all.

## What this means here

`egress-class=vpn` pods leave the cluster through one gateway node's `tun0`, selected by the
`vpn-egress-steering` CiliumEgressGatewayPolicy. Cilium resolves that interface to an **ifindex** and
caches it. When OpenVPN reconnects it destroys and re-creates `tun0`, which gets a **new ifindex** — and
Cilium does not re-resolve on its own. The gateway node publishes
`homelab.keithjackson.dev/egress-uplink=<tun0 address>-<ifindex>` from the OpenVPN up-script on every
connect, and that node write is what makes Cilium re-resolve.

If the label write does not happen, the cache keeps pointing at a destroyed interface. **`tun0` looks
perfectly healthy the whole time** — it is up, it has an address, `VPNTunnelDown` stays quiet — while
every vpn-class pod is silently dropped. That exact shape ran for roughly **80 minutes** undetected on
the dev gateway, which is why these two alerts exist.

`VPNEgressSteeringStale` compares how many times `tun0`'s ifindex changed in the last hour against how
many distinct `egress-uplink` label values appeared in the same hour. More tunnel changes than label
changes means a reconnect happened that the label did not follow.

**`VPNEgressUplinkLabelMissing` is the watchdog for that comparison.** If the label is absent entirely,
the staleness rule has nothing to compare against and is **inert** — present, evaluating, and incapable
of firing. So the warning matters more than its severity suggests: it means your critical alert is
switched off.

Today's gateway is **`k3s-prod-3`** (`homelab.keithjackson.dev/egress-gateway=true`), and the policy has
been in place since early September. It is a **singleton** — there is no second gateway to fail over to.

## First commands

Run cluster commands through the pinned toolbox, which supplies the kube context:

```
./infra/devcontainer/dev.sh run kubectl --context k3s-prod get nodes \
  -L homelab.keithjackson.dev/egress-gateway -L homelab.keithjackson.dev/egress-uplink
```

You want exactly one node carrying `egress-gateway=true`, and that same node carrying a non-empty
`egress-uplink` of the form `<address>-<ifindex>`.

```
# 2. what Cilium currently believes, on the gateway node itself
ssh k3s-prod-3 'sudo cilium-dbg bpf egress list'
ssh k3s-prod-3 'ip -o link show tun0'          # the REAL ifindex, first field
```

Compare the ifindex in the label and in `bpf egress list` against `ip link`. **A mismatch is the whole
bug.**

```
# 3. is the re-steer mechanism itself alive?
ssh k3s-prod-3 'systemctl status vpn-egress-resteer.service vpn-egress-resteer.timer --no-pager'
ssh k3s-prod-3 'journalctl -u vpn-egress-resteer --since -2h --no-pager | tail -30'
```

```
# 4. are vpn-class pods actually broken, or only the metric?
./infra/devcontainer/dev.sh run kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io \
  vpn-egress-steering -o yaml | head -40
```

Prove egress from a pod the policy selects rather than inferring it — the observable symptom is
"outbound traffic silently disappears", so test outbound traffic:

```
./infra/devcontainer/dev.sh run kubectl --context k3s-prod -n qbittorrent exec deploy/qbittorrent -- \
  sh -c 'curl -s --max-time 10 https://ifconfig.me || echo EGRESS-FAILED'
```

That should return the **VPN provider's** address. The gateway VIP or a timeout both mean egress is
broken; your own ISP address means the kill-switch is not holding and traffic is leaking, which is worse
than a drop — see `/runbooks/vpn-down`.

## Likely causes, most likely first

1. **A tunnel reconnect that the label did not follow.** The common case, and what the alert is built to
   catch. The OpenVPN up-script either did not run or could not write the node label (RBAC, expired
   kubeconfig on the node, API unreachable at that moment). Check `journalctl -u vpn-egress-resteer` and
   the up-script's own log around the reconnect time.
2. **Cilium cached a dead ifindex despite the label changing.** The label write happened but the
   re-resolution did not take. Forcing any node-label write makes Cilium re-resolve — that is the same
   lever the up-script pulls, so it is a safe nudge, not a workaround:

       ./infra/devcontainer/dev.sh run kubectl --context k3s-prod annotate node k3s-prod-3 \
         egress-resteer-nudge="$(date -u +%s)" --overwrite

   Re-applying the CiliumEgressGatewayPolicy has the same effect. Re-check `bpf egress list` after.
3. **`metricLabelsAllowlist` lost the label** (this fires `VPNEgressUplinkLabelMissing`, not the stale
   alert). kube-state-metrics exposes **no** custom node labels by default; the allowlist entry in
   `platform/monitoring/kube-prometheus-stack.yaml` is what publishes them. If that entry is edited or
   dropped, `kube_node_labels` carries no `label_homelab_keithjackson_dev_egress_uplink` and the critical
   alert goes quiet without anything looking broken. Confirm what is actually exported:

       ./infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
         prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
         /bin/promtool query instant http://localhost:9090 'kube_node_labels'

   Note that KSM does **not** restart on a ConfigMap-only change; the deployment carries no checksum
   annotation over it, so a `rollout restart -n monitoring deploy/kube-prometheus-stack-kube-state-metrics`
   is required after editing the allowlist.
4. **The gateway node was replaced or relabelled.** If `egress-gateway=true` moved, the up-script must be
   installed on the new node (`infra/ansible/roles/vpn_egress_gateway`). A gateway label with no
   up-script behind it produces exactly this alert on the next reconnect.

## When to escalate or stop

- **Stop and do not "fix" it by deleting the policy.** Removing `vpn-egress-steering` does not restore
  egress; it sends vpn-class traffic out of your **own** ISP connection, which is the leak the class
  exists to prevent. A broken tunnel that drops traffic is the intended failure mode.
- **Escalate to a maintenance window** if the ifindex mismatch returns after every reconnect. That is the
  up-script or its RBAC, not a transient, and chasing it live means repeatedly breaking egress for the
  vpn-class workloads.
- **This is a singleton gateway.** There is no failover. If `k3s-prod-3` needs to be drained while this is
  unresolved, accept that vpn-class egress is down for the duration rather than improvising a second
  gateway under time pressure.
- If `VPNEgressUplinkLabelMissing` is firing and you cannot restore the label quickly, say so explicitly
  in the incident notes: for as long as it is missing, **`VPNEgressSteeringStale` is not watching
  anything**, so the absence of that critical alert proves nothing.