jakman85andClaude Opus 5
a3f4ddb971
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
124 lines
7.0 KiB
Markdown
124 lines
7.0 KiB
Markdown
# vpn-egress-stale — VPN egress steering did not follow the tunnel
|
|
|
|
**Alerts:** `VPNEgressSteeringStale` (critical, 15m) and `VPNEgressUplinkLabelMissing` (warning, 30m)
|
|
**File:** `platform/monitoring/alerts.yaml`
|
|
**Related:** `/runbooks/vpn-down` is for the tunnel being DOWN. This page is for the tunnel being **up**
|
|
while the pods that must use it have no egress at all.
|
|
|
|
## What this means here
|
|
|
|
`egress-class=vpn` pods leave the cluster through one gateway node's `tun0`, selected by the
|
|
`vpn-egress-steering` CiliumEgressGatewayPolicy. Cilium resolves that interface to an **ifindex** and
|
|
caches it. When OpenVPN reconnects it destroys and re-creates `tun0`, which gets a **new ifindex** — and
|
|
Cilium does not re-resolve on its own. The gateway node publishes
|
|
`homelab.keithjackson.dev/egress-uplink=<tun0 address>-<ifindex>` from the OpenVPN up-script on every
|
|
connect, and that node write is what makes Cilium re-resolve.
|
|
|
|
If the label write does not happen, the cache keeps pointing at a destroyed interface. **`tun0` looks
|
|
perfectly healthy the whole time** — it is up, it has an address, `VPNTunnelDown` stays quiet — while
|
|
every vpn-class pod is silently dropped. That exact shape ran for roughly **80 minutes** undetected on
|
|
the dev gateway, which is why these two alerts exist.
|
|
|
|
`VPNEgressSteeringStale` compares how many times `tun0`'s ifindex changed in the last hour against how
|
|
many distinct `egress-uplink` label values appeared in the same hour. More tunnel changes than label
|
|
changes means a reconnect happened that the label did not follow.
|
|
|
|
**`VPNEgressUplinkLabelMissing` is the watchdog for that comparison.** If the label is absent entirely,
|
|
the staleness rule has nothing to compare against and is **inert** — present, evaluating, and incapable
|
|
of firing. So the warning matters more than its severity suggests: it means your critical alert is
|
|
switched off.
|
|
|
|
Today's gateway is **`k3s-prod-3`** (`homelab.keithjackson.dev/egress-gateway=true`), and the policy has
|
|
been in place since early September. It is a **singleton** — there is no second gateway to fail over to.
|
|
|
|
## First commands
|
|
|
|
Run cluster commands through the pinned toolbox, which supplies the kube context:
|
|
|
|
```
|
|
./infra/devcontainer/dev.sh run kubectl --context k3s-prod get nodes \
|
|
-L homelab.keithjackson.dev/egress-gateway -L homelab.keithjackson.dev/egress-uplink
|
|
```
|
|
|
|
You want exactly one node carrying `egress-gateway=true`, and that same node carrying a non-empty
|
|
`egress-uplink` of the form `<address>-<ifindex>`.
|
|
|
|
```
|
|
# 2. what Cilium currently believes, on the gateway node itself
|
|
ssh k3s-prod-3 'sudo cilium-dbg bpf egress list'
|
|
ssh k3s-prod-3 'ip -o link show tun0' # the REAL ifindex, first field
|
|
```
|
|
|
|
Compare the ifindex in the label and in `bpf egress list` against `ip link`. **A mismatch is the whole
|
|
bug.**
|
|
|
|
```
|
|
# 3. is the re-steer mechanism itself alive?
|
|
ssh k3s-prod-3 'systemctl status vpn-egress-resteer.service vpn-egress-resteer.timer --no-pager'
|
|
ssh k3s-prod-3 'journalctl -u vpn-egress-resteer --since -2h --no-pager | tail -30'
|
|
```
|
|
|
|
```
|
|
# 4. are vpn-class pods actually broken, or only the metric?
|
|
./infra/devcontainer/dev.sh run kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io \
|
|
vpn-egress-steering -o yaml | head -40
|
|
```
|
|
|
|
Prove egress from a pod the policy selects rather than inferring it — the observable symptom is
|
|
"outbound traffic silently disappears", so test outbound traffic:
|
|
|
|
```
|
|
./infra/devcontainer/dev.sh run kubectl --context k3s-prod -n qbittorrent exec deploy/qbittorrent -- \
|
|
sh -c 'curl -s --max-time 10 https://ifconfig.me || echo EGRESS-FAILED'
|
|
```
|
|
|
|
That should return the **VPN provider's** address. The gateway VIP or a timeout both mean egress is
|
|
broken; your own ISP address means the kill-switch is not holding and traffic is leaking, which is worse
|
|
than a drop — see `/runbooks/vpn-down`.
|
|
|
|
## Likely causes, most likely first
|
|
|
|
1. **A tunnel reconnect that the label did not follow.** The common case, and what the alert is built to
|
|
catch. The OpenVPN up-script either did not run or could not write the node label (RBAC, expired
|
|
kubeconfig on the node, API unreachable at that moment). Check `journalctl -u vpn-egress-resteer` and
|
|
the up-script's own log around the reconnect time.
|
|
2. **Cilium cached a dead ifindex despite the label changing.** The label write happened but the
|
|
re-resolution did not take. Forcing any node-label write makes Cilium re-resolve — that is the same
|
|
lever the up-script pulls, so it is a safe nudge, not a workaround:
|
|
|
|
./infra/devcontainer/dev.sh run kubectl --context k3s-prod annotate node k3s-prod-3 \
|
|
egress-resteer-nudge="$(date -u +%s)" --overwrite
|
|
|
|
Re-applying the CiliumEgressGatewayPolicy has the same effect. Re-check `bpf egress list` after.
|
|
3. **`metricLabelsAllowlist` lost the label** (this fires `VPNEgressUplinkLabelMissing`, not the stale
|
|
alert). kube-state-metrics exposes **no** custom node labels by default; the allowlist entry in
|
|
`platform/monitoring/kube-prometheus-stack.yaml` is what publishes them. If that entry is edited or
|
|
dropped, `kube_node_labels` carries no `label_homelab_keithjackson_dev_egress_uplink` and the critical
|
|
alert goes quiet without anything looking broken. Confirm what is actually exported:
|
|
|
|
./infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
|
|
prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
|
|
/bin/promtool query instant http://localhost:9090 'kube_node_labels'
|
|
|
|
Note that KSM does **not** restart on a ConfigMap-only change; the deployment carries no checksum
|
|
annotation over it, so a `rollout restart -n monitoring deploy/kube-prometheus-stack-kube-state-metrics`
|
|
is required after editing the allowlist.
|
|
4. **The gateway node was replaced or relabelled.** If `egress-gateway=true` moved, the up-script must be
|
|
installed on the new node (`infra/ansible/roles/vpn_egress_gateway`). A gateway label with no
|
|
up-script behind it produces exactly this alert on the next reconnect.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Stop and do not "fix" it by deleting the policy.** Removing `vpn-egress-steering` does not restore
|
|
egress; it sends vpn-class traffic out of your **own** ISP connection, which is the leak the class
|
|
exists to prevent. A broken tunnel that drops traffic is the intended failure mode.
|
|
- **Escalate to a maintenance window** if the ifindex mismatch returns after every reconnect. That is the
|
|
up-script or its RBAC, not a transient, and chasing it live means repeatedly breaking egress for the
|
|
vpn-class workloads.
|
|
- **This is a singleton gateway.** There is no failover. If `k3s-prod-3` needs to be drained while this is
|
|
unresolved, accept that vpn-class egress is down for the duration rather than improvising a second
|
|
gateway under time pressure.
|
|
- If `VPNEgressUplinkLabelMissing` is firing and you cannot restore the label quickly, say so explicitly
|
|
in the incident notes: for as long as it is missing, **`VPNEgressSteeringStale` is not watching
|
|
anything**, so the absence of that critical alert proves nothing.
|