0 Fork 0

Files

T

jakman85andClaude Opus 5 a3f4ddb971 merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate

Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.

WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:

  * alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
    which did not exist. An alert firing now would hand the operator a 404.
  * alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
    in it must be tested or a "tested" file quietly grows untested rules.
  * alert-metrics-have-scrape-targets — see the guard bug below.

WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.

Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.

runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.

platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.

Two fixes the branch needed that were defects in the checks themselves:

1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
   `nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
   contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
   which these alerts need) therefore FAILED the check while the property it guards
   was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
   guard that breaks when you correctly extend the thing it guards teaches people to
   edit the guard, which is how it stops guarding.

2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
   "asserted on" is a bare word match over the test file — my header COMMENT
   mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
   register entry and claim coverage that did not exist, the fixtures now assert
   VPNTunnelDown QUIET in all four cases. That required making them faithful: they
   had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
   would have been firing throughout unnoticed, because promtool only checks the
   alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
   these rules exist for.

NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.

Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.

Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-19 00:52:06 -05:00

7.0 KiB

Raw Permalink Blame History

vpn-egress-stale — VPN egress steering did not follow the tunnel

Alerts: VPNEgressSteeringStale (critical, 15m) and VPNEgressUplinkLabelMissing (warning, 30m) File: platform/monitoring/alerts.yaml Related: /runbooks/vpn-down is for the tunnel being DOWN. This page is for the tunnel being up while the pods that must use it have no egress at all.

What this means here

egress-class=vpn pods leave the cluster through one gateway node's tun0, selected by the vpn-egress-steering CiliumEgressGatewayPolicy. Cilium resolves that interface to an ifindex and caches it. When OpenVPN reconnects it destroys and re-creates tun0, which gets a new ifindex — and Cilium does not re-resolve on its own. The gateway node publishes homelab.keithjackson.dev/egress-uplink=<tun0 address>-<ifindex> from the OpenVPN up-script on every connect, and that node write is what makes Cilium re-resolve.

If the label write does not happen, the cache keeps pointing at a destroyed interface. tun0 looks perfectly healthy the whole time — it is up, it has an address, VPNTunnelDown stays quiet — while every vpn-class pod is silently dropped. That exact shape ran for roughly 80 minutes undetected on the dev gateway, which is why these two alerts exist.

VPNEgressSteeringStale compares how many times tun0's ifindex changed in the last hour against how many distinct egress-uplink label values appeared in the same hour. More tunnel changes than label changes means a reconnect happened that the label did not follow.

VPNEgressUplinkLabelMissing is the watchdog for that comparison. If the label is absent entirely, the staleness rule has nothing to compare against and is inert — present, evaluating, and incapable of firing. So the warning matters more than its severity suggests: it means your critical alert is switched off.

Today's gateway is k3s-prod-3 (homelab.keithjackson.dev/egress-gateway=true), and the policy has been in place since early September. It is a singleton — there is no second gateway to fail over to.

First commands

Run cluster commands through the pinned toolbox, which supplies the kube context:

./infra/devcontainer/dev.sh run kubectl --context k3s-prod get nodes \
  -L homelab.keithjackson.dev/egress-gateway -L homelab.keithjackson.dev/egress-uplink

You want exactly one node carrying egress-gateway=true, and that same node carrying a non-empty egress-uplink of the form <address>-<ifindex>.

# 2. what Cilium currently believes, on the gateway node itself
ssh k3s-prod-3 'sudo cilium-dbg bpf egress list'
ssh k3s-prod-3 'ip -o link show tun0'          # the REAL ifindex, first field

Compare the ifindex in the label and in bpf egress list against ip link. A mismatch is the whole bug.

# 3. is the re-steer mechanism itself alive?
ssh k3s-prod-3 'systemctl status vpn-egress-resteer.service vpn-egress-resteer.timer --no-pager'
ssh k3s-prod-3 'journalctl -u vpn-egress-resteer --since -2h --no-pager | tail -30'

# 4. are vpn-class pods actually broken, or only the metric?
./infra/devcontainer/dev.sh run kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io \
  vpn-egress-steering -o yaml | head -40

Prove egress from a pod the policy selects rather than inferring it — the observable symptom is "outbound traffic silently disappears", so test outbound traffic:

./infra/devcontainer/dev.sh run kubectl --context k3s-prod -n qbittorrent exec deploy/qbittorrent -- \
  sh -c 'curl -s --max-time 10 https://ifconfig.me || echo EGRESS-FAILED'

That should return the VPN provider's address. The gateway VIP or a timeout both mean egress is broken; your own ISP address means the kill-switch is not holding and traffic is leaking, which is worse than a drop — see /runbooks/vpn-down.

Likely causes, most likely first

A tunnel reconnect that the label did not follow. The common case, and what the alert is built to catch. The OpenVPN up-script either did not run or could not write the node label (RBAC, expired kubeconfig on the node, API unreachable at that moment). Check journalctl -u vpn-egress-resteer and the up-script's own log around the reconnect time.
Cilium cached a dead ifindex despite the label changing. The label write happened but the re-resolution did not take. Forcing any node-label write makes Cilium re-resolve — that is the same lever the up-script pulls, so it is a safe nudge, not a workaround:
```
./infra/devcontainer/dev.sh run kubectl --context k3s-prod annotate node k3s-prod-3 \
  egress-resteer-nudge="$(date -u +%s)" --overwrite
```
Re-applying the CiliumEgressGatewayPolicy has the same effect. Re-check bpf egress list after.
metricLabelsAllowlist lost the label (this fires VPNEgressUplinkLabelMissing, not the stale alert). kube-state-metrics exposes no custom node labels by default; the allowlist entry in platform/monitoring/kube-prometheus-stack.yaml is what publishes them. If that entry is edited or dropped, kube_node_labels carries no label_homelab_keithjackson_dev_egress_uplink and the critical alert goes quiet without anything looking broken. Confirm what is actually exported:
```
./infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
  prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
  /bin/promtool query instant http://localhost:9090 'kube_node_labels'
```
Note that KSM does not restart on a ConfigMap-only change; the deployment carries no checksum annotation over it, so a rollout restart -n monitoring deploy/kube-prometheus-stack-kube-state-metrics is required after editing the allowlist.
The gateway node was replaced or relabelled. If egress-gateway=true moved, the up-script must be installed on the new node (infra/ansible/roles/vpn_egress_gateway). A gateway label with no up-script behind it produces exactly this alert on the next reconnect.

When to escalate or stop

Stop and do not "fix" it by deleting the policy. Removing vpn-egress-steering does not restore egress; it sends vpn-class traffic out of your own ISP connection, which is the leak the class exists to prevent. A broken tunnel that drops traffic is the intended failure mode.
Escalate to a maintenance window if the ifindex mismatch returns after every reconnect. That is the up-script or its RBAC, not a transient, and chasing it live means repeatedly breaking egress for the vpn-class workloads.
This is a singleton gateway. There is no failover. If k3s-prod-3 needs to be drained while this is unresolved, accept that vpn-class egress is down for the duration rather than improvising a second gateway under time pressure.
If VPNEgressUplinkLabelMissing is firing and you cannot restore the label quickly, say so explicitly in the incident notes: for as long as it is missing, VPNEgressSteeringStale is not watching anything, so the absence of that critical alert proves nothing.

7.0 KiB Raw Permalink Blame History