feat(monitoring): alert when egress steering goes stale, not just when tun0 dies

`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as
vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first
resolution, so after a reconnect the BPF egress map points at a destroyed
interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat
in exactly that state for ~80 minutes with nothing alerting.

Two rules close it:

* VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was
  re-created more times in the last hour than the gateway node's `egress-uplink`
  label changed, i.e. Cilium was never told about a reconnect.
* VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the
  watchdog: a gateway node whose uplink label never reaches kube-state-metrics is
  a cluster where the rule above is inert.

Both gated on a gateway node existing, reusing `VPNTunnelDown`'s
`kube_node_labels` guard — a cluster with no gateway cannot be made to fire them.

The direct label-vs-map comparison is NOT available and nothing was invented to
pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the
cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS
publishes no series carrying egress-map contents at all — the map is readable only
via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the
live ifindex) against a number inside a label value, so the level-triggered
equality test is not expressible; the same invariant is counted over a window
instead. Both limits, and the textfile-collector/black-box-probe routes that would
lift them, are written down in the rule comments and the DECISIONS-LOG.

`egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same
kustomization, because without it the first rule has nothing to compare.

Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr
extracted from `kubectl kustomize platform-config`: empty when healthy; empty when
evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at
a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier
today. Nothing on prod was disturbed to produce that.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-12 20:46:56 -05:00
1 parent 19fc8bcb27
commit 16c6234696
4 files changed
+306 -1

No files matched your search

+25
View File
@@ -157,6 +157,31 @@ kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
gateway node is the next stop.
### And now something watches it (2026-09-12)
The self-heal closed the datapath hole; it did not close the monitoring one, and a mechanism that
silently stops working is the same outage again. Two rules in `platform/monitoring/alerts.yaml`:
- **`VPNEgressSteeringStale`** (critical, `for: 15m`) — fires when the live tun0 was re-created
more times in the last hour than the `egress-uplink` label changed, i.e. Cilium was never told
about a reconnect. This is the case `VPNTunnelDown` cannot see: tun0 present, bytes flowing,
steering pointed at a destroyed ifindex.
- **`VPNEgressUplinkLabelMissing`** (warning, `for: 30m`) — fires when a node carries
`egress-gateway=true` but no `egress-uplink` label reaches kube-state-metrics, which is the state
in which the rule above would be inert. The watchdog for the watchdog.
Both are gated on a gateway node existing (the same `kube_node_labels` guard `VPNTunnelDown` uses),
so a cluster with no gateway cannot be made to fire them. `egress-uplink` had to be added to
`kube-state-metrics.metricLabelsAllowlist` in `platform/monitoring/kube-prometheus-stack.yaml` for
the label to exist as a metric at all — the two changes must ship together.
**What still is not watched:** "the label was written and Cilium still ignored it." Cilium OSS
exposes no metric carrying the egress map's contents (the map is readable only through
`cilium-dbg bpf egress list`), and the cilium-agent is not scraped on either cluster in the first
place — no `cilium_*` or `hubble_*` series exist. Closing that last gap means either exporting the
map's ifindex from the re-steer script through a node-exporter textfile collector, or a black-box
probe from a vpn-class pod. Neither exists yet.
## Resilience — there is NO gateway failover (spec correction, escalated)
**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)