feat(monitoring): alert when egress steering goes stale, not just when tun0 dies
`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first resolution, so after a reconnect the BPF egress map points at a destroyed interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat in exactly that state for ~80 minutes with nothing alerting. Two rules close it: * VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was re-created more times in the last hour than the gateway node's `egress-uplink` label changed, i.e. Cilium was never told about a reconnect. * VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the watchdog: a gateway node whose uplink label never reaches kube-state-metrics is a cluster where the rule above is inert. Both gated on a gateway node existing, reusing `VPNTunnelDown`'s `kube_node_labels` guard — a cluster with no gateway cannot be made to fire them. The direct label-vs-map comparison is NOT available and nothing was invented to pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS publishes no series carrying egress-map contents at all — the map is readable only via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the live ifindex) against a number inside a label value, so the level-triggered equality test is not expressible; the same invariant is counted over a window instead. Both limits, and the textfile-collector/black-box-probe routes that would lift them, are written down in the rule comments and the DECISIONS-LOG. `egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same kustomization, because without it the first rule has nothing to compare. Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr extracted from `kubectl kustomize platform-config`: empty when healthy; empty when evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier today. Nothing on prod was disturbed to produce that. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
19fc8bcb27
commit
16c6234696
4 files changed
+306
-1
No files matched your search
@@ -157,6 +157,31 @@ kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
|
||||
If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
|
||||
gateway node is the next stop.
|
||||
|
||||
### And now something watches it (2026-09-12)
|
||||
|
||||
The self-heal closed the datapath hole; it did not close the monitoring one, and a mechanism that
|
||||
silently stops working is the same outage again. Two rules in `platform/monitoring/alerts.yaml`:
|
||||
|
||||
- **`VPNEgressSteeringStale`** (critical, `for: 15m`) — fires when the live tun0 was re-created
|
||||
more times in the last hour than the `egress-uplink` label changed, i.e. Cilium was never told
|
||||
about a reconnect. This is the case `VPNTunnelDown` cannot see: tun0 present, bytes flowing,
|
||||
steering pointed at a destroyed ifindex.
|
||||
- **`VPNEgressUplinkLabelMissing`** (warning, `for: 30m`) — fires when a node carries
|
||||
`egress-gateway=true` but no `egress-uplink` label reaches kube-state-metrics, which is the state
|
||||
in which the rule above would be inert. The watchdog for the watchdog.
|
||||
|
||||
Both are gated on a gateway node existing (the same `kube_node_labels` guard `VPNTunnelDown` uses),
|
||||
so a cluster with no gateway cannot be made to fire them. `egress-uplink` had to be added to
|
||||
`kube-state-metrics.metricLabelsAllowlist` in `platform/monitoring/kube-prometheus-stack.yaml` for
|
||||
the label to exist as a metric at all — the two changes must ship together.
|
||||
|
||||
**What still is not watched:** "the label was written and Cilium still ignored it." Cilium OSS
|
||||
exposes no metric carrying the egress map's contents (the map is readable only through
|
||||
`cilium-dbg bpf egress list`), and the cilium-agent is not scraped on either cluster in the first
|
||||
place — no `cilium_*` or `hubble_*` series exist. Closing that last gap means either exporting the
|
||||
map's ifindex from the re-steer script through a node-exporter textfile collector, or a black-box
|
||||
probe from a vpn-class pod. Neither exists yet.
|
||||
|
||||
## Resilience — there is NO gateway failover (spec correction, escalated)
|
||||
|
||||
**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)
|
||||
|
||||
Reference in new issue
Block a user