feat(monitoring): alert when egress steering goes stale, not just when tun0 dies

`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as
vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first
resolution, so after a reconnect the BPF egress map points at a destroyed
interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat
in exactly that state for ~80 minutes with nothing alerting.

Two rules close it:

* VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was
  re-created more times in the last hour than the gateway node's `egress-uplink`
  label changed, i.e. Cilium was never told about a reconnect.
* VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the
  watchdog: a gateway node whose uplink label never reaches kube-state-metrics is
  a cluster where the rule above is inert.

Both gated on a gateway node existing, reusing `VPNTunnelDown`'s
`kube_node_labels` guard — a cluster with no gateway cannot be made to fire them.

The direct label-vs-map comparison is NOT available and nothing was invented to
pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the
cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS
publishes no series carrying egress-map contents at all — the map is readable only
via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the
live ifindex) against a number inside a label value, so the level-triggered
equality test is not expressible; the same invariant is counted over a window
instead. Both limits, and the textfile-collector/black-box-probe routes that would
lift them, are written down in the rule comments and the DECISIONS-LOG.

`egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same
kustomization, because without it the first rule has nothing to compare.

Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr
extracted from `kubectl kustomize platform-config`: empty when healthy; empty when
evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at
a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier
today. Nothing on prod was disturbed to produce that.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/monitoring/kube-prometheus-stack.yaml

@@ -230,8 +230,20 @@ spec:
       # without anyone having to remember to re-enable it later. Scoped to one label on one resource:
       # 3 extra series, no cardinality concern. Label name is sanitized by KSM to
       # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
+      #
+      # ADDED 2026-09-12 — `homelab.keithjackson.dev/egress-uplink`. The gateway node publishes
+      # `<tun0 address>-<ifindex>` here from the OpenVPN up-script on every connect (roles/
+      # vpn_egress_gateway), and that node write is what makes Cilium re-resolve its cached egress
+      # interface. Exporting the label is what lets `VPNEgressSteeringStale` and
+      # `VPNEgressUplinkLabelMissing` (alerts.yaml) watch whether the re-steer mechanism actually
+      # converged — the one thing nothing watched while the dev gateway sat dead for ~80 minutes.
+      # Cost: the label exists on exactly ONE node (the singleton gateway), so this adds a label
+      # DIMENSION to the same 3 `kube_node_labels` series, not new series. Its value changes only on
+      # a tunnel reconnect (a handful of times a day at most), so the churn is negligible — but note
+      # that a value change does retire one series and start another, which is precisely the property
+      # the alerts key on. Sanitized name: `label_homelab_keithjackson_dev_egress_uplink`.
       metricLabelsAllowlist:
-        - "nodes=[homelab.keithjackson.dev/egress-gateway]"
+        - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
       resources:
         requests:
           cpu: 20m