feat(monitoring): alert when egress steering goes stale, not just when tun0 dies

`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as
vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first
resolution, so after a reconnect the BPF egress map points at a destroyed
interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat
in exactly that state for ~80 minutes with nothing alerting.

Two rules close it:

* VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was
  re-created more times in the last hour than the gateway node's `egress-uplink`
  label changed, i.e. Cilium was never told about a reconnect.
* VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the
  watchdog: a gateway node whose uplink label never reaches kube-state-metrics is
  a cluster where the rule above is inert.

Both gated on a gateway node existing, reusing `VPNTunnelDown`'s
`kube_node_labels` guard — a cluster with no gateway cannot be made to fire them.

The direct label-vs-map comparison is NOT available and nothing was invented to
pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the
cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS
publishes no series carrying egress-map contents at all — the map is readable only
via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the
live ifindex) against a number inside a label value, so the level-triggered
equality test is not expressible; the same invariant is counted over a window
instead. Both limits, and the textfile-collector/black-box-probe routes that would
lift them, are written down in the rule comments and the DECISIONS-LOG.

`egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same
kustomization, because without it the first rule has nothing to compare.

Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr
extracted from `kubectl kustomize platform-config`: empty when healthy; empty when
evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at
a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier
today. Nothing on prod was disturbed to produce that.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/monitoring/alerts.yaml

@@ -328,6 +328,132 @@ spec:
             summary: "Egress VPN tunnel down"
             description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
+        # -----------------------------------------------------------------------------------------
+        # THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06
+        #
+        # `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods
+        # having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0`
+        # exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect
+        # re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points
+        # at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0,
+        # 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and
+        # carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in
+        # exactly that state for ~80 minutes on 2026-09-12 with nothing alerting.
+        #
+        # The datapath now self-heals: `roles/vpn_egress_gateway` publishes
+        # `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node from the
+        # OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case),
+        # and that node write is what makes Cilium re-resolve. These two rules watch whether it
+        # ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage.
+        #
+        # WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's
+        # `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT
+        # observable in Prometheus, verified on prod 2026-09-12 rather than assumed:
+        #   * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`,
+        #     `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and
+        #     there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium
+        #     metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every
+        #     `cilium_*` and `hubble_*` query returns an empty vector today (which also means
+        #     `HubbleDeniedFlowSpike` above cannot fire yet).
+        #   * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map
+        #     CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF
+        #     map, not an exporter. No metric name is invented here to paper over that.
+        # So the observable is the label: the input to the re-steer, compared against the live tun0
+        # identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real
+        # ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and
+        # `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster).
+        #
+        # WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not
+        # firing, the label write failing or never being deployed, the convergence timer being dead —
+        # i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch
+        # "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can,
+        # until either the re-steer script exports the map's ifindex through a node-exporter textfile
+        # collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists.
+        # A probe was considered and rejected for now: there is no blackbox-exporter and no probe
+        # CronJob in this repo, so it means a NEW always-on component whose own failures become a new
+        # alerting surface — worth doing, but not as the thing that closes a hole today.
+        #
+        # WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric
+        # VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no
+        # label-to-number conversion, so the level-triggered equality test is not expressible. What IS
+        # expressible is the same invariant counted over a window: the published label must change at
+        # least as often as tun0 is re-created. `count(count_over_time(max by (<label>) (...)[1h:1m]))`
+        # is the number of DISTINCT label values seen in the last hour (verified live: 1 for a label
+        # that held still, 2 for one that changed value inside the window); the `max by (<label>)`
+        # wrapper is load-bearing — it aggregates away `pod`/`instance`, so a kube-state-metrics
+        # restart is not mistaken for a label change. Both sides reduce to a single unlabelled series,
+        # which is why `and on()` joins cleanly, exactly as `VPNTunnelDown` does.
+        #
+        # Consequence, stated plainly: this is bounded by its window. A missed flap is alerted from
+        # +15m to +1h and then the alert RESOLVES even if egress is still dead. That is a deliberate
+        # trade against the repo's alert-fatigue history (an unbounded critical on the one channel
+        # that must stay trustworthy is worse), and it still turns an 80-minute silent outage into an
+        # email within 15 minutes. The textfile-collector gauge is what would make it level-triggered.
+        - alert: VPNEgressSteeringStale
+          # FIRES when the live tun0 was re-created more times in the last hour than the gateway
+          # node's `egress-uplink` label changed — i.e. Cilium was not told about at least one
+          # tunnel reconnect, so the BPF egress map is pointing at an ifindex that no longer exists
+          # and vpn-class pods have no egress. Convergence is NOT this: the label is rewritten within
+          # seconds of a connect (worst case ~5m, the up-script's 30 x 10s kube-API retry budget on a
+          # cold boot), so `for: 15m` cannot fire on a healthy reconnect.
+          expr: |
+            (
+                changes(max(node_network_iface_id{device="tun0"})[1h:1m])
+              >
+                (
+                  count(
+                    count_over_time(
+                      max by (label_homelab_keithjackson_dev_egress_uplink) (
+                        kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""}
+                      )[1h:1m]
+                    )
+                  ) - 1
+                )
+            )
+            and on()
+            (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "VPN egress steering is stale — vpn-class pods have no egress"
+            description: >-
+              The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
+              Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
+              being dropped, silently, while tun0 still looks healthy. Compare
+              `kubectl get nodes -L homelab.keithjackson.dev/egress-uplink` against
+              `cilium-dbg bpf egress list` on the gateway node; re-applying the CiliumEgressGatewayPolicy
+              (or setting any node label) forces a fresh resolution.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+        - alert: VPNEgressUplinkLabelMissing
+          # The watchdog for the rule above: `VPNEgressSteeringStale` needs the `egress-uplink` label
+          # in kube-state-metrics, and if that label is missing its comparison degrades to an empty
+          # vector — it fails OPEN, alerting nothing. So alert on the precondition itself. A gateway
+          # node with no published uplink means the re-steer mechanism is not in place at all
+          # (roles/vpn_egress_gateway not applied on this node, the up-script never ran, or the
+          # kube-state-metrics label allowlist in kube-prometheus-stack.yaml lost the entry), which is
+          # the configuration that produced the 80-minute dev outage. Warning, not critical: it is a
+          # rollout/config gap to fix in working hours, not a page. `for: 30m` so that the Helm upgrade
+          # that rolls kube-state-metrics — both changes ship in this same kustomization — cannot page
+          # on the way in, and so an Ansible run in progress does not either.
+          expr: |
+            (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+            and on()
+            absent(kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Egress gateway node publishes no egress-uplink label"
+            description: >-
+              A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
+              `homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without it
+              nothing can tell whether Cilium's egress steering followed the last tunnel reconnect, and
+              `VPNEgressSteeringStale` is inert. Check the label on the node, then the
+              `vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
+              metricLabelsAllowlist.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+        # -----------------------------------------------------------------------------------------
         # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
         # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
         # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`

platform/monitoring/kube-prometheus-stack.yaml

@@ -230,8 +230,20 @@ spec:
       # without anyone having to remember to re-enable it later. Scoped to one label on one resource:
       # 3 extra series, no cardinality concern. Label name is sanitized by KSM to
       # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
+      #
+      # ADDED 2026-09-12 — `homelab.keithjackson.dev/egress-uplink`. The gateway node publishes
+      # `<tun0 address>-<ifindex>` here from the OpenVPN up-script on every connect (roles/
+      # vpn_egress_gateway), and that node write is what makes Cilium re-resolve its cached egress
+      # interface. Exporting the label is what lets `VPNEgressSteeringStale` and
+      # `VPNEgressUplinkLabelMissing` (alerts.yaml) watch whether the re-steer mechanism actually
+      # converged — the one thing nothing watched while the dev gateway sat dead for ~80 minutes.
+      # Cost: the label exists on exactly ONE node (the singleton gateway), so this adds a label
+      # DIMENSION to the same 3 `kube_node_labels` series, not new series. Its value changes only on
+      # a tunnel reconnect (a handful of times a day at most), so the churn is negligible — but note
+      # that a value change does retire one series and start another, which is precisely the property
+      # the alerts key on. Sanitized name: `label_homelab_keithjackson_dev_egress_uplink`.
       metricLabelsAllowlist:
-        - "nodes=[homelab.keithjackson.dev/egress-gateway]"
+        - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
       resources:
         requests:
           cpu: 20m