From 16c62346964385fadda0d6c9b837928b735d6464 Mon Sep 17 00:00:00 2001
From: Keith Jackson <keithallenjackson@gmail.com>
Date: Sat, 12 Sep 2026 20:46:56 -0500
Subject: [PATCH] feat(monitoring): alert when egress steering goes stale, not
 just when tun0 dies
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as
vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first
resolution, so after a reconnect the BPF egress map points at a destroyed
interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat
in exactly that state for ~80 minutes with nothing alerting.

Two rules close it:

* VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was
  re-created more times in the last hour than the gateway node's `egress-uplink`
  label changed, i.e. Cilium was never told about a reconnect.
* VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the
  watchdog: a gateway node whose uplink label never reaches kube-state-metrics is
  a cluster where the rule above is inert.

Both gated on a gateway node existing, reusing `VPNTunnelDown`'s
`kube_node_labels` guard — a cluster with no gateway cannot be made to fire them.

The direct label-vs-map comparison is NOT available and nothing was invented to
pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the
cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS
publishes no series carrying egress-map contents at all — the map is readable only
via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the
live ifindex) against a number inside a label value, so the level-triggered
equality test is not expressible; the same invariant is counted over a window
instead. Both limits, and the textfile-collector/black-box-probe routes that would
lift them, are written down in the rule comments and the DECISIONS-LOG.

`egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same
kustomization, because without it the first rule has nothing to compare.

Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr
extracted from `kubectl kustomize platform-config`: empty when healthy; empty when
evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at
a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier
today. Nothing on prod was disturbed to produce that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---
 DECISIONS-LOG.md                              | 142 ++++++++++++++++++
 platform/monitoring/alerts.yaml               | 126 ++++++++++++++++
 .../monitoring/kube-prometheus-stack.yaml     |  14 +-
 platform/vpn-egress/README.md                 |  25 +++
 4 files changed, 306 insertions(+), 1 deletion(-)

diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md
index b20b43f..684a55b 100644
--- a/DECISIONS-LOG.md
+++ b/DECISIONS-LOG.md
@@ -2331,3 +2331,145 @@ vpn-class egress dead until someone noticed.
    > gateway whose tunnel interface is dynamic requires an explicit re-resolution trigger on every
    > interface change. This is a standing operational requirement of the ADR-0015 design, not an
    > implementation detail.*
+
+## 2026-09-12 — The silent-gateway outage now has an alert, and the direct signal turned out not to exist (branch `feat/egress-gateway-staleness-alert`)
+
+Hand-off item 4 from the section above: *"A silent gateway outage still has no alert… nothing
+watches the invariant `egress-uplink` label == `cilium-dbg bpf egress list`'s Egress Ifindex."*
+Closed, as far as Prometheus can close it — and the part that cannot be closed is recorded here
+rather than papered over with a plausible-looking metric name.
+
+### The direct signal is not available, verified rather than assumed
+
+The invariant wants the BPF egress map's contents. They are not in Prometheus, for two independent
+reasons, both checked read-only against live prod:
+
+| check | result |
+|---|---|
+| `{__name__=~"cilium.*egress.*"}`, `cilium_drop.*`, `hubble_drop.*` | **empty vector** — no `cilium_*`/`hubble_*` series exist at all |
+| container ports on the cilium DaemonSet | `health=9879`, `peer-service=4244` — **no metrics port**; only the operator has `operator-prometheus-serve-addr :9963` |
+| `kubectl get servicemonitors -A \| grep -iE 'cilium\|hubble'` | none — Hubble/Cilium metrics export is still the open TODO in `platform/cilium/helmrelease.yaml` |
+
+So `HubbleDeniedFlowSpike` cannot fire today either; that is a separate, pre-existing gap, noted
+and left alone. And even with agent metrics turned on, **Cilium OSS publishes no series carrying
+egress-map contents** — the map is readable only through `cilium-dbg bpf egress list`, a CLI over
+the BPF map. Nothing was invented to fill the hole.
+
+What node-exporter *does* export is the live tun0 **ifindex as a metric value**:
+`node_network_iface_id{device="tun0"}` = `129` on `k3s-prod-3`. Read-only on the live cluster, all
+three agree right now:
+
+```
+node label    homelab.keithjackson.dev/egress-uplink = 10.150.192.86-129
+node-exporter node_network_iface_id{device="tun0"}   = 129
+bpf map       Egress IP 10.150.192.86 / Egress Ifindex 129   (all 16 rows, 4 vpn-class pods)
+```
+
+(The map on the two NON-gateway agents reads `Egress IP 0.0.0.0 / Ifindex 0` with
+`Gateway IP 192.168.2.12` — only the gateway node resolves the interface. Worth knowing before
+debugging the wrong agent.)
+
+### PromQL cannot express the equality test — so the invariant is counted instead
+
+The label's value is `<addr>-<ifindex>`; the live ifindex is a metric VALUE. PromQL has no
+label-to-number conversion, so `label == value` is **not expressible**, and there is no
+level-triggered form of this comparison. What is expressible is the same invariant counted over a
+window: **the published label must change at least as often as tun0 is re-created.**
+
+```promql
+(
+    changes(max(node_network_iface_id{device="tun0"})[1h:1m])
+  >
+    (count(count_over_time(max by (label_homelab_keithjackson_dev_egress_uplink) (
+       kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})[1h:1m])) - 1)
+)
+and on()
+(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+```
+
+`count(count_over_time(max by (<label>) (…)[1h:1m]))` is the number of DISTINCT label values in the
+window. The `max by (<label>)` wrapper is load-bearing: it aggregates away `pod`/`instance` so a
+kube-state-metrics restart is not read as a label change. Both sides reduce to a single unlabelled
+series, which is why `and on()` joins — the same reason `VPNTunnelDown` needs `on()`.
+
+### Proof it evaluates — executed against prod's Prometheus, port-forward + `/api/v1/query`
+
+There is no promtool in this repo, so query execution IS the verification. The **rendered** expr was
+extracted from `kubectl kustomize platform-config` output and run verbatim:
+
+| state | query | result |
+|---|---|---|
+| healthy, now | rendered `VPNEgressSteeringStale` | `[]` — not firing |
+| **no gateway node** (`@ 1789245000`, before prod's stand-up: no tun0 series, no labels) | same expr | `[]` — cannot fire |
+| **stale**, constructed from real data | same expr with the already-exported `egress-gateway` label substituted for `egress-uplink`, window 3h | **`1` — FIRES** |
+| same expr, window 1h | | `[]` |
+
+The stale case is honest and cost prod nothing: prod's tun0 really did go **128 → 129** ~2.5h
+earlier when it moved to PIA `.135`, and the `egress-gateway` label really did *not* follow it
+(`changes(...[3h:1m])` = 1, distinct label values = 1 → `1 > 0`). That is bit-for-bit the shape of
+a label that missed a flap. The healthy counterfactual is the one thing left as reasoning rather
+than a query: over that same 3h the REAL `egress-uplink` label held two values
+(`10.72.192.7-128`, then `10.150.192.86-129`), giving `1 > 1` = false. It could not be queried
+because the label was not exported yet — which is the next point.
+
+Supporting queries, also live: the distinct-value idiom returns **1** for a label that held still
+over the window and **2** for one whose value changed inside it (node 3's `egress-gateway` label
+over 12h, which went `""` → `"true"` at stand-up); the gateway guard returns 1; subquery +
+`changes()` returns 0/1/1/1 over 15m/3h/6h/12h as the flap history requires.
+
+### Behaviour at the edges
+
+- **No gateway node:** three independent reasons it cannot fire — no tun0 series, no uplink label,
+  and the `kube_node_labels` guard `VPNTunnelDown` already established. Verified at a past
+  timestamp when prod genuinely had no gateway. This matters: `VPNTunnelDown` fired permanently on
+  prod for exactly this reason and had to be retrofitted.
+- **Convergence:** the up-script rewrites the label within seconds of a connect, worst case ~5m
+  (its 30 × 10s kube-API retry budget on a cold boot, when the API is not up yet). The condition is
+  true during that gap, so `for: 15m` — comfortably past the 2-minute timer and the 5-minute retry
+  budget — is what keeps a normal reconnect quiet.
+- **Bounded, deliberately:** a missed flap alerts from +15m to +1h, then RESOLVES even if egress is
+  still dead. An unbounded critical on `email-critical` is the worse failure for this repo, and 15
+  minutes still beats 80. A level-triggered version needs the map's ifindex as a real metric — the
+  node-exporter textfile collector route, still open.
+
+### The second rule exists because the first one fails OPEN
+
+`VPNEgressSteeringStale` needs `egress-uplink` in kube-state-metrics. Without it the comparison
+degrades to an empty vector and alerts nothing — so `VPNEgressUplinkLabelMissing` (warning,
+`for: 30m`) watches the precondition: a node labelled `egress-gateway=true` with no `egress-uplink`
+label visible. Live right now it returns **1**, correctly: the KSM allowlist extension is in this
+same branch and not yet deployed. It self-clears on the first reconcile, and `for: 30m` covers the
+kube-state-metrics roll. The rule and the allowlist are in the same Flux kustomization on purpose.
+
+`kube-state-metrics.metricLabelsAllowlist` now reads
+`nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]` — a label
+DIMENSION on the same 3 `kube_node_labels` series, not new series, on exactly one node.
+
+### Rejected, with reasons
+
+- **A black-box egress probe** is the most honest signal — it is the only one that answers "does a
+  vpn-class pod's traffic actually leave via the tunnel?" and the only one that catches "the label
+  was written and Cilium ignored it anyway". Rejected **for now**: there is no blackbox-exporter and
+  no probe CronJob in this repo, so it means a new always-on component (itself vpn-class, itself an
+  alerting surface, itself dependent on a third-party endpoint) — and it could not have been
+  verified on prod today. It remains the right next step.
+- **A node-exporter textfile metric from the re-steer script** (hand-off item 4's own suggestion) is
+  the only route to a true level-triggered label-vs-MAP comparison. It needs a textfile directory
+  mounted into the node-exporter DaemonSet plus an Ansible change applied on a prod node — neither
+  of which this branch could execute or verify. Left open, now with the reason recorded.
+- **`absent(node_network_receive_bytes_total{device="tun0"})`-style presence checks** — that IS
+  `VPNTunnelDown`, and it is exactly the blind spot: green while steering points at a dead ifindex.
+
+### Left for the operator / not settled here
+
+1. **The rules are unproven in the firing sense.** The expressions evaluate against live prod data
+   and the stale branch was exercised with a real flap, but no alert has actually reached
+   Alertmanager or the inbox. `conformance/tests/l3-fault-injection/vpn-killswitch` stops and
+   restarts the tunnel and is the natural place to assert `VPNEgressSteeringStale` stays quiet
+   through a healthy reconnect; the alerting L3 test with the SMTP sink is where a fired-and-emailed
+   proof would come from. Neither was fired at prod unasked.
+2. **`HubbleDeniedFlowSpike` cannot fire** — no cilium/hubble metrics are scraped on either cluster.
+   Pre-existing, orthogonal, and the same TODO in `platform/cilium/helmrelease.yaml` blocks it.
+3. **Window vs. stickiness** is a judgement call, not a measurement: 1h chosen so a missed flap
+   alerts for ~45 minutes. If the operator would rather have a sticky alert that needs manual
+   acknowledgement, that is the textfile-collector design, not a tweak to this one.
diff --git a/platform/monitoring/alerts.yaml b/platform/monitoring/alerts.yaml
index 5260b8e..568648f 100644
--- a/platform/monitoring/alerts.yaml
+++ b/platform/monitoring/alerts.yaml
@@ -328,6 +328,132 @@ spec:
             summary: "Egress VPN tunnel down"
             description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
+        # -----------------------------------------------------------------------------------------
+        # THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06
+        #
+        # `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods
+        # having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0`
+        # exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect
+        # re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points
+        # at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0,
+        # 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and
+        # carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in
+        # exactly that state for ~80 minutes on 2026-09-12 with nothing alerting.
+        #
+        # The datapath now self-heals: `roles/vpn_egress_gateway` publishes
+        # `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node from the
+        # OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case),
+        # and that node write is what makes Cilium re-resolve. These two rules watch whether it
+        # ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage.
+        #
+        # WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's
+        # `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT
+        # observable in Prometheus, verified on prod 2026-09-12 rather than assumed:
+        #   * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`,
+        #     `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and
+        #     there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium
+        #     metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every
+        #     `cilium_*` and `hubble_*` query returns an empty vector today (which also means
+        #     `HubbleDeniedFlowSpike` above cannot fire yet).
+        #   * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map
+        #     CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF
+        #     map, not an exporter. No metric name is invented here to paper over that.
+        # So the observable is the label: the input to the re-steer, compared against the live tun0
+        # identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real
+        # ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and
+        # `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster).
+        #
+        # WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not
+        # firing, the label write failing or never being deployed, the convergence timer being dead —
+        # i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch
+        # "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can,
+        # until either the re-steer script exports the map's ifindex through a node-exporter textfile
+        # collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists.
+        # A probe was considered and rejected for now: there is no blackbox-exporter and no probe
+        # CronJob in this repo, so it means a NEW always-on component whose own failures become a new
+        # alerting surface — worth doing, but not as the thing that closes a hole today.
+        #
+        # WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric
+        # VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no
+        # label-to-number conversion, so the level-triggered equality test is not expressible. What IS
+        # expressible is the same invariant counted over a window: the published label must change at
+        # least as often as tun0 is re-created. `count(count_over_time(max by (<label>) (...)[1h:1m]))`
+        # is the number of DISTINCT label values seen in the last hour (verified live: 1 for a label
+        # that held still, 2 for one that changed value inside the window); the `max by (<label>)`
+        # wrapper is load-bearing — it aggregates away `pod`/`instance`, so a kube-state-metrics
+        # restart is not mistaken for a label change. Both sides reduce to a single unlabelled series,
+        # which is why `and on()` joins cleanly, exactly as `VPNTunnelDown` does.
+        #
+        # Consequence, stated plainly: this is bounded by its window. A missed flap is alerted from
+        # +15m to +1h and then the alert RESOLVES even if egress is still dead. That is a deliberate
+        # trade against the repo's alert-fatigue history (an unbounded critical on the one channel
+        # that must stay trustworthy is worse), and it still turns an 80-minute silent outage into an
+        # email within 15 minutes. The textfile-collector gauge is what would make it level-triggered.
+        - alert: VPNEgressSteeringStale
+          # FIRES when the live tun0 was re-created more times in the last hour than the gateway
+          # node's `egress-uplink` label changed — i.e. Cilium was not told about at least one
+          # tunnel reconnect, so the BPF egress map is pointing at an ifindex that no longer exists
+          # and vpn-class pods have no egress. Convergence is NOT this: the label is rewritten within
+          # seconds of a connect (worst case ~5m, the up-script's 30 x 10s kube-API retry budget on a
+          # cold boot), so `for: 15m` cannot fire on a healthy reconnect.
+          expr: |
+            (
+                changes(max(node_network_iface_id{device="tun0"})[1h:1m])
+              >
+                (
+                  count(
+                    count_over_time(
+                      max by (label_homelab_keithjackson_dev_egress_uplink) (
+                        kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""}
+                      )[1h:1m]
+                    )
+                  ) - 1
+                )
+            )
+            and on()
+            (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "VPN egress steering is stale — vpn-class pods have no egress"
+            description: >-
+              The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
+              Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
+              being dropped, silently, while tun0 still looks healthy. Compare
+              `kubectl get nodes -L homelab.keithjackson.dev/egress-uplink` against
+              `cilium-dbg bpf egress list` on the gateway node; re-applying the CiliumEgressGatewayPolicy
+              (or setting any node label) forces a fresh resolution.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+        - alert: VPNEgressUplinkLabelMissing
+          # The watchdog for the rule above: `VPNEgressSteeringStale` needs the `egress-uplink` label
+          # in kube-state-metrics, and if that label is missing its comparison degrades to an empty
+          # vector — it fails OPEN, alerting nothing. So alert on the precondition itself. A gateway
+          # node with no published uplink means the re-steer mechanism is not in place at all
+          # (roles/vpn_egress_gateway not applied on this node, the up-script never ran, or the
+          # kube-state-metrics label allowlist in kube-prometheus-stack.yaml lost the entry), which is
+          # the configuration that produced the 80-minute dev outage. Warning, not critical: it is a
+          # rollout/config gap to fix in working hours, not a page. `for: 30m` so that the Helm upgrade
+          # that rolls kube-state-metrics — both changes ship in this same kustomization — cannot page
+          # on the way in, and so an Ansible run in progress does not either.
+          expr: |
+            (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+            and on()
+            absent(kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Egress gateway node publishes no egress-uplink label"
+            description: >-
+              A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
+              `homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without it
+              nothing can tell whether Cilium's egress steering followed the last tunnel reconnect, and
+              `VPNEgressSteeringStale` is inert. Check the label on the node, then the
+              `vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
+              metricLabelsAllowlist.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+        # -----------------------------------------------------------------------------------------
         # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
         # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
         # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
diff --git a/platform/monitoring/kube-prometheus-stack.yaml b/platform/monitoring/kube-prometheus-stack.yaml
index bd72b3b..774f8b9 100644
--- a/platform/monitoring/kube-prometheus-stack.yaml
+++ b/platform/monitoring/kube-prometheus-stack.yaml
@@ -230,8 +230,20 @@ spec:
       # without anyone having to remember to re-enable it later. Scoped to one label on one resource:
       # 3 extra series, no cardinality concern. Label name is sanitized by KSM to
       # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
+      #
+      # ADDED 2026-09-12 — `homelab.keithjackson.dev/egress-uplink`. The gateway node publishes
+      # `<tun0 address>-<ifindex>` here from the OpenVPN up-script on every connect (roles/
+      # vpn_egress_gateway), and that node write is what makes Cilium re-resolve its cached egress
+      # interface. Exporting the label is what lets `VPNEgressSteeringStale` and
+      # `VPNEgressUplinkLabelMissing` (alerts.yaml) watch whether the re-steer mechanism actually
+      # converged — the one thing nothing watched while the dev gateway sat dead for ~80 minutes.
+      # Cost: the label exists on exactly ONE node (the singleton gateway), so this adds a label
+      # DIMENSION to the same 3 `kube_node_labels` series, not new series. Its value changes only on
+      # a tunnel reconnect (a handful of times a day at most), so the churn is negligible — but note
+      # that a value change does retire one series and start another, which is precisely the property
+      # the alerts key on. Sanitized name: `label_homelab_keithjackson_dev_egress_uplink`.
       metricLabelsAllowlist:
-        - "nodes=[homelab.keithjackson.dev/egress-gateway]"
+        - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
       resources:
         requests:
           cpu: 20m
diff --git a/platform/vpn-egress/README.md b/platform/vpn-egress/README.md
index 6883332..b0dc022 100644
--- a/platform/vpn-egress/README.md
+++ b/platform/vpn-egress/README.md
@@ -157,6 +157,31 @@ kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
 If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
 gateway node is the next stop.
 
+### And now something watches it (2026-09-12)
+
+The self-heal closed the datapath hole; it did not close the monitoring one, and a mechanism that
+silently stops working is the same outage again. Two rules in `platform/monitoring/alerts.yaml`:
+
+- **`VPNEgressSteeringStale`** (critical, `for: 15m`) — fires when the live tun0 was re-created
+  more times in the last hour than the `egress-uplink` label changed, i.e. Cilium was never told
+  about a reconnect. This is the case `VPNTunnelDown` cannot see: tun0 present, bytes flowing,
+  steering pointed at a destroyed ifindex.
+- **`VPNEgressUplinkLabelMissing`** (warning, `for: 30m`) — fires when a node carries
+  `egress-gateway=true` but no `egress-uplink` label reaches kube-state-metrics, which is the state
+  in which the rule above would be inert. The watchdog for the watchdog.
+
+Both are gated on a gateway node existing (the same `kube_node_labels` guard `VPNTunnelDown` uses),
+so a cluster with no gateway cannot be made to fire them. `egress-uplink` had to be added to
+`kube-state-metrics.metricLabelsAllowlist` in `platform/monitoring/kube-prometheus-stack.yaml` for
+the label to exist as a metric at all — the two changes must ship together.
+
+**What still is not watched:** "the label was written and Cilium still ignored it." Cilium OSS
+exposes no metric carrying the egress map's contents (the map is readable only through
+`cilium-dbg bpf egress list`), and the cilium-agent is not scraped on either cluster in the first
+place — no `cilium_*` or `hubble_*` series exist. Closing that last gap means either exporting the
+map's ifindex from the re-steer script through a node-exporter textfile collector, or a black-box
+probe from a vpn-class pod. Neither exists yet.
+
 ## Resilience — there is NO gateway failover (spec correction, escalated)
 
 **ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)