feat(monitoring): alert when egress steering goes stale, not just when tun0 dies

`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as
vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first
resolution, so after a reconnect the BPF egress map points at a destroyed
interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat
in exactly that state for ~80 minutes with nothing alerting.

Two rules close it:

* VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was
  re-created more times in the last hour than the gateway node's `egress-uplink`
  label changed, i.e. Cilium was never told about a reconnect.
* VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the
  watchdog: a gateway node whose uplink label never reaches kube-state-metrics is
  a cluster where the rule above is inert.

Both gated on a gateway node existing, reusing `VPNTunnelDown`'s
`kube_node_labels` guard — a cluster with no gateway cannot be made to fire them.

The direct label-vs-map comparison is NOT available and nothing was invented to
pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the
cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS
publishes no series carrying egress-map contents at all — the map is readable only
via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the
live ifindex) against a number inside a label value, so the level-triggered
equality test is not expressible; the same invariant is counted over a window
instead. Both limits, and the textfile-collector/black-box-probe routes that would
lift them, are written down in the rule comments and the DECISIONS-LOG.

`egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same
kustomization, because without it the first rule has nothing to compare.

Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr
extracted from `kubectl kustomize platform-config`: empty when healthy; empty when
evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at
a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier
today. Nothing on prod was disturbed to produce that.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -2331,3 +2331,145 @@ vpn-class egress dead until someone noticed.
    > gateway whose tunnel interface is dynamic requires an explicit re-resolution trigger on every
    > interface change. This is a standing operational requirement of the ADR-0015 design, not an
    > implementation detail.*
+
+## 2026-09-12 — The silent-gateway outage now has an alert, and the direct signal turned out not to exist (branch `feat/egress-gateway-staleness-alert`)
+
+Hand-off item 4 from the section above: *"A silent gateway outage still has no alert… nothing
+watches the invariant `egress-uplink` label == `cilium-dbg bpf egress list`'s Egress Ifindex."*
+Closed, as far as Prometheus can close it — and the part that cannot be closed is recorded here
+rather than papered over with a plausible-looking metric name.
+
+### The direct signal is not available, verified rather than assumed
+
+The invariant wants the BPF egress map's contents. They are not in Prometheus, for two independent
+reasons, both checked read-only against live prod:
+
+| check | result |
+|---|---|
+| `{__name__=~"cilium.*egress.*"}`, `cilium_drop.*`, `hubble_drop.*` | **empty vector** — no `cilium_*`/`hubble_*` series exist at all |
+| container ports on the cilium DaemonSet | `health=9879`, `peer-service=4244` — **no metrics port**; only the operator has `operator-prometheus-serve-addr :9963` |
+| `kubectl get servicemonitors -A \| grep -iE 'cilium\|hubble'` | none — Hubble/Cilium metrics export is still the open TODO in `platform/cilium/helmrelease.yaml` |
+
+So `HubbleDeniedFlowSpike` cannot fire today either; that is a separate, pre-existing gap, noted
+and left alone. And even with agent metrics turned on, **Cilium OSS publishes no series carrying
+egress-map contents** — the map is readable only through `cilium-dbg bpf egress list`, a CLI over
+the BPF map. Nothing was invented to fill the hole.
+
+What node-exporter *does* export is the live tun0 **ifindex as a metric value**:
+`node_network_iface_id{device="tun0"}` = `129` on `k3s-prod-3`. Read-only on the live cluster, all
+three agree right now:
+
+```
+node label    homelab.keithjackson.dev/egress-uplink = 10.150.192.86-129
+node-exporter node_network_iface_id{device="tun0"}   = 129
+bpf map       Egress IP 10.150.192.86 / Egress Ifindex 129   (all 16 rows, 4 vpn-class pods)
+```
+
+(The map on the two NON-gateway agents reads `Egress IP 0.0.0.0 / Ifindex 0` with
+`Gateway IP 192.168.2.12` — only the gateway node resolves the interface. Worth knowing before
+debugging the wrong agent.)
+
+### PromQL cannot express the equality test — so the invariant is counted instead
+
+The label's value is `<addr>-<ifindex>`; the live ifindex is a metric VALUE. PromQL has no
+label-to-number conversion, so `label == value` is **not expressible**, and there is no
+level-triggered form of this comparison. What is expressible is the same invariant counted over a
+window: **the published label must change at least as often as tun0 is re-created.**
+
+```promql
+(
+    changes(max(node_network_iface_id{device="tun0"})[1h:1m])
+  >
+    (count(count_over_time(max by (label_homelab_keithjackson_dev_egress_uplink) (
+       kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})[1h:1m])) - 1)
+)
+and on()
+(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+```
+
+`count(count_over_time(max by (<label>) (…)[1h:1m]))` is the number of DISTINCT label values in the
+window. The `max by (<label>)` wrapper is load-bearing: it aggregates away `pod`/`instance` so a
+kube-state-metrics restart is not read as a label change. Both sides reduce to a single unlabelled
+series, which is why `and on()` joins — the same reason `VPNTunnelDown` needs `on()`.
+
+### Proof it evaluates — executed against prod's Prometheus, port-forward + `/api/v1/query`
+
+There is no promtool in this repo, so query execution IS the verification. The **rendered** expr was
+extracted from `kubectl kustomize platform-config` output and run verbatim:
+
+| state | query | result |
+|---|---|---|
+| healthy, now | rendered `VPNEgressSteeringStale` | `[]` — not firing |
+| **no gateway node** (`@ 1789245000`, before prod's stand-up: no tun0 series, no labels) | same expr | `[]` — cannot fire |
+| **stale**, constructed from real data | same expr with the already-exported `egress-gateway` label substituted for `egress-uplink`, window 3h | **`1` — FIRES** |
+| same expr, window 1h | | `[]` |
+
+The stale case is honest and cost prod nothing: prod's tun0 really did go **128 → 129** ~2.5h
+earlier when it moved to PIA `.135`, and the `egress-gateway` label really did *not* follow it
+(`changes(...[3h:1m])` = 1, distinct label values = 1 → `1 > 0`). That is bit-for-bit the shape of
+a label that missed a flap. The healthy counterfactual is the one thing left as reasoning rather
+than a query: over that same 3h the REAL `egress-uplink` label held two values
+(`10.72.192.7-128`, then `10.150.192.86-129`), giving `1 > 1` = false. It could not be queried
+because the label was not exported yet — which is the next point.
+
+Supporting queries, also live: the distinct-value idiom returns **1** for a label that held still
+over the window and **2** for one whose value changed inside it (node 3's `egress-gateway` label
+over 12h, which went `""` → `"true"` at stand-up); the gateway guard returns 1; subquery +
+`changes()` returns 0/1/1/1 over 15m/3h/6h/12h as the flap history requires.
+
+### Behaviour at the edges
+
+- **No gateway node:** three independent reasons it cannot fire — no tun0 series, no uplink label,
+  and the `kube_node_labels` guard `VPNTunnelDown` already established. Verified at a past
+  timestamp when prod genuinely had no gateway. This matters: `VPNTunnelDown` fired permanently on
+  prod for exactly this reason and had to be retrofitted.
+- **Convergence:** the up-script rewrites the label within seconds of a connect, worst case ~5m
+  (its 30 × 10s kube-API retry budget on a cold boot, when the API is not up yet). The condition is
+  true during that gap, so `for: 15m` — comfortably past the 2-minute timer and the 5-minute retry
+  budget — is what keeps a normal reconnect quiet.
+- **Bounded, deliberately:** a missed flap alerts from +15m to +1h, then RESOLVES even if egress is
+  still dead. An unbounded critical on `email-critical` is the worse failure for this repo, and 15
+  minutes still beats 80. A level-triggered version needs the map's ifindex as a real metric — the
+  node-exporter textfile collector route, still open.
+
+### The second rule exists because the first one fails OPEN
+
+`VPNEgressSteeringStale` needs `egress-uplink` in kube-state-metrics. Without it the comparison
+degrades to an empty vector and alerts nothing — so `VPNEgressUplinkLabelMissing` (warning,
+`for: 30m`) watches the precondition: a node labelled `egress-gateway=true` with no `egress-uplink`
+label visible. Live right now it returns **1**, correctly: the KSM allowlist extension is in this
+same branch and not yet deployed. It self-clears on the first reconcile, and `for: 30m` covers the
+kube-state-metrics roll. The rule and the allowlist are in the same Flux kustomization on purpose.
+
+`kube-state-metrics.metricLabelsAllowlist` now reads
+`nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]` — a label
+DIMENSION on the same 3 `kube_node_labels` series, not new series, on exactly one node.
+
+### Rejected, with reasons
+
+- **A black-box egress probe** is the most honest signal — it is the only one that answers "does a
+  vpn-class pod's traffic actually leave via the tunnel?" and the only one that catches "the label
+  was written and Cilium ignored it anyway". Rejected **for now**: there is no blackbox-exporter and
+  no probe CronJob in this repo, so it means a new always-on component (itself vpn-class, itself an
+  alerting surface, itself dependent on a third-party endpoint) — and it could not have been
+  verified on prod today. It remains the right next step.
+- **A node-exporter textfile metric from the re-steer script** (hand-off item 4's own suggestion) is
+  the only route to a true level-triggered label-vs-MAP comparison. It needs a textfile directory
+  mounted into the node-exporter DaemonSet plus an Ansible change applied on a prod node — neither
+  of which this branch could execute or verify. Left open, now with the reason recorded.
+- **`absent(node_network_receive_bytes_total{device="tun0"})`-style presence checks** — that IS
+  `VPNTunnelDown`, and it is exactly the blind spot: green while steering points at a dead ifindex.
+
+### Left for the operator / not settled here
+
+1. **The rules are unproven in the firing sense.** The expressions evaluate against live prod data
+   and the stale branch was exercised with a real flap, but no alert has actually reached
+   Alertmanager or the inbox. `conformance/tests/l3-fault-injection/vpn-killswitch` stops and
+   restarts the tunnel and is the natural place to assert `VPNEgressSteeringStale` stays quiet
+   through a healthy reconnect; the alerting L3 test with the SMTP sink is where a fired-and-emailed
+   proof would come from. Neither was fired at prod unasked.
+2. **`HubbleDeniedFlowSpike` cannot fire** — no cilium/hubble metrics are scraped on either cluster.
+   Pre-existing, orthogonal, and the same TODO in `platform/cilium/helmrelease.yaml` blocks it.
+3. **Window vs. stickiness** is a judgement call, not a measurement: 1h chosen so a missed flap
+   alerts for ~45 minutes. If the operator would rather have a sticky alert that needs manual
+   acknowledgement, that is the textfile-collector design, not a tweak to this one.