merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate

Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.

WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:

  * alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
    which did not exist. An alert firing now would hand the operator a 404.
  * alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
    in it must be tested or a "tested" file quietly grows untested rules.
  * alert-metrics-have-scrape-targets — see the guard bug below.

WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.

Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.

runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.

platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.

Two fixes the branch needed that were defects in the checks themselves:

1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
   `nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
   contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
   which these alerts need) therefore FAILED the check while the property it guards
   was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
   guard that breaks when you correctly extend the thing it guards teaches people to
   edit the guard, which is how it stops guarding.

2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
   "asserted on" is a bare word match over the test file — my header COMMENT
   mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
   register entry and claim coverage that did not exist, the fixtures now assert
   VPNTunnelDown QUIET in all four cases. That required making them faithful: they
   had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
   would have been firing throughout unnoticed, because promtool only checks the
   alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
   these rules exist for.

NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.

Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.

Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.

Co-Authored-By: Claude Opus 5 <[email protected]>

DECISIONS-LOG.md

@@ -2424,6 +2424,148 @@ container dies with `PermissionError` on every run; my older base had `0400` and
 - The apex flip itself is `migration/dns-apex-inversion`'s job. This change only guarantees the realm
   follows git once the knob moves; it does not move it.
 
+## 2026-09-12 — The silent-gateway outage now has an alert, and the direct signal turned out not to exist (branch `feat/egress-gateway-staleness-alert`)
+
+Hand-off item 4 from the section above: *"A silent gateway outage still has no alert… nothing
+watches the invariant `egress-uplink` label == `cilium-dbg bpf egress list`'s Egress Ifindex."*
+Closed, as far as Prometheus can close it — and the part that cannot be closed is recorded here
+rather than papered over with a plausible-looking metric name.
+
+### The direct signal is not available, verified rather than assumed
+
+The invariant wants the BPF egress map's contents. They are not in Prometheus, for two independent
+reasons, both checked read-only against live prod:
+
+| check | result |
+|---|---|
+| `{__name__=~"cilium.*egress.*"}`, `cilium_drop.*`, `hubble_drop.*` | **empty vector** — no `cilium_*`/`hubble_*` series exist at all |
+| container ports on the cilium DaemonSet | `health=9879`, `peer-service=4244` — **no metrics port**; only the operator has `operator-prometheus-serve-addr :9963` |
+| `kubectl get servicemonitors -A \| grep -iE 'cilium\|hubble'` | none — Hubble/Cilium metrics export is still the open TODO in `platform/cilium/helmrelease.yaml` |
+
+So `HubbleDeniedFlowSpike` cannot fire today either; that is a separate, pre-existing gap, noted
+and left alone. And even with agent metrics turned on, **Cilium OSS publishes no series carrying
+egress-map contents** — the map is readable only through `cilium-dbg bpf egress list`, a CLI over
+the BPF map. Nothing was invented to fill the hole.
+
+What node-exporter *does* export is the live tun0 **ifindex as a metric value**:
+`node_network_iface_id{device="tun0"}` = `129` on `k3s-prod-3`. Read-only on the live cluster, all
+three agree right now:
+
+```
+node label    homelab.keithjackson.dev/egress-uplink = 10.150.192.86-129
+node-exporter node_network_iface_id{device="tun0"}   = 129
+bpf map       Egress IP 10.150.192.86 / Egress Ifindex 129   (all 16 rows, 4 vpn-class pods)
+```
+
+(The map on the two NON-gateway agents reads `Egress IP 0.0.0.0 / Ifindex 0` with
+`Gateway IP 192.168.2.12` — only the gateway node resolves the interface. Worth knowing before
+debugging the wrong agent.)
+
+### PromQL cannot express the equality test — so the invariant is counted instead
+
+The label's value is `<addr>-<ifindex>`; the live ifindex is a metric VALUE. PromQL has no
+label-to-number conversion, so `label == value` is **not expressible**, and there is no
+level-triggered form of this comparison. What is expressible is the same invariant counted over a
+window: **the published label must change at least as often as tun0 is re-created.**
+
+```promql
+(
+    changes(max(node_network_iface_id{device="tun0"})[1h:1m])
+  >
+    (count(count_over_time(max by (label_homelab_keithjackson_dev_egress_uplink) (
+       kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})[1h:1m])) - 1)
+)
+and on()
+(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+```
+
+`count(count_over_time(max by (<label>) (…)[1h:1m]))` is the number of DISTINCT label values in the
+window. The `max by (<label>)` wrapper is load-bearing: it aggregates away `pod`/`instance` so a
+kube-state-metrics restart is not read as a label change. Both sides reduce to a single unlabelled
+series, which is why `and on()` joins — the same reason `VPNTunnelDown` needs `on()`.
+
+### Proof it evaluates — executed against prod's Prometheus, port-forward + `/api/v1/query`
+
+There is no promtool in this repo, so query execution IS the verification. The **rendered** expr was
+extracted from `kubectl kustomize platform-config` output and run verbatim:
+
+| state | query | result |
+|---|---|---|
+| healthy, now | rendered `VPNEgressSteeringStale` | `[]` — not firing |
+| **no gateway node** (`@ 1789245000`, before prod's stand-up: no tun0 series, no labels) | same expr | `[]` — cannot fire |
+| **stale**, constructed from real data | same expr with the already-exported `egress-gateway` label substituted for `egress-uplink`, window 3h | **`1` — FIRES** |
+| same expr, window 1h | | `[]` |
+
+The stale case is honest and cost prod nothing: prod's tun0 really did go **128 → 129** ~2.5h
+earlier when it moved to PIA `.135`, and the `egress-gateway` label really did *not* follow it
+(`changes(...[3h:1m])` = 1, distinct label values = 1 → `1 > 0`). That is bit-for-bit the shape of
+a label that missed a flap. The healthy counterfactual is the one thing left as reasoning rather
+than a query: over that same 3h the REAL `egress-uplink` label held two values
+(`10.72.192.7-128`, then `10.150.192.86-129`), giving `1 > 1` = false. It could not be queried
+because the label was not exported yet — which is the next point.
+
+Supporting queries, also live: the distinct-value idiom returns **1** for a label that held still
+over the window and **2** for one whose value changed inside it (node 3's `egress-gateway` label
+over 12h, which went `""` → `"true"` at stand-up); the gateway guard returns 1; subquery +
+`changes()` returns 0/1/1/1 over 15m/3h/6h/12h as the flap history requires.
+
+### Behaviour at the edges
+
+- **No gateway node:** three independent reasons it cannot fire — no tun0 series, no uplink label,
+  and the `kube_node_labels` guard `VPNTunnelDown` already established. Verified at a past
+  timestamp when prod genuinely had no gateway. This matters: `VPNTunnelDown` fired permanently on
+  prod for exactly this reason and had to be retrofitted.
+- **Convergence:** the up-script rewrites the label within seconds of a connect, worst case ~5m
+  (its 30 × 10s kube-API retry budget on a cold boot, when the API is not up yet). The condition is
+  true during that gap, so `for: 15m` — comfortably past the 2-minute timer and the 5-minute retry
+  budget — is what keeps a normal reconnect quiet.
+- **Bounded, deliberately:** a missed flap alerts from +15m to +1h, then RESOLVES even if egress is
+  still dead. An unbounded critical on `email-critical` is the worse failure for this repo, and 15
+  minutes still beats 80. A level-triggered version needs the map's ifindex as a real metric — the
+  node-exporter textfile collector route, still open.
+
+### The second rule exists because the first one fails OPEN
+
+`VPNEgressSteeringStale` needs `egress-uplink` in kube-state-metrics. Without it the comparison
+degrades to an empty vector and alerts nothing — so `VPNEgressUplinkLabelMissing` (warning,
+`for: 30m`) watches the precondition: a node labelled `egress-gateway=true` with no `egress-uplink`
+label visible. Live right now it returns **1**, correctly: the KSM allowlist extension is in this
+same branch and not yet deployed. It self-clears on the first reconcile, and `for: 30m` covers the
+kube-state-metrics roll. The rule and the allowlist are in the same Flux kustomization on purpose.
+
+`kube-state-metrics.metricLabelsAllowlist` now reads
+`nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]` — a label
+DIMENSION on the same 3 `kube_node_labels` series, not new series, on exactly one node.
+
+### Rejected, with reasons
+
+- **A black-box egress probe** is the most honest signal — it is the only one that answers "does a
+  vpn-class pod's traffic actually leave via the tunnel?" and the only one that catches "the label
+  was written and Cilium ignored it anyway". Rejected **for now**: there is no blackbox-exporter and
+  no probe CronJob in this repo, so it means a new always-on component (itself vpn-class, itself an
+  alerting surface, itself dependent on a third-party endpoint) — and it could not have been
+  verified on prod today. It remains the right next step.
+- **A node-exporter textfile metric from the re-steer script** (hand-off item 4's own suggestion) is
+  the only route to a true level-triggered label-vs-MAP comparison. It needs a textfile directory
+  mounted into the node-exporter DaemonSet plus an Ansible change applied on a prod node — neither
+  of which this branch could execute or verify. Left open, now with the reason recorded.
+- **`absent(node_network_receive_bytes_total{device="tun0"})`-style presence checks** — that IS
+  `VPNTunnelDown`, and it is exactly the blind spot: green while steering points at a dead ifindex.
+
+### Left for the operator / not settled here
+
+1. **The rules are unproven in the firing sense.** The expressions evaluate against live prod data
+   and the stale branch was exercised with a real flap, but no alert has actually reached
+   Alertmanager or the inbox. `conformance/tests/l3-fault-injection/vpn-killswitch` stops and
+   restarts the tunnel and is the natural place to assert `VPNEgressSteeringStale` stays quiet
+   through a healthy reconnect; the alerting L3 test with the SMTP sink is where a fired-and-emailed
+   proof would come from. Neither was fired at prod unasked.
+2. **`HubbleDeniedFlowSpike` cannot fire** — no cilium/hubble metrics are scraped on either cluster.
+   Pre-existing, orthogonal, and the same TODO in `platform/cilium/helmrelease.yaml` blocks it.
+3. **Window vs. stickiness** is a judgement call, not a measurement: 1h chosen so a missed flap
+   alerts for ~45 minutes. If the operator would rather have a sticky alert that needs manual
+   acknowledgement, that is the textfile-collector design, not a tweak to this one.
+
 ## 2026-09-13 — **PROD IS THE APEX.** Dev decommissioned; and a GitOps deadlock found the hard way
 
 Prod serves `home.keithjackson.dev`. Dev's three VMs (Proxmox 101/102/105) are stopped with