diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index b6b4f64..ba12028 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -2424,6 +2424,148 @@ container dies with `PermissionError` on every run; my older base had `0400` and - The apex flip itself is `migration/dns-apex-inversion`'s job. This change only guarantees the realm follows git once the knob moves; it does not move it. +## 2026-09-12 — The silent-gateway outage now has an alert, and the direct signal turned out not to exist (branch `feat/egress-gateway-staleness-alert`) + +Hand-off item 4 from the section above: *"A silent gateway outage still has no alert… nothing +watches the invariant `egress-uplink` label == `cilium-dbg bpf egress list`'s Egress Ifindex."* +Closed, as far as Prometheus can close it — and the part that cannot be closed is recorded here +rather than papered over with a plausible-looking metric name. + +### The direct signal is not available, verified rather than assumed + +The invariant wants the BPF egress map's contents. They are not in Prometheus, for two independent +reasons, both checked read-only against live prod: + +| check | result | +|---|---| +| `{__name__=~"cilium.*egress.*"}`, `cilium_drop.*`, `hubble_drop.*` | **empty vector** — no `cilium_*`/`hubble_*` series exist at all | +| container ports on the cilium DaemonSet | `health=9879`, `peer-service=4244` — **no metrics port**; only the operator has `operator-prometheus-serve-addr :9963` | +| `kubectl get servicemonitors -A \| grep -iE 'cilium\|hubble'` | none — Hubble/Cilium metrics export is still the open TODO in `platform/cilium/helmrelease.yaml` | + +So `HubbleDeniedFlowSpike` cannot fire today either; that is a separate, pre-existing gap, noted +and left alone. And even with agent metrics turned on, **Cilium OSS publishes no series carrying +egress-map contents** — the map is readable only through `cilium-dbg bpf egress list`, a CLI over +the BPF map. Nothing was invented to fill the hole. + +What node-exporter *does* export is the live tun0 **ifindex as a metric value**: +`node_network_iface_id{device="tun0"}` = `129` on `k3s-prod-3`. Read-only on the live cluster, all +three agree right now: + +``` +node label homelab.keithjackson.dev/egress-uplink = 10.150.192.86-129 +node-exporter node_network_iface_id{device="tun0"} = 129 +bpf map Egress IP 10.150.192.86 / Egress Ifindex 129 (all 16 rows, 4 vpn-class pods) +``` + +(The map on the two NON-gateway agents reads `Egress IP 0.0.0.0 / Ifindex 0` with +`Gateway IP 192.168.2.12` — only the gateway node resolves the interface. Worth knowing before +debugging the wrong agent.) + +### PromQL cannot express the equality test — so the invariant is counted instead + +The label's value is `-`; the live ifindex is a metric VALUE. PromQL has no +label-to-number conversion, so `label == value` is **not expressible**, and there is no +level-triggered form of this comparison. What is expressible is the same invariant counted over a +window: **the published label must change at least as often as tun0 is re-created.** + +```promql +( + changes(max(node_network_iface_id{device="tun0"})[1h:1m]) + > + (count(count_over_time(max by (label_homelab_keithjackson_dev_egress_uplink) ( + kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})[1h:1m])) - 1) +) +and on() +(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0) +``` + +`count(count_over_time(max by (