feat(monitoring): alert when egress steering goes stale, not just when tun0 dies
`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first resolution, so after a reconnect the BPF egress map points at a destroyed interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat in exactly that state for ~80 minutes with nothing alerting. Two rules close it: * VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was re-created more times in the last hour than the gateway node's `egress-uplink` label changed, i.e. Cilium was never told about a reconnect. * VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the watchdog: a gateway node whose uplink label never reaches kube-state-metrics is a cluster where the rule above is inert. Both gated on a gateway node existing, reusing `VPNTunnelDown`'s `kube_node_labels` guard — a cluster with no gateway cannot be made to fire them. The direct label-vs-map comparison is NOT available and nothing was invented to pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS publishes no series carrying egress-map contents at all — the map is readable only via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the live ifindex) against a number inside a label value, so the level-triggered equality test is not expressible; the same invariant is counted over a window instead. Both limits, and the textfile-collector/black-box-probe routes that would lift them, are written down in the rule comments and the DECISIONS-LOG. `egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same kustomization, because without it the first rule has nothing to compare. Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr extracted from `kubectl kustomize platform-config`: empty when healthy; empty when evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier today. Nothing on prod was disturbed to produce that. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
19fc8bcb27
commit
16c6234696
4 files changed
+306
-1
No files matched your search
@@ -2331,3 +2331,145 @@ vpn-class egress dead until someone noticed.
|
||||
> gateway whose tunnel interface is dynamic requires an explicit re-resolution trigger on every
|
||||
> interface change. This is a standing operational requirement of the ADR-0015 design, not an
|
||||
> implementation detail.*
|
||||
|
||||
## 2026-09-12 — The silent-gateway outage now has an alert, and the direct signal turned out not to exist (branch `feat/egress-gateway-staleness-alert`)
|
||||
|
||||
Hand-off item 4 from the section above: *"A silent gateway outage still has no alert… nothing
|
||||
watches the invariant `egress-uplink` label == `cilium-dbg bpf egress list`'s Egress Ifindex."*
|
||||
Closed, as far as Prometheus can close it — and the part that cannot be closed is recorded here
|
||||
rather than papered over with a plausible-looking metric name.
|
||||
|
||||
### The direct signal is not available, verified rather than assumed
|
||||
|
||||
The invariant wants the BPF egress map's contents. They are not in Prometheus, for two independent
|
||||
reasons, both checked read-only against live prod:
|
||||
|
||||
| check | result |
|
||||
|---|---|
|
||||
| `{__name__=~"cilium.*egress.*"}`, `cilium_drop.*`, `hubble_drop.*` | **empty vector** — no `cilium_*`/`hubble_*` series exist at all |
|
||||
| container ports on the cilium DaemonSet | `health=9879`, `peer-service=4244` — **no metrics port**; only the operator has `operator-prometheus-serve-addr :9963` |
|
||||
| `kubectl get servicemonitors -A \| grep -iE 'cilium\|hubble'` | none — Hubble/Cilium metrics export is still the open TODO in `platform/cilium/helmrelease.yaml` |
|
||||
|
||||
So `HubbleDeniedFlowSpike` cannot fire today either; that is a separate, pre-existing gap, noted
|
||||
and left alone. And even with agent metrics turned on, **Cilium OSS publishes no series carrying
|
||||
egress-map contents** — the map is readable only through `cilium-dbg bpf egress list`, a CLI over
|
||||
the BPF map. Nothing was invented to fill the hole.
|
||||
|
||||
What node-exporter *does* export is the live tun0 **ifindex as a metric value**:
|
||||
`node_network_iface_id{device="tun0"}` = `129` on `k3s-prod-3`. Read-only on the live cluster, all
|
||||
three agree right now:
|
||||
|
||||
```
|
||||
node label homelab.keithjackson.dev/egress-uplink = 10.150.192.86-129
|
||||
node-exporter node_network_iface_id{device="tun0"} = 129
|
||||
bpf map Egress IP 10.150.192.86 / Egress Ifindex 129 (all 16 rows, 4 vpn-class pods)
|
||||
```
|
||||
|
||||
(The map on the two NON-gateway agents reads `Egress IP 0.0.0.0 / Ifindex 0` with
|
||||
`Gateway IP 192.168.2.12` — only the gateway node resolves the interface. Worth knowing before
|
||||
debugging the wrong agent.)
|
||||
|
||||
### PromQL cannot express the equality test — so the invariant is counted instead
|
||||
|
||||
The label's value is `<addr>-<ifindex>`; the live ifindex is a metric VALUE. PromQL has no
|
||||
label-to-number conversion, so `label == value` is **not expressible**, and there is no
|
||||
level-triggered form of this comparison. What is expressible is the same invariant counted over a
|
||||
window: **the published label must change at least as often as tun0 is re-created.**
|
||||
|
||||
```promql
|
||||
(
|
||||
changes(max(node_network_iface_id{device="tun0"})[1h:1m])
|
||||
>
|
||||
(count(count_over_time(max by (label_homelab_keithjackson_dev_egress_uplink) (
|
||||
kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})[1h:1m])) - 1)
|
||||
)
|
||||
and on()
|
||||
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
||||
```
|
||||
|
||||
`count(count_over_time(max by (<label>) (…)[1h:1m]))` is the number of DISTINCT label values in the
|
||||
window. The `max by (<label>)` wrapper is load-bearing: it aggregates away `pod`/`instance` so a
|
||||
kube-state-metrics restart is not read as a label change. Both sides reduce to a single unlabelled
|
||||
series, which is why `and on()` joins — the same reason `VPNTunnelDown` needs `on()`.
|
||||
|
||||
### Proof it evaluates — executed against prod's Prometheus, port-forward + `/api/v1/query`
|
||||
|
||||
There is no promtool in this repo, so query execution IS the verification. The **rendered** expr was
|
||||
extracted from `kubectl kustomize platform-config` output and run verbatim:
|
||||
|
||||
| state | query | result |
|
||||
|---|---|---|
|
||||
| healthy, now | rendered `VPNEgressSteeringStale` | `[]` — not firing |
|
||||
| **no gateway node** (`@ 1789245000`, before prod's stand-up: no tun0 series, no labels) | same expr | `[]` — cannot fire |
|
||||
| **stale**, constructed from real data | same expr with the already-exported `egress-gateway` label substituted for `egress-uplink`, window 3h | **`1` — FIRES** |
|
||||
| same expr, window 1h | | `[]` |
|
||||
|
||||
The stale case is honest and cost prod nothing: prod's tun0 really did go **128 → 129** ~2.5h
|
||||
earlier when it moved to PIA `.135`, and the `egress-gateway` label really did *not* follow it
|
||||
(`changes(...[3h:1m])` = 1, distinct label values = 1 → `1 > 0`). That is bit-for-bit the shape of
|
||||
a label that missed a flap. The healthy counterfactual is the one thing left as reasoning rather
|
||||
than a query: over that same 3h the REAL `egress-uplink` label held two values
|
||||
(`10.72.192.7-128`, then `10.150.192.86-129`), giving `1 > 1` = false. It could not be queried
|
||||
because the label was not exported yet — which is the next point.
|
||||
|
||||
Supporting queries, also live: the distinct-value idiom returns **1** for a label that held still
|
||||
over the window and **2** for one whose value changed inside it (node 3's `egress-gateway` label
|
||||
over 12h, which went `""` → `"true"` at stand-up); the gateway guard returns 1; subquery +
|
||||
`changes()` returns 0/1/1/1 over 15m/3h/6h/12h as the flap history requires.
|
||||
|
||||
### Behaviour at the edges
|
||||
|
||||
- **No gateway node:** three independent reasons it cannot fire — no tun0 series, no uplink label,
|
||||
and the `kube_node_labels` guard `VPNTunnelDown` already established. Verified at a past
|
||||
timestamp when prod genuinely had no gateway. This matters: `VPNTunnelDown` fired permanently on
|
||||
prod for exactly this reason and had to be retrofitted.
|
||||
- **Convergence:** the up-script rewrites the label within seconds of a connect, worst case ~5m
|
||||
(its 30 × 10s kube-API retry budget on a cold boot, when the API is not up yet). The condition is
|
||||
true during that gap, so `for: 15m` — comfortably past the 2-minute timer and the 5-minute retry
|
||||
budget — is what keeps a normal reconnect quiet.
|
||||
- **Bounded, deliberately:** a missed flap alerts from +15m to +1h, then RESOLVES even if egress is
|
||||
still dead. An unbounded critical on `email-critical` is the worse failure for this repo, and 15
|
||||
minutes still beats 80. A level-triggered version needs the map's ifindex as a real metric — the
|
||||
node-exporter textfile collector route, still open.
|
||||
|
||||
### The second rule exists because the first one fails OPEN
|
||||
|
||||
`VPNEgressSteeringStale` needs `egress-uplink` in kube-state-metrics. Without it the comparison
|
||||
degrades to an empty vector and alerts nothing — so `VPNEgressUplinkLabelMissing` (warning,
|
||||
`for: 30m`) watches the precondition: a node labelled `egress-gateway=true` with no `egress-uplink`
|
||||
label visible. Live right now it returns **1**, correctly: the KSM allowlist extension is in this
|
||||
same branch and not yet deployed. It self-clears on the first reconcile, and `for: 30m` covers the
|
||||
kube-state-metrics roll. The rule and the allowlist are in the same Flux kustomization on purpose.
|
||||
|
||||
`kube-state-metrics.metricLabelsAllowlist` now reads
|
||||
`nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]` — a label
|
||||
DIMENSION on the same 3 `kube_node_labels` series, not new series, on exactly one node.
|
||||
|
||||
### Rejected, with reasons
|
||||
|
||||
- **A black-box egress probe** is the most honest signal — it is the only one that answers "does a
|
||||
vpn-class pod's traffic actually leave via the tunnel?" and the only one that catches "the label
|
||||
was written and Cilium ignored it anyway". Rejected **for now**: there is no blackbox-exporter and
|
||||
no probe CronJob in this repo, so it means a new always-on component (itself vpn-class, itself an
|
||||
alerting surface, itself dependent on a third-party endpoint) — and it could not have been
|
||||
verified on prod today. It remains the right next step.
|
||||
- **A node-exporter textfile metric from the re-steer script** (hand-off item 4's own suggestion) is
|
||||
the only route to a true level-triggered label-vs-MAP comparison. It needs a textfile directory
|
||||
mounted into the node-exporter DaemonSet plus an Ansible change applied on a prod node — neither
|
||||
of which this branch could execute or verify. Left open, now with the reason recorded.
|
||||
- **`absent(node_network_receive_bytes_total{device="tun0"})`-style presence checks** — that IS
|
||||
`VPNTunnelDown`, and it is exactly the blind spot: green while steering points at a dead ifindex.
|
||||
|
||||
### Left for the operator / not settled here
|
||||
|
||||
1. **The rules are unproven in the firing sense.** The expressions evaluate against live prod data
|
||||
and the stale branch was exercised with a real flap, but no alert has actually reached
|
||||
Alertmanager or the inbox. `conformance/tests/l3-fault-injection/vpn-killswitch` stops and
|
||||
restarts the tunnel and is the natural place to assert `VPNEgressSteeringStale` stays quiet
|
||||
through a healthy reconnect; the alerting L3 test with the SMTP sink is where a fired-and-emailed
|
||||
proof would come from. Neither was fired at prod unasked.
|
||||
2. **`HubbleDeniedFlowSpike` cannot fire** — no cilium/hubble metrics are scraped on either cluster.
|
||||
Pre-existing, orthogonal, and the same TODO in `platform/cilium/helmrelease.yaml` blocks it.
|
||||
3. **Window vs. stickiness** is a judgement call, not a measurement: 1h chosen so a missed flap
|
||||
alerts for ~45 minutes. If the operator would rather have a sticky alert that needs manual
|
||||
acknowledgement, that is the textfile-collector design, not a tweak to this one.
|
||||
@@ -328,6 +328,132 @@ spec:
|
||||
summary: "Egress VPN tunnel down"
|
||||
description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06
|
||||
#
|
||||
# `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods
|
||||
# having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0`
|
||||
# exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect
|
||||
# re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points
|
||||
# at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0,
|
||||
# 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and
|
||||
# carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in
|
||||
# exactly that state for ~80 minutes on 2026-09-12 with nothing alerting.
|
||||
#
|
||||
# The datapath now self-heals: `roles/vpn_egress_gateway` publishes
|
||||
# `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node from the
|
||||
# OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case),
|
||||
# and that node write is what makes Cilium re-resolve. These two rules watch whether it
|
||||
# ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage.
|
||||
#
|
||||
# WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's
|
||||
# `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT
|
||||
# observable in Prometheus, verified on prod 2026-09-12 rather than assumed:
|
||||
# * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`,
|
||||
# `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and
|
||||
# there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium
|
||||
# metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every
|
||||
# `cilium_*` and `hubble_*` query returns an empty vector today (which also means
|
||||
# `HubbleDeniedFlowSpike` above cannot fire yet).
|
||||
# * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map
|
||||
# CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF
|
||||
# map, not an exporter. No metric name is invented here to paper over that.
|
||||
# So the observable is the label: the input to the re-steer, compared against the live tun0
|
||||
# identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real
|
||||
# ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and
|
||||
# `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster).
|
||||
#
|
||||
# WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not
|
||||
# firing, the label write failing or never being deployed, the convergence timer being dead —
|
||||
# i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch
|
||||
# "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can,
|
||||
# until either the re-steer script exports the map's ifindex through a node-exporter textfile
|
||||
# collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists.
|
||||
# A probe was considered and rejected for now: there is no blackbox-exporter and no probe
|
||||
# CronJob in this repo, so it means a NEW always-on component whose own failures become a new
|
||||
# alerting surface — worth doing, but not as the thing that closes a hole today.
|
||||
#
|
||||
# WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric
|
||||
# VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no
|
||||
# label-to-number conversion, so the level-triggered equality test is not expressible. What IS
|
||||
# expressible is the same invariant counted over a window: the published label must change at
|
||||
# least as often as tun0 is re-created. `count(count_over_time(max by (<label>) (...)[1h:1m]))`
|
||||
# is the number of DISTINCT label values seen in the last hour (verified live: 1 for a label
|
||||
# that held still, 2 for one that changed value inside the window); the `max by (<label>)`
|
||||
# wrapper is load-bearing — it aggregates away `pod`/`instance`, so a kube-state-metrics
|
||||
# restart is not mistaken for a label change. Both sides reduce to a single unlabelled series,
|
||||
# which is why `and on()` joins cleanly, exactly as `VPNTunnelDown` does.
|
||||
#
|
||||
# Consequence, stated plainly: this is bounded by its window. A missed flap is alerted from
|
||||
# +15m to +1h and then the alert RESOLVES even if egress is still dead. That is a deliberate
|
||||
# trade against the repo's alert-fatigue history (an unbounded critical on the one channel
|
||||
# that must stay trustworthy is worse), and it still turns an 80-minute silent outage into an
|
||||
# email within 15 minutes. The textfile-collector gauge is what would make it level-triggered.
|
||||
- alert: VPNEgressSteeringStale
|
||||
# FIRES when the live tun0 was re-created more times in the last hour than the gateway
|
||||
# node's `egress-uplink` label changed — i.e. Cilium was not told about at least one
|
||||
# tunnel reconnect, so the BPF egress map is pointing at an ifindex that no longer exists
|
||||
# and vpn-class pods have no egress. Convergence is NOT this: the label is rewritten within
|
||||
# seconds of a connect (worst case ~5m, the up-script's 30 x 10s kube-API retry budget on a
|
||||
# cold boot), so `for: 15m` cannot fire on a healthy reconnect.
|
||||
expr: |
|
||||
(
|
||||
changes(max(node_network_iface_id{device="tun0"})[1h:1m])
|
||||
>
|
||||
(
|
||||
count(
|
||||
count_over_time(
|
||||
max by (label_homelab_keithjackson_dev_egress_uplink) (
|
||||
kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""}
|
||||
)[1h:1m]
|
||||
)
|
||||
) - 1
|
||||
)
|
||||
)
|
||||
and on()
|
||||
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "VPN egress steering is stale — vpn-class pods have no egress"
|
||||
description: >-
|
||||
The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
|
||||
Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
|
||||
being dropped, silently, while tun0 still looks healthy. Compare
|
||||
`kubectl get nodes -L homelab.keithjackson.dev/egress-uplink` against
|
||||
`cilium-dbg bpf egress list` on the gateway node; re-applying the CiliumEgressGatewayPolicy
|
||||
(or setting any node label) forces a fresh resolution.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
- alert: VPNEgressUplinkLabelMissing
|
||||
# The watchdog for the rule above: `VPNEgressSteeringStale` needs the `egress-uplink` label
|
||||
# in kube-state-metrics, and if that label is missing its comparison degrades to an empty
|
||||
# vector — it fails OPEN, alerting nothing. So alert on the precondition itself. A gateway
|
||||
# node with no published uplink means the re-steer mechanism is not in place at all
|
||||
# (roles/vpn_egress_gateway not applied on this node, the up-script never ran, or the
|
||||
# kube-state-metrics label allowlist in kube-prometheus-stack.yaml lost the entry), which is
|
||||
# the configuration that produced the 80-minute dev outage. Warning, not critical: it is a
|
||||
# rollout/config gap to fix in working hours, not a page. `for: 30m` so that the Helm upgrade
|
||||
# that rolls kube-state-metrics — both changes ship in this same kustomization — cannot page
|
||||
# on the way in, and so an Ansible run in progress does not either.
|
||||
expr: |
|
||||
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
||||
and on()
|
||||
absent(kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Egress gateway node publishes no egress-uplink label"
|
||||
description: >-
|
||||
A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
|
||||
`homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without it
|
||||
nothing can tell whether Cilium's egress steering followed the last tunnel reconnect, and
|
||||
`VPNEgressSteeringStale` is inert. Check the label on the node, then the
|
||||
`vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
|
||||
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
|
||||
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
|
||||
|
||||
@@ -230,8 +230,20 @@ spec:
|
||||
# without anyone having to remember to re-enable it later. Scoped to one label on one resource:
|
||||
# 3 extra series, no cardinality concern. Label name is sanitized by KSM to
|
||||
# `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
|
||||
#
|
||||
# ADDED 2026-09-12 — `homelab.keithjackson.dev/egress-uplink`. The gateway node publishes
|
||||
# `<tun0 address>-<ifindex>` here from the OpenVPN up-script on every connect (roles/
|
||||
# vpn_egress_gateway), and that node write is what makes Cilium re-resolve its cached egress
|
||||
# interface. Exporting the label is what lets `VPNEgressSteeringStale` and
|
||||
# `VPNEgressUplinkLabelMissing` (alerts.yaml) watch whether the re-steer mechanism actually
|
||||
# converged — the one thing nothing watched while the dev gateway sat dead for ~80 minutes.
|
||||
# Cost: the label exists on exactly ONE node (the singleton gateway), so this adds a label
|
||||
# DIMENSION to the same 3 `kube_node_labels` series, not new series. Its value changes only on
|
||||
# a tunnel reconnect (a handful of times a day at most), so the churn is negligible — but note
|
||||
# that a value change does retire one series and start another, which is precisely the property
|
||||
# the alerts key on. Sanitized name: `label_homelab_keithjackson_dev_egress_uplink`.
|
||||
metricLabelsAllowlist:
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway]"
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 20m
|
||||
|
||||
@@ -157,6 +157,31 @@ kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
|
||||
If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
|
||||
gateway node is the next stop.
|
||||
|
||||
### And now something watches it (2026-09-12)
|
||||
|
||||
The self-heal closed the datapath hole; it did not close the monitoring one, and a mechanism that
|
||||
silently stops working is the same outage again. Two rules in `platform/monitoring/alerts.yaml`:
|
||||
|
||||
- **`VPNEgressSteeringStale`** (critical, `for: 15m`) — fires when the live tun0 was re-created
|
||||
more times in the last hour than the `egress-uplink` label changed, i.e. Cilium was never told
|
||||
about a reconnect. This is the case `VPNTunnelDown` cannot see: tun0 present, bytes flowing,
|
||||
steering pointed at a destroyed ifindex.
|
||||
- **`VPNEgressUplinkLabelMissing`** (warning, `for: 30m`) — fires when a node carries
|
||||
`egress-gateway=true` but no `egress-uplink` label reaches kube-state-metrics, which is the state
|
||||
in which the rule above would be inert. The watchdog for the watchdog.
|
||||
|
||||
Both are gated on a gateway node existing (the same `kube_node_labels` guard `VPNTunnelDown` uses),
|
||||
so a cluster with no gateway cannot be made to fire them. `egress-uplink` had to be added to
|
||||
`kube-state-metrics.metricLabelsAllowlist` in `platform/monitoring/kube-prometheus-stack.yaml` for
|
||||
the label to exist as a metric at all — the two changes must ship together.
|
||||
|
||||
**What still is not watched:** "the label was written and Cilium still ignored it." Cilium OSS
|
||||
exposes no metric carrying the egress map's contents (the map is readable only through
|
||||
`cilium-dbg bpf egress list`), and the cilium-agent is not scraped on either cluster in the first
|
||||
place — no `cilium_*` or `hubble_*` series exist. Closing that last gap means either exporting the
|
||||
map's ifindex from the re-steer script through a node-exporter textfile collector, or a black-box
|
||||
probe from a vpn-class pod. Neither exists yet.
|
||||
|
||||
## Resilience — there is NO gateway failover (spec correction, escalated)
|
||||
|
||||
**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)
|
||||
|
||||
Reference in new issue
Block a user