merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
commit
a3f4ddb971
14 files changed
+864
-5
No files matched your search
@@ -904,6 +904,132 @@ spec:
|
||||
summary: "Egress VPN tunnel down"
|
||||
description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06
|
||||
#
|
||||
# `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods
|
||||
# having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0`
|
||||
# exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect
|
||||
# re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points
|
||||
# at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0,
|
||||
# 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and
|
||||
# carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in
|
||||
# exactly that state for ~80 minutes on 2026-09-12 with nothing alerting.
|
||||
#
|
||||
# The datapath now self-heals: `roles/vpn_egress_gateway` publishes
|
||||
# `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node from the
|
||||
# OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case),
|
||||
# and that node write is what makes Cilium re-resolve. These two rules watch whether it
|
||||
# ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage.
|
||||
#
|
||||
# WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's
|
||||
# `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT
|
||||
# observable in Prometheus, verified on prod 2026-09-12 rather than assumed:
|
||||
# * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`,
|
||||
# `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and
|
||||
# there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium
|
||||
# metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every
|
||||
# `cilium_*` and `hubble_*` query returns an empty vector today (which also means
|
||||
# `HubbleDeniedFlowSpike` above cannot fire yet).
|
||||
# * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map
|
||||
# CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF
|
||||
# map, not an exporter. No metric name is invented here to paper over that.
|
||||
# So the observable is the label: the input to the re-steer, compared against the live tun0
|
||||
# identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real
|
||||
# ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and
|
||||
# `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster).
|
||||
#
|
||||
# WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not
|
||||
# firing, the label write failing or never being deployed, the convergence timer being dead —
|
||||
# i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch
|
||||
# "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can,
|
||||
# until either the re-steer script exports the map's ifindex through a node-exporter textfile
|
||||
# collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists.
|
||||
# A probe was considered and rejected for now: there is no blackbox-exporter and no probe
|
||||
# CronJob in this repo, so it means a NEW always-on component whose own failures become a new
|
||||
# alerting surface — worth doing, but not as the thing that closes a hole today.
|
||||
#
|
||||
# WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric
|
||||
# VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no
|
||||
# label-to-number conversion, so the level-triggered equality test is not expressible. What IS
|
||||
# expressible is the same invariant counted over a window: the published label must change at
|
||||
# least as often as tun0 is re-created. `count(count_over_time(max by (<label>) (...)[1h:1m]))`
|
||||
# is the number of DISTINCT label values seen in the last hour (verified live: 1 for a label
|
||||
# that held still, 2 for one that changed value inside the window); the `max by (<label>)`
|
||||
# wrapper is load-bearing — it aggregates away `pod`/`instance`, so a kube-state-metrics
|
||||
# restart is not mistaken for a label change. Both sides reduce to a single unlabelled series,
|
||||
# which is why `and on()` joins cleanly, exactly as `VPNTunnelDown` does.
|
||||
#
|
||||
# Consequence, stated plainly: this is bounded by its window. A missed flap is alerted from
|
||||
# +15m to +1h and then the alert RESOLVES even if egress is still dead. That is a deliberate
|
||||
# trade against the repo's alert-fatigue history (an unbounded critical on the one channel
|
||||
# that must stay trustworthy is worse), and it still turns an 80-minute silent outage into an
|
||||
# email within 15 minutes. The textfile-collector gauge is what would make it level-triggered.
|
||||
- alert: VPNEgressSteeringStale
|
||||
# FIRES when the live tun0 was re-created more times in the last hour than the gateway
|
||||
# node's `egress-uplink` label changed — i.e. Cilium was not told about at least one
|
||||
# tunnel reconnect, so the BPF egress map is pointing at an ifindex that no longer exists
|
||||
# and vpn-class pods have no egress. Convergence is NOT this: the label is rewritten within
|
||||
# seconds of a connect (worst case ~5m, the up-script's 30 x 10s kube-API retry budget on a
|
||||
# cold boot), so `for: 15m` cannot fire on a healthy reconnect.
|
||||
expr: |
|
||||
(
|
||||
changes(max(node_network_iface_id{device="tun0"})[1h:1m])
|
||||
>
|
||||
(
|
||||
count(
|
||||
count_over_time(
|
||||
max by (label_homelab_keithjackson_dev_egress_uplink) (
|
||||
kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""}
|
||||
)[1h:1m]
|
||||
)
|
||||
) - 1
|
||||
)
|
||||
)
|
||||
and on()
|
||||
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "VPN egress steering is stale — vpn-class pods have no egress"
|
||||
description: >-
|
||||
The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
|
||||
Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
|
||||
being dropped, silently, while tun0 still looks healthy. Compare
|
||||
`kubectl get nodes -L homelab.keithjackson.dev/egress-uplink` against
|
||||
`cilium-dbg bpf egress list` on the gateway node; re-applying the CiliumEgressGatewayPolicy
|
||||
(or setting any node label) forces a fresh resolution.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
- alert: VPNEgressUplinkLabelMissing
|
||||
# The watchdog for the rule above: `VPNEgressSteeringStale` needs the `egress-uplink` label
|
||||
# in kube-state-metrics, and if that label is missing its comparison degrades to an empty
|
||||
# vector — it fails OPEN, alerting nothing. So alert on the precondition itself. A gateway
|
||||
# node with no published uplink means the re-steer mechanism is not in place at all
|
||||
# (roles/vpn_egress_gateway not applied on this node, the up-script never ran, or the
|
||||
# kube-state-metrics label allowlist in kube-prometheus-stack.yaml lost the entry), which is
|
||||
# the configuration that produced the 80-minute dev outage. Warning, not critical: it is a
|
||||
# rollout/config gap to fix in working hours, not a page. `for: 30m` so that the Helm upgrade
|
||||
# that rolls kube-state-metrics — both changes ship in this same kustomization — cannot page
|
||||
# on the way in, and so an Ansible run in progress does not either.
|
||||
expr: |
|
||||
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
||||
and on()
|
||||
absent(kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Egress gateway node publishes no egress-uplink label"
|
||||
description: >-
|
||||
A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
|
||||
`homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without it
|
||||
nothing can tell whether Cilium's egress steering followed the last tunnel reconnect, and
|
||||
`VPNEgressSteeringStale` is inert. Check the label on the node, then the
|
||||
`vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
|
||||
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
|
||||
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
|
||||
|
||||
@@ -231,7 +231,7 @@ spec:
|
||||
# 3 extra series, no cardinality concern. Label name is sanitized by KSM to
|
||||
# `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
|
||||
metricLabelsAllowlist:
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway]"
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
|
||||
# --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
|
||||
# Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
|
||||
# `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
# promtool rule-unit tests for VPNEgressSteeringStale + VPNEgressUplinkLabelMissing
|
||||
# (platform/monitoring/alerts.yaml)
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh
|
||||
# (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS
|
||||
# `egress-class=vpn` pods leave through one gateway node's tun0. Cilium resolves that interface to an
|
||||
# IFINDEX and caches it; an OpenVPN reconnect destroys and re-creates tun0 with a NEW ifindex, and the
|
||||
# gateway node's up-script publishes `homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex>` so that
|
||||
# node write forces Cilium to re-resolve. If the label does not follow the tunnel, the cache points at a
|
||||
# destroyed interface and every vpn-class pod is dropped — while tun0 itself stays up, so VPNTunnelDown
|
||||
# says nothing. That shape ran ~80 minutes undetected on the dev gateway.
|
||||
#
|
||||
# VPNEgressSteeringStale detects it by comparing HOW MANY TIMES tun0's ifindex changed in the last hour
|
||||
# against HOW MANY DISTINCT `egress-uplink` label values appeared in the same hour. The subtlety worth
|
||||
# testing is that a label VALUE change is not a sample change: it RETIRES one series and STARTS another,
|
||||
# so "the label followed the tunnel" shows up as two overlapping series in the window, not as a
|
||||
# changing value. Tests 1 and 2 are the same tunnel reconnect with and without that second series, so
|
||||
# they isolate exactly that mechanism — anything that broke the series-counting half would make test 2
|
||||
# fire, and anything that broke the tun0 half would make test 1 quiet.
|
||||
#
|
||||
# VPNEgressUplinkLabelMissing is the WATCHDOG for the rule above: with no uplink label anywhere, the
|
||||
# staleness comparison has nothing to compare and is structurally INERT — present, evaluating, and
|
||||
# incapable of firing. Tests 3 and 4 pin that, because an inert critical alert is the failure mode this
|
||||
# whole group exists to prevent.
|
||||
#
|
||||
# Fixture values are prod's real shape: gateway k3s-prod-3, tun0 ifindex 139 with address
|
||||
# 10.150.192.86 (live label value `10.150.192.86-139`), reconnecting to ifindex 140.
|
||||
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# =============================================================================================
|
||||
# 1. THE BUG — tun0 was re-created, the label did NOT follow. Steering is stale: FIRES.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: tunnel reconnect with no label update - VPNEgressSteeringStale fires
|
||||
input_series:
|
||||
# tun0 re-created at t=41m: ifindex 139 -> 140. One change inside the trailing hour.
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '139+0x40 140+0x40'
|
||||
# tun0 is UP and passing traffic the whole time — this is what makes the failure deceptive, and
|
||||
# it is asserted below: VPNTunnelDown stays QUIET while egress is actually broken.
|
||||
- series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '0+120000x80'
|
||||
# The node label NEVER changes, so only ONE distinct uplink value exists in the window.
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-139"}'
|
||||
values: '1+0x80'
|
||||
alert_rule_test:
|
||||
# 1 tunnel change > (1 distinct label value - 1) = 0, held for 15m.
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
exp_annotations:
|
||||
summary: "VPN egress steering is stale — vpn-class pods have no egress"
|
||||
description: >-
|
||||
The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
|
||||
Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
|
||||
being dropped, silently, while tun0 still looks healthy. Compare `kubectl get nodes -L
|
||||
homelab.keithjackson.dev/egress-uplink` against `cilium-dbg bpf egress list` on the
|
||||
gateway node; re-applying the CiliumEgressGatewayPolicy (or setting any node label)
|
||||
forces a fresh resolution.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# Not yet: at 45m the condition has been true only ~4m, under the 15m `for`.
|
||||
- eval_time: 45m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
# No gateway label AND no tun0 history, so VPNTunnelDown's `or` guard is false: a cluster with
|
||||
# no VPN gateway must not be told its tunnel is down forever.
|
||||
- eval_time: 45m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
# The watchdog must stay quiet — the label IS present here.
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
# THE HEADLINE CLAIM, asserted rather than asserted-in-prose: the tunnel alert says NOTHING
|
||||
# while vpn-class egress is completely broken. If VPNTunnelDown covered this failure, the
|
||||
# staleness rule would not need to exist.
|
||||
- eval_time: 60m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 2. THE HEALTHY RECONNECT — same tunnel change, but the label followed it. STAYS QUIET.
|
||||
# This is the half that proves the rule is not simply "any reconnect alerts".
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: tunnel reconnect WITH the label following - stays quiet
|
||||
input_series:
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '139+0x40 140+0x40'
|
||||
- series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '0+120000x80'
|
||||
# A label VALUE change retires one series and starts another. Both are inside the trailing
|
||||
# hour, so the window holds TWO distinct uplink values: 2 - 1 = 1, which is NOT < 1 change.
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-139"}'
|
||||
values: '1+0x40 _x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-140"}'
|
||||
values: '_x41 1+0x39'
|
||||
alert_rule_test:
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
- eval_time: 75m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
- eval_time: 60m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 3. THE INERT-ALERT CASE — a gateway node publishing NO uplink label at all.
|
||||
# VPNEgressSteeringStale cannot fire in this state, which is exactly why this warning exists.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: gateway node with no egress-uplink label - VPNEgressUplinkLabelMissing fires
|
||||
input_series:
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true"}'
|
||||
values: '1+0x60'
|
||||
# tun0 is reconnecting the whole time. The staleness rule STILL cannot fire, because it has no
|
||||
# label series to compare against — that is the inertness being alerted on.
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '139+0x30 140+0x30'
|
||||
- series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
|
||||
values: '0+120000x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 35m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
exp_annotations:
|
||||
summary: "Egress gateway node publishes no egress-uplink label"
|
||||
description: >-
|
||||
A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
|
||||
`homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without
|
||||
it nothing can tell whether Cilium's egress steering followed the last tunnel reconnect,
|
||||
and `VPNEgressSteeringStale` is inert. Check the label on the node, then the
|
||||
`vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# Under the 30m `for`.
|
||||
- eval_time: 20m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
# And the proof of inertness: the critical rule is silent despite a real reconnect.
|
||||
- eval_time: 55m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
- eval_time: 55m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 4. NO GATEWAY NODE AT ALL — neither rule may fire. Both are guarded by an `and on()` on the
|
||||
# egress-gateway label so they stay silent on a cluster with no VPN gateway, rather than
|
||||
# alerting forever about an absent feature.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: no egress-gateway node - both rules silent
|
||||
input_series:
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}'
|
||||
values: '1+0x60'
|
||||
- series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.10:9100",device="eth0"}'
|
||||
values: '2+0x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 45m
|
||||
alertname: VPNEgressUplinkLabelMissing
|
||||
exp_alerts: []
|
||||
- eval_time: 45m
|
||||
alertname: VPNEgressSteeringStale
|
||||
exp_alerts: []
|
||||
# No gateway label AND no tun0 history, so VPNTunnelDown's `or` guard is false: a cluster with
|
||||
# no VPN gateway must not be told its tunnel is down forever.
|
||||
- eval_time: 45m
|
||||
alertname: VPNTunnelDown
|
||||
exp_alerts: []
|
||||
@@ -157,6 +157,31 @@ kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
|
||||
If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
|
||||
gateway node is the next stop.
|
||||
|
||||
### And now something watches it (2026-09-12)
|
||||
|
||||
The self-heal closed the datapath hole; it did not close the monitoring one, and a mechanism that
|
||||
silently stops working is the same outage again. Two rules in `platform/monitoring/alerts.yaml`:
|
||||
|
||||
- **`VPNEgressSteeringStale`** (critical, `for: 15m`) — fires when the live tun0 was re-created
|
||||
more times in the last hour than the `egress-uplink` label changed, i.e. Cilium was never told
|
||||
about a reconnect. This is the case `VPNTunnelDown` cannot see: tun0 present, bytes flowing,
|
||||
steering pointed at a destroyed ifindex.
|
||||
- **`VPNEgressUplinkLabelMissing`** (warning, `for: 30m`) — fires when a node carries
|
||||
`egress-gateway=true` but no `egress-uplink` label reaches kube-state-metrics, which is the state
|
||||
in which the rule above would be inert. The watchdog for the watchdog.
|
||||
|
||||
Both are gated on a gateway node existing (the same `kube_node_labels` guard `VPNTunnelDown` uses),
|
||||
so a cluster with no gateway cannot be made to fire them. `egress-uplink` had to be added to
|
||||
`kube-state-metrics.metricLabelsAllowlist` in `platform/monitoring/kube-prometheus-stack.yaml` for
|
||||
the label to exist as a metric at all — the two changes must ship together.
|
||||
|
||||
**What still is not watched:** "the label was written and Cilium still ignored it." Cilium OSS
|
||||
exposes no metric carrying the egress map's contents (the map is readable only through
|
||||
`cilium-dbg bpf egress list`), and the cilium-agent is not scraped on either cluster in the first
|
||||
place — no `cilium_*` or `hubble_*` series exist. Closing that last gap means either exporting the
|
||||
map's ifindex from the re-steer script through a node-exporter textfile collector, or a black-box
|
||||
probe from a vpn-class pod. Neither exists yet.
|
||||
|
||||
## Resilience — there is NO gateway failover (spec correction, escalated)
|
||||
|
||||
**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)
|
||||
|
||||
Reference in new issue
Block a user