monitoring: retarget VPNTunnelDown to node tun0 presence; drop obsolete EgressGatewayUnhealthy
gluetun retirement removes the gluetun_vpn_up gauge and the in-cluster gateway pod. Point
VPNTunnelDown at absent(node_network_receive_bytes_total{device=tun0}) — tun0 exists iff the
host OpenVPN tunnel is up (node_network_up unreliable: tun0 operstate=unknown). This also
fires if the gateway node/node-exporter is down (series goes stale). Remove the pod-readiness
EgressGatewayUnhealthy alert — there is no gateway pod under ADR-0015.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
1 parent
7b7982e28d
commit
d519ede4e3
1 file changed
+12
-14
@@ -213,11 +213,14 @@ spec:
|
|||||||
groups:
|
groups:
|
||||||
- name: platform-health.networking
|
- name: platform-health.networking
|
||||||
rules:
|
rules:
|
||||||
# VPN tunnel health. The concrete metric depends on the egress-gateway VPN sidecar chosen by
|
# VPN tunnel health (ADR-0015: node-level host OpenVPN tunnel on the egress-gateway node, steered
|
||||||
# WS-NET (docs/03). Written against a gluetun-style up gauge; confirm/adjust at bring-up.
|
# by Cilium Egress Gateway — there is NO in-cluster gateway pod). OpenVPN creates/destroys the tun0
|
||||||
|
# device with the connection, so the tun0 netdev series exists iff the tunnel is up. node_network_up
|
||||||
|
# is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent()
|
||||||
|
# also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering
|
||||||
|
# gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24.
|
||||||
- alert: VPNTunnelDown
|
- alert: VPNTunnelDown
|
||||||
# TODO(WS-NET): confirm metric name of the egress VPN exporter (docs/03).
|
expr: absent(node_network_receive_bytes_total{device="tun0"})
|
||||||
expr: max(gluetun_vpn_up) == 0 or absent(gluetun_vpn_up)
|
|
||||||
for: 5m
|
for: 5m
|
||||||
labels:
|
labels:
|
||||||
severity: critical
|
severity: critical
|
||||||
@@ -225,16 +228,11 @@ spec:
|
|||||||
summary: "Egress VPN tunnel down"
|
summary: "Egress VPN tunnel down"
|
||||||
description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (R-SEC-03 leak risk)."
|
description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (R-SEC-03 leak risk)."
|
||||||
runbook_url: "https://home.keithjackson.dev/runbooks/vpn-down"
|
runbook_url: "https://home.keithjackson.dev/runbooks/vpn-down"
|
||||||
- alert: EgressGatewayUnhealthy
|
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
|
||||||
expr: |
|
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
|
||||||
max by (pod) (kube_pod_status_ready{namespace="egress", condition="true"}) == 0
|
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
|
||||||
for: 10m
|
# above (tun0 presence, which also goes absent if the gateway node is down); Cilium datapath
|
||||||
labels:
|
# health is covered by the Cilium platform alerts.
|
||||||
severity: critical
|
|
||||||
annotations:
|
|
||||||
summary: "Egress gateway unhealthy"
|
|
||||||
description: "No ready egress-gateway pod for 10m — VPN-scoped egress is unavailable."
|
|
||||||
runbook_url: "https://home.keithjackson.dev/runbooks/egress-gateway"
|
|
||||||
- alert: HubbleDeniedFlowSpike
|
- alert: HubbleDeniedFlowSpike
|
||||||
# Cilium/Hubble denied (policy-drop) flows. Requires hubble metrics enabled (docs/03).
|
# Cilium/Hubble denied (policy-drop) flows. Requires hubble metrics enabled (docs/03).
|
||||||
expr: sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5
|
expr: sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5
|
||||||
|
|||||||
Reference in new issue
Block a user