fix(kubevirt): correct KubeVirtControlPlaneDown expr — wrong metric name + job label
The alert fired continuously for ~21h while the control plane was fully
healthy (both virt-controller and virt-api pods 1/1 Ready, KubeVirt CR
Deployed/Available=True). Two bugs, each making it fire regardless of state:
1. `kubevirt_virt_controller_ready` does not exist — the real metric is
`kubevirt_virt_controller_ready_status` (KubeVirt suffixes `_status`),
so that clause was always ABSENT (never matched).
2. `absent(up{job="kubevirt"} == 1)` — the KubeVirt scrape job label is
`kubevirt-prometheus-metrics` (the Service name), not `kubevirt`, so
`up{job="kubevirt"}` matched nothing and absent() was permanently 1.
Fix: use `sum(kubevirt_virt_{controller,api}_ready_status) == 0` (fires only
when every replica is unready) and the correct job label. virt-api added to
match the alert's stated scope (virt-controller/virt-api).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
1 parent
6a7e9d15e7
commit
b57b3e9d71
1 file changed
+8
-4
@@ -43,12 +43,16 @@ spec:
|
||||
summary: "KubeVirt VM crash-looping: {{ $labels.namespace }}/{{ $labels.pod }}"
|
||||
description: "virt-launcher pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted more than 3 times in 15m — the guest VM is crash-looping (panic / OOM / qemu crash)."
|
||||
runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-crashloop"
|
||||
# Control plane down: virt-handler (the per-node virt agent) or the KubeVirt API is unreachable,
|
||||
# so no VM lifecycle operations can proceed. absent() also fires if the whole target disappears.
|
||||
# Control plane down: no virt-controller or virt-api replica reports ready, so no VM lifecycle
|
||||
# operations can proceed. The metrics are `kubevirt_virt_{controller,api}_ready_status` (per-pod
|
||||
# 1=ready); sum()==0 means every replica is unready. The absent() clause also fires if the whole
|
||||
# KubeVirt metrics target disappears — its scrape job label is `kubevirt-prometheus-metrics` (the
|
||||
# Service name), NOT `kubevirt`.
|
||||
- alert: KubeVirtControlPlaneDown
|
||||
expr: |
|
||||
(kubevirt_virt_controller_ready == 0)
|
||||
or (absent(up{job="kubevirt"} == 1))
|
||||
(sum(kubevirt_virt_controller_ready_status) == 0)
|
||||
or (sum(kubevirt_virt_api_ready_status) == 0)
|
||||
or (absent(up{job="kubevirt-prometheus-metrics"} == 1))
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
|
||||
Reference in new issue
Block a user