fix(kubevirt): correct KubeVirtControlPlaneDown expr — wrong metric name + job label

The alert fired continuously for ~21h while the control plane was fully
healthy (both virt-controller and virt-api pods 1/1 Ready, KubeVirt CR
Deployed/Available=True). Two bugs, each making it fire regardless of state:

  1. `kubevirt_virt_controller_ready` does not exist — the real metric is
     `kubevirt_virt_controller_ready_status` (KubeVirt suffixes `_status`),
     so that clause was always ABSENT (never matched).
  2. `absent(up{job="kubevirt"} == 1)` — the KubeVirt scrape job label is
     `kubevirt-prometheus-metrics` (the Service name), not `kubevirt`, so
     `up{job="kubevirt"}` matched nothing and absent() was permanently 1.

Fix: use `sum(kubevirt_virt_{controller,api}_ready_status) == 0` (fires only
when every replica is unready) and the correct job label. virt-api added to
match the alert's stated scope (virt-controller/virt-api).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-26 10:45:23 -05:00
1 parent 6a7e9d15e7
commit b57b3e9d71
1 file changed
+8 -4
+8 -4
View File
@@ -43,12 +43,16 @@ spec:
summary: "KubeVirt VM crash-looping: {{ $labels.namespace }}/{{ $labels.pod }}"
description: "virt-launcher pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted more than 3 times in 15m — the guest VM is crash-looping (panic / OOM / qemu crash)."
runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-crashloop"
# Control plane down: virt-handler (the per-node virt agent) or the KubeVirt API is unreachable,
# so no VM lifecycle operations can proceed. absent() also fires if the whole target disappears.
# Control plane down: no virt-controller or virt-api replica reports ready, so no VM lifecycle
# operations can proceed. The metrics are `kubevirt_virt_{controller,api}_ready_status` (per-pod
# 1=ready); sum()==0 means every replica is unready. The absent() clause also fires if the whole
# KubeVirt metrics target disappears — its scrape job label is `kubevirt-prometheus-metrics` (the
# Service name), NOT `kubevirt`.
- alert: KubeVirtControlPlaneDown
expr: |
(kubevirt_virt_controller_ready == 0)
or (absent(up{job="kubevirt"} == 1))
(sum(kubevirt_virt_controller_ready_status) == 0)
or (sum(kubevirt_virt_api_ready_status) == 0)
or (absent(up{job="kubevirt-prometheus-metrics"} == 1))
for: 10m
labels:
severity: critical