From b57b3e9d71a0873c9ded679453a64cc51e453b22 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Wed, 26 Aug 2026 10:45:23 -0500 Subject: [PATCH] =?UTF-8?q?fix(kubevirt):=20correct=20KubeVirtControlPlane?= =?UTF-8?q?Down=20expr=20=E2=80=94=20wrong=20metric=20name=20+=20job=20lab?= =?UTF-8?q?el?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The alert fired continuously for ~21h while the control plane was fully healthy (both virt-controller and virt-api pods 1/1 Ready, KubeVirt CR Deployed/Available=True). Two bugs, each making it fire regardless of state: 1. `kubevirt_virt_controller_ready` does not exist — the real metric is `kubevirt_virt_controller_ready_status` (KubeVirt suffixes `_status`), so that clause was always ABSENT (never matched). 2. `absent(up{job="kubevirt"} == 1)` — the KubeVirt scrape job label is `kubevirt-prometheus-metrics` (the Service name), not `kubevirt`, so `up{job="kubevirt"}` matched nothing and absent() was permanently 1. Fix: use `sum(kubevirt_virt_{controller,api}_ready_status) == 0` (fires only when every replica is unready) and the correct job label. virt-api added to match the alert's stated scope (virt-controller/virt-api). Co-Authored-By: Claude Opus 4.8 --- platform/kubevirt/alerts.yaml | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/platform/kubevirt/alerts.yaml b/platform/kubevirt/alerts.yaml index acb3c3a..fbaff98 100644 --- a/platform/kubevirt/alerts.yaml +++ b/platform/kubevirt/alerts.yaml @@ -43,12 +43,16 @@ spec: summary: "KubeVirt VM crash-looping: {{ $labels.namespace }}/{{ $labels.pod }}" description: "virt-launcher pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted more than 3 times in 15m — the guest VM is crash-looping (panic / OOM / qemu crash)." runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-crashloop" - # Control plane down: virt-handler (the per-node virt agent) or the KubeVirt API is unreachable, - # so no VM lifecycle operations can proceed. absent() also fires if the whole target disappears. + # Control plane down: no virt-controller or virt-api replica reports ready, so no VM lifecycle + # operations can proceed. The metrics are `kubevirt_virt_{controller,api}_ready_status` (per-pod + # 1=ready); sum()==0 means every replica is unready. The absent() clause also fires if the whole + # KubeVirt metrics target disappears — its scrape job label is `kubevirt-prometheus-metrics` (the + # Service name), NOT `kubevirt`. - alert: KubeVirtControlPlaneDown expr: | - (kubevirt_virt_controller_ready == 0) - or (absent(up{job="kubevirt"} == 1)) + (sum(kubevirt_virt_controller_ready_status) == 0) + or (sum(kubevirt_virt_api_ready_status) == 0) + or (absent(up{job="kubevirt-prometheus-metrics"} == 1)) for: 10m labels: severity: critical