jakman85andClaude Opus 4.8
b57b3e9d71
The alert fired continuously for ~21h while the control plane was fully
healthy (both virt-controller and virt-api pods 1/1 Ready, KubeVirt CR
Deployed/Available=True). Two bugs, each making it fire regardless of state:
1. `kubevirt_virt_controller_ready` does not exist — the real metric is
`kubevirt_virt_controller_ready_status` (KubeVirt suffixes `_status`),
so that clause was always ABSENT (never matched).
2. `absent(up{job="kubevirt"} == 1)` — the KubeVirt scrape job label is
`kubevirt-prometheus-metrics` (the Service name), not `kubevirt`, so
`up{job="kubevirt"}` matched nothing and absent() was permanently 1.
Fix: use `sum(kubevirt_virt_{controller,api}_ready_status) == 0` (fires only
when every replica is unready) and the correct job label. virt-api added to
match the alert's stated scope (virt-controller/virt-api).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
75 lines
4.7 KiB
YAML
75 lines
4.7 KiB
YAML
---
|
|
# KubeVirt platform-health alerts (docs/10, R-OBS-04): "VM crash-loops / failed launches" as a
|
|
# first-class platform signal, alongside the node/Longhorn/Flux/cert/VPN/backup/identity rules in
|
|
# platform/monitoring/alerts.yaml. PrometheusRule is a CR of the monitoring.coreos.com CRD installed
|
|
# by kube-prometheus-stack, so this is applied by the `platform-config` Flux Kustomization (dependsOn:
|
|
# platform), NOT the controllers pass (F-MON1) — identical wiring to platform/monitoring/alerts.yaml.
|
|
#
|
|
# The `release: kube-prometheus-stack` label makes the operator's Prometheus select these rules. Metric
|
|
# names follow KubeVirt's `kubevirt_*` series; where a series depends on the deployed exporter it is
|
|
# flagged to confirm at bring-up (same convention as platform/monitoring/alerts.yaml).
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-kubevirt
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.kubevirt
|
|
rules:
|
|
# Failed launches: a VMI that reached the Failed phase (scheduling/launch/qemu failure). Any
|
|
# sustained Failed VMI is page-worthy — the IaaS path is broken for that workload.
|
|
- alert: KubeVirtVMLaunchFailing
|
|
expr: sum by (namespace) (kubevirt_vmi_phase_count{phase="Failed"}) > 0
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "KubeVirt VM(s) failing to launch in {{ $labels.namespace }}"
|
|
description: "One or more VirtualMachineInstances are in the Failed phase in namespace {{ $labels.namespace }} for 5m — a failed VM launch (image import, scheduling, or qemu error). Check `kubectl get vmi -n {{ $labels.namespace }}` and the virt-launcher pod events."
|
|
runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-launch-failed"
|
|
# Crash-loop: the VM's virt-launcher pod is restarting repeatedly (guest kernel panic, OOM, or a
|
|
# crashing qemu). >3 restarts in 15m is a crash-loop, not a one-off reboot.
|
|
- alert: KubeVirtVMCrashLooping
|
|
expr: |
|
|
increase(kube_pod_container_status_restarts_total{pod=~"virt-launcher-.*"}[15m]) > 3
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "KubeVirt VM crash-looping: {{ $labels.namespace }}/{{ $labels.pod }}"
|
|
description: "virt-launcher pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted more than 3 times in 15m — the guest VM is crash-looping (panic / OOM / qemu crash)."
|
|
runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-crashloop"
|
|
# Control plane down: no virt-controller or virt-api replica reports ready, so no VM lifecycle
|
|
# operations can proceed. The metrics are `kubevirt_virt_{controller,api}_ready_status` (per-pod
|
|
# 1=ready); sum()==0 means every replica is unready. The absent() clause also fires if the whole
|
|
# KubeVirt metrics target disappears — its scrape job label is `kubevirt-prometheus-metrics` (the
|
|
# Service name), NOT `kubevirt`.
|
|
- alert: KubeVirtControlPlaneDown
|
|
expr: |
|
|
(sum(kubevirt_virt_controller_ready_status) == 0)
|
|
or (sum(kubevirt_virt_api_ready_status) == 0)
|
|
or (absent(up{job="kubevirt-prometheus-metrics"} == 1))
|
|
for: 10m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "KubeVirt control plane degraded"
|
|
description: "The KubeVirt control plane (virt-controller/virt-api) is not ready or its metrics are absent for 10m — VM lifecycle operations (create/start/stop) will fail (docs/06)."
|
|
runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-control-plane"
|
|
# CDI import stuck: a DataVolume import/clone still in flight after 30m blocks the VM it backs.
|
|
# kubevirt_cdi_import_pods_high_restart counts importer pods stuck restarting; a positive value
|
|
# sustained means an import is not progressing (metric name confirmed against CDI at bring-up).
|
|
- alert: CDIDataVolumeImportStuck
|
|
expr: sum(kubevirt_cdi_import_pods_high_restart) > 0
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CDI DataVolume import stuck: {{ $labels.namespace }}/{{ $labels.name }}"
|
|
description: "A CDI DataVolume import/clone in {{ $labels.namespace }} has not completed after 30m — the VM disk it backs cannot start. Check the importer pod (often an egress/default-deny block on the image source, or a slow mirror)."
|
|
runbook_url: "https://home.keithjackson.dev/runbooks/cdi-import-stuck"
|