jakman85andClaude Opus 4.8
dbffb8fab8
- platform/kubevirt: pinned KubeVirt v1.9.0 + CDI v1.66.0 operators (remote bases); KubeVirt/CDI CRs tuned for Longhorn; useEmulation=true (prefer real KVM, emulate fallback). - Controllers/configs split (F-MON1): CRs + ServiceMonitors + alerts via platform-config. - Observability: ServiceMonitors, KubeVirt platform-health alerts (R-OBS-04), VM dashboard. - catalog/vm-demo.yaml (kind: vm) + rendered VirtualMachine/DataVolume fixture. - Conformance: vm-lifecycle (R-PLAT-03/04), vm-data-survival (R-STATE-01, L3), vm-default-deny (R-NET-01) — self-skipping until KubeVirt+longhorn exist. - ci/SPEC_PIN: adopt accepted spec b633b7f (ADR-0015 egress work already merged). - conformance/run.sh: document R-ENV-02/R-GITOPS-01 coverage gaps (WS-GITOPS/INFRA-owned). Co-Authored-By: Claude Opus 4.8 <[email protected]>
71 lines
4.4 KiB
YAML
71 lines
4.4 KiB
YAML
---
|
|
# KubeVirt platform-health alerts (docs/10, R-OBS-04): "VM crash-loops / failed launches" as a
|
|
# first-class platform signal, alongside the node/Longhorn/Flux/cert/VPN/backup/identity rules in
|
|
# platform/monitoring/alerts.yaml. PrometheusRule is a CR of the monitoring.coreos.com CRD installed
|
|
# by kube-prometheus-stack, so this is applied by the `platform-config` Flux Kustomization (dependsOn:
|
|
# platform), NOT the controllers pass (F-MON1) — identical wiring to platform/monitoring/alerts.yaml.
|
|
#
|
|
# The `release: kube-prometheus-stack` label makes the operator's Prometheus select these rules. Metric
|
|
# names follow KubeVirt's `kubevirt_*` series; where a series depends on the deployed exporter it is
|
|
# flagged to confirm at bring-up (same convention as platform/monitoring/alerts.yaml).
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-kubevirt
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.kubevirt
|
|
rules:
|
|
# Failed launches: a VMI that reached the Failed phase (scheduling/launch/qemu failure). Any
|
|
# sustained Failed VMI is page-worthy — the IaaS path is broken for that workload.
|
|
- alert: KubeVirtVMLaunchFailing
|
|
expr: sum by (namespace) (kubevirt_vmi_phase_count{phase="Failed"}) > 0
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "KubeVirt VM(s) failing to launch in {{ $labels.namespace }}"
|
|
description: "One or more VirtualMachineInstances are in the Failed phase in namespace {{ $labels.namespace }} for 5m — a failed VM launch (image import, scheduling, or qemu error). Check `kubectl get vmi -n {{ $labels.namespace }}` and the virt-launcher pod events."
|
|
runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-launch-failed"
|
|
# Crash-loop: the VM's virt-launcher pod is restarting repeatedly (guest kernel panic, OOM, or a
|
|
# crashing qemu). >3 restarts in 15m is a crash-loop, not a one-off reboot.
|
|
- alert: KubeVirtVMCrashLooping
|
|
expr: |
|
|
increase(kube_pod_container_status_restarts_total{pod=~"virt-launcher-.*"}[15m]) > 3
|
|
for: 10m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "KubeVirt VM crash-looping: {{ $labels.namespace }}/{{ $labels.pod }}"
|
|
description: "virt-launcher pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted more than 3 times in 15m — the guest VM is crash-looping (panic / OOM / qemu crash)."
|
|
runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-crashloop"
|
|
# Control plane down: virt-handler (the per-node virt agent) or the KubeVirt API is unreachable,
|
|
# so no VM lifecycle operations can proceed. absent() also fires if the whole target disappears.
|
|
- alert: KubeVirtControlPlaneDown
|
|
expr: |
|
|
(kubevirt_virt_controller_ready == 0)
|
|
or (absent(up{job="kubevirt"} == 1))
|
|
for: 10m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "KubeVirt control plane degraded"
|
|
description: "The KubeVirt control plane (virt-controller/virt-api) is not ready or its metrics are absent for 10m — VM lifecycle operations (create/start/stop) will fail (docs/06)."
|
|
runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-control-plane"
|
|
# CDI import stuck: a DataVolume import/clone still in flight after 30m blocks the VM it backs.
|
|
# kubevirt_cdi_import_pods_high_restart counts importer pods stuck restarting; a positive value
|
|
# sustained means an import is not progressing (metric name confirmed against CDI at bring-up).
|
|
- alert: CDIDataVolumeImportStuck
|
|
expr: sum(kubevirt_cdi_import_pods_high_restart) > 0
|
|
for: 30m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "CDI DataVolume import stuck: {{ $labels.namespace }}/{{ $labels.name }}"
|
|
description: "A CDI DataVolume import/clone in {{ $labels.namespace }} has not completed after 30m — the VM disk it backs cannot start. Check the importer pod (often an egress/default-deny block on the image source, or a slow mirror)."
|
|
runbook_url: "https://home.keithjackson.dev/runbooks/cdi-import-stuck"
|