# kubevirt-control-plane — The KubeVirt control plane is degraded **Alert:** `KubeVirtControlPlaneDown` (critical, 10m) **File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt` **Expression:** ``` (sum(kubevirt_virt_controller_ready_status) == 0) or (sum(kubevirt_virt_api_ready_status) == 0) or (absent(up{job="kubevirt-prometheus-metrics"} == 1)) ``` ## What this means here No `virt-controller` or no `virt-api` replica reports ready, so **VM lifecycle operations — create, start, stop, migrate — will fail** (docs/06). Note carefully what this does *not* mean: **already-running VMs keep running.** `virt-launcher` pods and their qemu processes are independent of the control plane. So this is an "I cannot manage VMs" alert, not "the VMs are down". The `absent()` clause also fires if the whole KubeVirt metrics target disappears. The scrape job label is **`kubevirt-prometheus-metrics`** (the Service name), **not** `kubevirt` — that precision is a fix: this very alert once used a non-existent metric with the wrong job label and fired falsely (commit `b57b3e9`). There is a sibling precedent too: a broken HTTP CDI ServiceMonitor caused a false `CDIOperatorDown` (commit `6a7e9d1`). **So the first question with this alert is always whether it is telling the truth.** Live state, measured 2026-09-13: `kubevirt-prometheus-metrics` **is** among Prometheus's 19 scrape jobs and `kubevirt_virt_api_ready_status` returns **2 series**, so the first two clauses are evaluable and quiet. (`kubevirt_vmi_phase_count` was absent at the same moment — that metric only exists when VMIs do, which affects `/runbooks/kubevirt-launch-failed`, not this rule.) Platform context: **KubeVirt v1.9.0 + CDI v1.66.0**, on real nested KVM (`/dev/kvm` present, vmx/svm verified; the Proxmox nodes use `cpu_type=host`). The `kubevirt` and `cdi` namespaces deliberately carry **no `app-namespace` label**, so Kyverno does not generate the default-deny baseline into them — it would sever the virt control plane. Do not "fix" that by adding the label. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Is the control plane actually unhealthy, or is this a metrics artefact? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n kubevirt get pods -o wide; echo; \ kubectl --context k3s-prod get kubevirts.kubevirt.io -A \ -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,READY:.status.conditions[?\(@.type==\"Available\"\)].status' ``` Are the VMs themselves still running? (This is the reassuring half.) ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get vmi -A -o wide; echo; \ kubectl --context k3s-prod get vm -A' ``` Is the scrape target up, and do the two metrics exist? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ for q in "up{job=\"kubevirt-prometheus-metrics\"}" "sum(kubevirt_virt_controller_ready_status)" "sum(kubevirt_virt_api_ready_status)"; do \ printf "%-55s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \ | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done' ``` Why a component is unready: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n kubevirt logs deploy/virt-controller --tail=80 2>&1 | tail -30; echo ====; \ kubectl --context k3s-prod -n kubevirt logs deploy/virt-api --tail=80 2>&1 | tail -30; echo ====; \ kubectl --context k3s-prod -n kubevirt get events --sort-by=.lastTimestamp | tail -20' ``` Is the Flux tier that owns KubeVirt healthy? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \ -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message' ``` ## Likely causes, most likely first ### 1. The metrics target, not the control plane Check the pods first. If `virt-api` and `virt-controller` are Running and Ready, this is a monitoring fault — and there is direct precedent for false positives on this exact alert (`b57b3e9`) and its CDI sibling (`6a7e9d1`). Remediation: fix the ServiceMonitor/metric in git. Cross-read `/runbooks/targets-down`. ### 2. A node that hosted the control-plane pods went away `virt-controller` and `virt-api` are ordinary Deployments. Go to `/runbooks/node-down`. ### 3. The KubeVirt Flux tier failed to reconcile Specific known hazard: **`platform/kubevirt` still uses remote kustomize bases**, and the kustomize-controller's git fetch has a **27s timeout** that remote GitHub bases flake against — a `BuildFail` there is probably this. `external-snapshotter` was vendored into the repo for exactly this reason; vendoring KubeVirt the same way is the durable fix. See `/runbooks/flux-reconcile`. ### 4. Node pressure Heavy sustained node load — TCG-emulated VMs are the recorded example — can degrade the Cilium datapath into intermittent LB-VIP reachability problems, which looks like control-plane flapping. A restart cleared it. Cross-read `/runbooks/node-io` and `/runbooks/node-cpu`. ### 5. A bad operator upgrade `/runbooks/update-rollback`. ### 6. A NetworkPolicy severed the control plane Only if someone added `homelab.keithjackson.dev/app-namespace=true` to `kubevirt` or `cdi`. Those namespaces are deliberately unlabelled so Kyverno does not inject a default-deny baseline that would cut virt-handler off. Reverting that label is the fix. ## When to escalate or stop - **Prioritise honestly.** VM *lifecycle* is broken; running VMs are not. The UniFi VM keeps serving. Treat this as urgent-but-not-catastrophic unless a VM also needs to start. - **Do not drain a node to move virt pods.** This cluster does not drain, and a VM live-migrates on drain **only** with per-VM `evictionStrategy: LiveMigrate` — a blanket setting would wedge drains for RWO-disk VMs. - **Do not delete a VMI or `virt-launcher` pod** while the control plane is down: nothing will bring the VM back until `virt-controller` is ready again. - **Do not add the `app-namespace` label to `kubevirt` or `cdi`** to "harden" them. - Escalate for vendoring the remote bases or for any operator upgrade. If the alert is a false positive, fixing the rule is itself worth a change — a permanently-firing critical is a recorded, expensive failure mode here. --- Source of truth: `runbooks/kubevirt-control-plane.md`. Index: `/runbooks/` Records: commits `b57b3e9` (this alert's wrong metric/job label) and `6a7e9d1` (false `CDIOperatorDown`); `DECISIONS-LOG.md` (KubeVirt v1.9.0 + CDI v1.66.0 on nested KVM; the remote-base 27s fetch timeout).