home-network-k3s-impl/runbooks/kubevirt-control-plane.md

kubevirt-control-plane — The KubeVirt control plane is degraded

Alert: KubeVirtControlPlaneDown (critical, 10m) File: platform/kubevirt/alerts.yaml, group platform-health.kubevirt Expression:

(sum(kubevirt_virt_controller_ready_status) == 0)
or (sum(kubevirt_virt_api_ready_status) == 0)
or (absent(up{job="kubevirt-prometheus-metrics"} == 1))

What this means here

No virt-controller or no virt-api replica reports ready, so VM lifecycle operations — create, start, stop, migrate — will fail (docs/06). Note carefully what this does not mean: already-running VMs keep running. virt-launcher pods and their qemu processes are independent of the control plane. So this is an "I cannot manage VMs" alert, not "the VMs are down".

The absent() clause also fires if the whole KubeVirt metrics target disappears. The scrape job label is kubevirt-prometheus-metrics (the Service name), not kubevirt — that precision is a fix: this very alert once used a non-existent metric with the wrong job label and fired falsely (commit b57b3e9). There is a sibling precedent too: a broken HTTP CDI ServiceMonitor caused a false CDIOperatorDown (commit 6a7e9d1). So the first question with this alert is always whether it is telling the truth.

Live state, measured 2026-09-13: kubevirt-prometheus-metrics is among Prometheus's 19 scrape jobs and kubevirt_virt_api_ready_status returns 2 series, so the first two clauses are evaluable and quiet. (kubevirt_vmi_phase_count was absent at the same moment — that metric only exists when VMIs do, which affects /runbooks/kubevirt-launch-failed, not this rule.)

Platform context: KubeVirt v1.9.0 + CDI v1.66.0, on real nested KVM (/dev/kvm present, vmx/svm verified; the Proxmox nodes use cpu_type=host). The kubevirt and cdi namespaces deliberately carry no app-namespace label, so Kyverno does not generate the default-deny baseline into them — it would sever the virt control plane. Do not "fix" that by adding the label.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git. main is a protected branch — a direct push is rejected. A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Is the control plane actually unhealthy, or is this a metrics artefact?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n kubevirt get pods -o wide; echo; \
   kubectl --context k3s-prod get kubevirts.kubevirt.io -A \
     -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,READY:.status.conditions[?\(@.type==\"Available\"\)].status'

Are the VMs themselves still running? (This is the reassuring half.)

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get vmi -A -o wide; echo; \
   kubectl --context k3s-prod get vm -A'

Is the scrape target up, and do the two metrics exist?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   for q in "up{job=\"kubevirt-prometheus-metrics\"}" "sum(kubevirt_virt_controller_ready_status)" "sum(kubevirt_virt_api_ready_status)"; do \
     printf "%-55s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \
       | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done'

Why a component is unready:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n kubevirt logs deploy/virt-controller --tail=80 2>&1 | tail -30; echo ====; \
   kubectl --context k3s-prod -n kubevirt logs deploy/virt-api --tail=80 2>&1 | tail -30; echo ====; \
   kubectl --context k3s-prod -n kubevirt get events --sort-by=.lastTimestamp | tail -20'

Is the Flux tier that owns KubeVirt healthy?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
     -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'

Likely causes, most likely first

1. The metrics target, not the control plane

Check the pods first. If virt-api and virt-controller are Running and Ready, this is a monitoring fault — and there is direct precedent for false positives on this exact alert (b57b3e9) and its CDI sibling (6a7e9d1). Remediation: fix the ServiceMonitor/metric in git. Cross-read /runbooks/targets-down.

2. A node that hosted the control-plane pods went away

virt-controller and virt-api are ordinary Deployments. Go to /runbooks/node-down.

3. The KubeVirt Flux tier failed to reconcile

Specific known hazard: platform/kubevirt still uses remote kustomize bases, and the kustomize-controller's git fetch has a 27s timeout that remote GitHub bases flake against — a BuildFail there is probably this. external-snapshotter was vendored into the repo for exactly this reason; vendoring KubeVirt the same way is the durable fix. See /runbooks/flux-reconcile.

4. Node pressure

Heavy sustained node load — TCG-emulated VMs are the recorded example — can degrade the Cilium datapath into intermittent LB-VIP reachability problems, which looks like control-plane flapping. A restart cleared it. Cross-read /runbooks/node-io and /runbooks/node-cpu.

5. A bad operator upgrade

/runbooks/update-rollback.

6. A NetworkPolicy severed the control plane

Only if someone added homelab.keithjackson.dev/app-namespace=true to kubevirt or cdi. Those namespaces are deliberately unlabelled so Kyverno does not inject a default-deny baseline that would cut virt-handler off. Reverting that label is the fix.

When to escalate or stop

• Prioritise honestly. VM lifecycle is broken; running VMs are not. The UniFi VM keeps serving. Treat this as urgent-but-not-catastrophic unless a VM also needs to start.
• Do not drain a node to move virt pods. This cluster does not drain, and a VM live-migrates on drain only with per-VM evictionStrategy: LiveMigrate — a blanket setting would wedge drains for RWO-disk VMs.
• Do not delete a VMI or virt-launcher pod while the control plane is down: nothing will bring the VM back until virt-controller is ready again.
• Do not add the app-namespace label to kubevirt or cdi to "harden" them.
• Escalate for vendoring the remote bases or for any operator upgrade. If the alert is a false positive, fixing the rule is itself worth a change — a permanently-firing critical is a recorded, expensive failure mode here.

---

Source of truth: runbooks/kubevirt-control-plane.md. Index: /runbooks/ Records: commits b57b3e9 (this alert's wrong metric/job label) and 6a7e9d1 (false CDIOperatorDown); DECISIONS-LOG.md (KubeVirt v1.9.0 + CDI v1.66.0 on nested KVM; the remote-base 27s fetch timeout).