home-network-k3s-impl/runbooks/kubevirt-control-plane.md

# kubevirt-control-plane — The KubeVirt control plane is degraded

**Alert:** `KubeVirtControlPlaneDown` (critical, 10m)
**File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt`
**Expression:**
```
(sum(kubevirt_virt_controller_ready_status) == 0)
or (sum(kubevirt_virt_api_ready_status) == 0)
or (absent(up{job="kubevirt-prometheus-metrics"} == 1))
```

## What this means here

No `virt-controller` or no `virt-api` replica reports ready, so **VM lifecycle operations — create, start,
stop, migrate — will fail** (docs/06). Note carefully what this does *not* mean: **already-running VMs
keep running.** `virt-launcher` pods and their qemu processes are independent of the control plane. So
this is an "I cannot manage VMs" alert, not "the VMs are down".

The `absent()` clause also fires if the whole KubeVirt metrics target disappears. The scrape job label is
**`kubevirt-prometheus-metrics`** (the Service name), **not** `kubevirt` — that precision is a fix: this
very alert once used a non-existent metric with the wrong job label and fired falsely (commit `b57b3e9`).
There is a sibling precedent too: a broken HTTP CDI ServiceMonitor caused a false `CDIOperatorDown`
(commit `6a7e9d1`). **So the first question with this alert is always whether it is telling the truth.**

Live state, measured 2026-09-13: `kubevirt-prometheus-metrics` **is** among Prometheus's 19 scrape jobs
and `kubevirt_virt_api_ready_status` returns **2 series**, so the first two clauses are evaluable and
quiet. (`kubevirt_vmi_phase_count` was absent at the same moment — that metric only exists when VMIs do,
which affects `/runbooks/kubevirt-launch-failed`, not this rule.)

Platform context: **KubeVirt v1.9.0 + CDI v1.66.0**, on real nested KVM (`/dev/kvm` present, vmx/svm
verified; the Proxmox nodes use `cpu_type=host`). The `kubevirt` and `cdi` namespaces deliberately carry
**no `app-namespace` label**, so Kyverno does not generate the default-deny baseline into them — it would
sever the virt control plane. Do not "fix" that by adding the label.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

Is the control plane actually unhealthy, or is this a metrics artefact?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n kubevirt get pods -o wide; echo; \
   kubectl --context k3s-prod get kubevirts.kubevirt.io -A \
     -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,READY:.status.conditions[?\(@.type==\"Available\"\)].status'
```

Are the VMs themselves still running? (This is the reassuring half.)

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get vmi -A -o wide; echo; \
   kubectl --context k3s-prod get vm -A'
```

Is the scrape target up, and do the two metrics exist?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   for q in "up{job=\"kubevirt-prometheus-metrics\"}" "sum(kubevirt_virt_controller_ready_status)" "sum(kubevirt_virt_api_ready_status)"; do \
     printf "%-55s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \
       | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done'
```

Why a component is unready:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n kubevirt logs deploy/virt-controller --tail=80 2>&1 | tail -30; echo ====; \
   kubectl --context k3s-prod -n kubevirt logs deploy/virt-api --tail=80 2>&1 | tail -30; echo ====; \
   kubectl --context k3s-prod -n kubevirt get events --sort-by=.lastTimestamp | tail -20'
```

Is the Flux tier that owns KubeVirt healthy?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get kustomizations.kustomize.toolkit.fluxcd.io -A \
     -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message'
```

## Likely causes, most likely first

### 1. The metrics target, not the control plane
Check the pods first. If `virt-api` and `virt-controller` are Running and Ready, this is a monitoring
fault — and there is direct precedent for false positives on this exact alert (`b57b3e9`) and its CDI
sibling (`6a7e9d1`). Remediation: fix the ServiceMonitor/metric in git. Cross-read
`/runbooks/targets-down`.

### 2. A node that hosted the control-plane pods went away
`virt-controller` and `virt-api` are ordinary Deployments. Go to `/runbooks/node-down`.

### 3. The KubeVirt Flux tier failed to reconcile
Specific known hazard: **`platform/kubevirt` still uses remote kustomize bases**, and the
kustomize-controller's git fetch has a **27s timeout** that remote GitHub bases flake against — a
`BuildFail` there is probably this. `external-snapshotter` was vendored into the repo for exactly this
reason; vendoring KubeVirt the same way is the durable fix. See `/runbooks/flux-reconcile`.

### 4. Node pressure
Heavy sustained node load — TCG-emulated VMs are the recorded example — can degrade the Cilium datapath
into intermittent LB-VIP reachability problems, which looks like control-plane flapping. A restart
cleared it. Cross-read `/runbooks/node-io` and `/runbooks/node-cpu`.

### 5. A bad operator upgrade
`/runbooks/update-rollback`.

### 6. A NetworkPolicy severed the control plane
Only if someone added `homelab.keithjackson.dev/app-namespace=true` to `kubevirt` or `cdi`. Those
namespaces are deliberately unlabelled so Kyverno does not inject a default-deny baseline that would cut
virt-handler off. Reverting that label is the fix.

## When to escalate or stop

- **Prioritise honestly.** VM *lifecycle* is broken; running VMs are not. The UniFi VM keeps serving.
  Treat this as urgent-but-not-catastrophic unless a VM also needs to start.
- **Do not drain a node to move virt pods.** This cluster does not drain, and a VM live-migrates on drain
  **only** with per-VM `evictionStrategy: LiveMigrate` — a blanket setting would wedge drains for
  RWO-disk VMs.
- **Do not delete a VMI or `virt-launcher` pod** while the control plane is down: nothing will bring the
  VM back until `virt-controller` is ready again.
- **Do not add the `app-namespace` label to `kubevirt` or `cdi`** to "harden" them.
- Escalate for vendoring the remote bases or for any operator upgrade. If the alert is a false positive,
  fixing the rule is itself worth a change — a permanently-firing critical is a recorded, expensive
  failure mode here.

---
Source of truth: `runbooks/kubevirt-control-plane.md`. Index: `/runbooks/`
Records: commits `b57b3e9` (this alert's wrong metric/job label) and `6a7e9d1` (false `CDIOperatorDown`);
`DECISIONS-LOG.md` (KubeVirt v1.9.0 + CDI v1.66.0 on nested KVM; the remote-base 27s fetch timeout).