home-network-k3s-impl/runbooks/kubevirt-launch-failed.md

kubevirt-launch-failed — One or more VirtualMachineInstances are in the Failed phase

Alert: KubeVirtVMLaunchFailing (critical, 5m) File: platform/kubevirt/alerts.yaml, group platform-health.kubevirt Expression: sum by (namespace) (kubevirt_vmi_phase_count{phase="Failed"}) > 0

What this means here

A VMI reached the Failed phase — a scheduling, launch or qemu failure. The IaaS path is broken for that workload, which is why it is page-worthy even though the VM tier is deliberately non-gating (apps-vms is wait: false, dependsOn: platform-config, because a VM's image import and in-guest install are legitimately long-not-Ready).

What actually runs here: the unifi namespace holds the UniFi VM (rootdisk 40 GiB, Block mode on longhorn-vm). UniFi OS Server is not yet installed in the guest — that is known, documented state, not a launch failure.

Live note, measured 2026-09-13: kubevirt_vmi_phase_count returned 0 series on prod. That metric only exists while VMIs do, so this alert is currently evaluating against nothing. Confirm the real state from kubectl get vmi -A, not from this alert's silence. (kubevirt-prometheus-metrics is a live scrape job and kubevirt_virt_api_ready_status does publish, so the target is up — see /runbooks/kubevirt-control-plane.)

The three k3s-specific launch failures this platform already hit and fixed

Recorded from bring-up, and worth checking against any new VM definition because all three were non-obvious:

1. containerDisk requires the ALPHA ImageVolume feature gate, which is off on k3s. The fix was to switch to a CDI DataVolume on Longhorn instead. A VM authored with containerDisk will not launch here.
2. A UEFI image needs firmware.efi set on the VM. Without it, boot fails.
3. useEmulation: true silently forces TCG on KubeVirt v1.9 — it is set to false deliberately. A VM that appears to launch but burns a whole core per vCPU is this; it also degrades the Cilium datapath under sustained load.

Also recorded: a virt-handler certificate error seen during that work was unrelated and cosmetic — do not chase it.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Which VMI, in which namespace, and what phase:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get vmi -A -o wide; echo; \
   kubectl --context k3s-prod get vm -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,STATUS:.status.printableStatus,READY:.status.ready'

The reason — the VMI status conditions and the launcher pod's events are where it is stated:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n unifi describe vmi unifi 2>&1 | sed -n "/Status:/,\$p" | head -60'

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n unifi get pods -l kubevirt.io=virt-launcher -o wide; echo; \
   kubectl --context k3s-prod -n unifi describe pod -l kubevirt.io=virt-launcher 2>&1 | sed -n "/Events:/,\$p"'

Is the disk there? (A missing or still-importing DataVolume is the most common cause.)

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get datavolumes.cdi.kubevirt.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,PHASE:.status.phase,PROGRESS:.status.progress; echo; \
   kubectl --context k3s-prod get pvc -A | grep -Ei "rootdisk|dv|unifi"'

Is nested virtualisation actually available on the node it was scheduled to?

ssh k3s-prod-1 'ls -l /dev/kvm; grep -c -E "vmx|svm" /proc/cpuinfo'

Is the storage class right for a live-migratable VM?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get sc; echo; \
   kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy'

And whether the KubeVirt control plane can act at all:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n kubevirt get pods'

Likely causes, most likely first

1. The disk is not ready — a CDI import that has not completed or has stalled

The VM cannot start without its DataVolume. Go to /runbooks/cdi-import-stuck. The specific recorded failure: the 3-replica default class failed the UniFi 60Gi import with insufficient storage / ReplicaSchedulingFailure, which is why a dedicated single-replica longhorn-scratch class exists and is referenced by the CDI CR's scratchSpaceStorageClass.

2. The VM is authored in a way k3s does not support

The three items above: containerDisk without the ALPHA ImageVolume gate (use a CDI DataVolume), missing firmware.efi for a UEFI image, or useEmulation: true forcing TCG. All are git changes to the VM definition.

3. The cloud-init payload is too large or corrupted

Two precise traps. KubeVirt caps inline userData at 2048 bytes — the original rich cloud-init exceeded it; userDataSecretRef was tried and reverted, and the shipped state is a minimal ~1.1 KB inline payload. And: apps-vms must NOT carry postBuild.substituteFrom — Flux's envsubst rewrote ${...} inside the cloud-init and corrupted it (commit da1f833). Do not restore substitution there for consistency with the app waves. If a VM stopped booting right after someone "tidied up" the Flux Kustomizations, this is it.

4. It cannot be scheduled

No node with room, or a storage constraint. A live-migratable VM needs longhorn-vm: RWX, Block mode only (migratable RWX is a raw block device, not the share-manager/NFS path), annotated storageclass.kubevirt.io/is-default-virt-class: "true" — which is not the cluster default class. A VM pointed at plain longhorn will not live-migrate.

5. The CDI importer cannot open the raw Block device

nonroot-devices: true is set on the k3s servers because the non-root CDI importer must open a raw Block device that Kubernetes never chowns for it (commit df5979f). A permission error in the importer on a Block PVC is this.

6. The KubeVirt control plane is down

Then nothing can launch. /runbooks/kubevirt-control-plane.

When to escalate or stop

• Scope it before escalating. The VM tier is non-gating and does not affect the container platform. One failed VM is not a platform outage — but the UniFi VM is the home network's controller, so treat that one as user-visible.
• Do not delete the VM or its DataVolume to "start clean". A 60 GiB re-import is hours of I/O on disks that share a partition with the embedded etcd WAL, and that is the mechanism behind the 2026-09-07 node outage. Fix the definition, not the data.
• Do not set evictionStrategy: LiveMigrate cluster-wide. A blanket setting would wedge drains for RWO-disk VMs; it is per-VM on purpose.
• Do not add postBuild.substituteFrom to apps-vms. It corrupts cloud-init.
• Escalate for anything that needs a fresh image import during a storage incident, and for changes to the k3s server flags (nonroot-devices) — those are Ansible/operator work.

---

Source of truth: runbooks/kubevirt-launch-failed.md. Index: /runbooks/ Records: DECISIONS-LOG.md Session 7 (the three k3s-specific launch failures); commits da1f833 (cloud-init corrupted by envsubst) and df5979f (nonroot-devices); the longhorn-scratch / UniFi 60Gi import finding.