home-network-k3s-impl/runbooks/kubevirt-crashloop.md

kubevirt-crashloop — A VM's virt-launcher pod is restarting repeatedly

Alert: KubeVirtVMCrashLooping (warning, 10m) File: platform/kubevirt/alerts.yaml, group platform-health.kubevirt Expression: increase(kube_pod_container_status_restarts_total{pod=~"virt-launcher-.*"}[15m]) > 3

What this means here

More than three restarts of a virt-launcher pod in 15 minutes. The threshold is chosen to distinguish a crash-loop from a one-off reboot: a guest that restarts once is not interesting, a guest that restarts four times in a quarter of an hour is either panicking, being OOM-killed, or crashing qemu.

Note what layer this watches: the pod, not the guest. virt-launcher wraps qemu, so a guest kernel panic, an OOM kill of the launcher, and a qemu crash all present the same way here. The distinction comes from the logs and the events, not from the alert.

This is a warning, correctly: the VM tier is deliberately non-gating (apps-vms is wait: false) and a crash-looping VM does not affect the container platform. The one caveat is that on this network the UniFi VM is the home network's controller, so that VM crash-looping is user-visible.

Two platform facts that shape the likely causes:

• These are HP EliteDesk 800 G5 Minis with ~22 GiB usable each, running the whole platform. A VM is the largest discretionary memory consumer on a node, so OOM is a live possibility rather than a theoretical one.
• useEmulation: false is set deliberately — on KubeVirt v1.9, true silently forces TCG. A VM running under TCG burns a whole core per vCPU, and heavy sustained TCG load has been observed to degrade the Cilium datapath into intermittent LB-VIP unreachability (cleared by a restart). So "the VM is unstable and the network is flaky" is usually one cause, not two.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Which launcher, how many restarts, on which node:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get pods -A -l kubevirt.io=virt-launcher -o wide; echo; \
   kubectl --context k3s-prod get vmi -A -o wide'

Why it died — the previous container's log is the one that matters:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n unifi logs -l kubevirt.io=virt-launcher --previous --tail=120 2>&1 | tail -60'

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n unifi describe pod -l kubevirt.io=virt-launcher 2>&1 | sed -n "/Last State/,\$p" | head -40'

Was it OOM-killed?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get events -A --field-selector reason=OOMKilling --sort-by=.lastTimestamp | tail -10; echo; \
   kubectl --context k3s-prod top pods -A --sort-by=memory | head -10'

ssh k3s-prod-1 'dmesg -T | grep -iE "oom|killed process" | tail -10; free -h; swapon --show'

(An empty swapon --show is what you want. On 2026-09-07 all three prod nodes came back from a reboot with 977 MiB of swap active while the role reported success — SwapTotal == 0 is not evidence swap is off. See /runbooks/node-memory.)

Is it really TCG rather than KVM?

ssh k3s-prod-1 'ls -l /dev/kvm; grep -c -E "vmx|svm" /proc/cpuinfo; ps -eo pid,pcpu,comm | grep -i qemu'

Is the disk healthy underneath?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
     -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state | grep -v healthy'

Likely causes, most likely first

1. The guest is panicking or the guest OS is broken

Read the serial console via the previous container's log. A guest that panics at boot after a change is usually a bad cloud-init or a bad image. Two precise traps here: KubeVirt caps inline userData at 2048 bytes (the shipped payload is a deliberately minimal ~1.1 KB inline blob; userDataSecretRef was tried and reverted), and apps-vms must NOT carry postBuild.substituteFrom — Flux's envsubst rewrote ${...} inside the cloud-init and corrupted it (commit da1f833). Do not restore substitution there for consistency with the app waves.

2. The launcher is being OOM-killed

On ~22 GiB nodes running the whole platform, a VM with a generous memory request plus Prometheus (3Gi) and Grafana (768Mi) is tight. Remediation: lower the VM's memory in git, or address node memory (/runbooks/node-memory). Check swap is genuinely off first — active swap makes the kubelet's accounting lie and eviction misbehave.

3. qemu is crashing

The launcher log will carry the qemu error. Most often a device or firmware mismatch: a UEFI image needs firmware.efi, and containerDisk does not work here at all (it requires the ALPHA ImageVolume gate, off on k3s — use a CDI DataVolume). See /runbooks/kubevirt-launch-failed.

4. The disk is unavailable or degraded mid-flight

A longhorn-vm volume that went degraded or faulted takes the guest with it. Cross-read /runbooks/longhorn-degraded and /runbooks/longhorn-faulted. Remember longhorn-vm is Block mode only and migratable: true; a VM on plain longhorn cannot live-migrate.

5. Node I/O pressure is starving the guest

A guest whose virtual disk cannot complete writes will wedge and be restarted. /runbooks/node-io — and note the guest's writes land on the same partition as the embedded etcd WAL, so a thrashing VM is a control-plane risk, not just its own problem.

6. It is running under TCG

useEmulation: true forces it on v1.9. Confirm with /dev/kvm presence and qemu CPU usage. Fix in git.

When to escalate or stop

• Bound the damage first. If the VM is thrashing the disk, its restarts are loading the node that etcd writes to. If /runbooks/node-io is also firing, stopping the VM (an operator action on the VM object, spec.running: false in git) is the right move — protecting the control plane beats keeping one VM trying.
• Do not delete the DataVolume or the PVC. A re-import is hours of I/O on the shared partition.
• Do not kubectl drain the node to relocate it. This cluster does not drain, and a VM live-migrates on drain only with per-VM evictionStrategy: LiveMigrate; a blanket setting would wedge drains for RWO-disk VMs.
• Escalate if the guest needs in-guest repair — that is console work, and for the UniFi VM it is also home-network-visible.
• If the alert is firing for a virt-launcher in a leaked test namespace, deleting that namespace is the fix; leaked TCG test namespaces have caused API blips here before.

---

Source of truth: runbooks/kubevirt-crashloop.md. Index: /runbooks/ Records: DECISIONS-LOG.md Session 7 (useEmulation, firmware.efi, containerDisk/ImageVolume); commit da1f833 (cloud-init corrupted by envsubst); the 2026-09-07 swap-persistence finding.