# kubevirt-crashloop — A VM's virt-launcher pod is restarting repeatedly **Alert:** `KubeVirtVMCrashLooping` (warning, 10m) **File:** `platform/kubevirt/alerts.yaml`, group `platform-health.kubevirt` **Expression:** `increase(kube_pod_container_status_restarts_total{pod=~"virt-launcher-.*"}[15m]) > 3` ## What this means here More than three restarts of a `virt-launcher` pod in 15 minutes. The threshold is chosen to distinguish a crash-**loop** from a one-off reboot: a guest that restarts once is not interesting, a guest that restarts four times in a quarter of an hour is either panicking, being OOM-killed, or crashing qemu. Note what layer this watches: the **pod**, not the guest. `virt-launcher` wraps qemu, so a guest kernel panic, an OOM kill of the launcher, and a qemu crash all present the same way here. The distinction comes from the logs and the events, not from the alert. This is a `warning`, correctly: the VM tier is deliberately non-gating (`apps-vms` is `wait: false`) and a crash-looping VM does not affect the container platform. The one caveat is that on this network the UniFi VM is the home network's controller, so *that* VM crash-looping is user-visible. Two platform facts that shape the likely causes: - These are **HP EliteDesk 800 G5 Minis with ~22 GiB usable each**, running the whole platform. A VM is the largest discretionary memory consumer on a node, so OOM is a live possibility rather than a theoretical one. - **`useEmulation: false` is set deliberately** — on KubeVirt v1.9, `true` silently forces TCG. A VM running under TCG burns a whole core per vCPU, and heavy sustained TCG load has been observed to degrade the Cilium datapath into intermittent LB-VIP unreachability (cleared by a restart). So "the VM is unstable **and** the network is flaky" is usually one cause, not two. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Which launcher, how many restarts, on which node: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get pods -A -l kubevirt.io=virt-launcher -o wide; echo; \ kubectl --context k3s-prod get vmi -A -o wide' ``` Why it died — the previous container's log is the one that matters: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n unifi logs -l kubevirt.io=virt-launcher --previous --tail=120 2>&1 | tail -60' ``` ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n unifi describe pod -l kubevirt.io=virt-launcher 2>&1 | sed -n "/Last State/,\$p" | head -40' ``` Was it OOM-killed? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get events -A --field-selector reason=OOMKilling --sort-by=.lastTimestamp | tail -10; echo; \ kubectl --context k3s-prod top pods -A --sort-by=memory | head -10' ``` ``` ssh k3s-prod-1 'dmesg -T | grep -iE "oom|killed process" | tail -10; free -h; swapon --show' ``` (An empty `swapon --show` is what you want. On 2026-09-07 all three prod nodes came back from a reboot with 977 MiB of swap **active** while the role reported success — `SwapTotal == 0` is not evidence swap is off. See `/runbooks/node-memory`.) Is it really TCG rather than KVM? ``` ssh k3s-prod-1 'ls -l /dev/kvm; grep -c -E "vmx|svm" /proc/cpuinfo; ps -eo pid,pcpu,comm | grep -i qemu' ``` Is the disk healthy underneath? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \ -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state | grep -v healthy' ``` ## Likely causes, most likely first ### 1. The guest is panicking or the guest OS is broken Read the serial console via the previous container's log. A guest that panics at boot after a change is usually a bad cloud-init or a bad image. Two precise traps here: **KubeVirt caps inline `userData` at 2048 bytes** (the shipped payload is a deliberately minimal ~1.1 KB inline blob; `userDataSecretRef` was tried and reverted), and **`apps-vms` must NOT carry `postBuild.substituteFrom`** — Flux's envsubst rewrote `${...}` inside the cloud-init and corrupted it (commit `da1f833`). Do not restore substitution there for consistency with the app waves. ### 2. The launcher is being OOM-killed On ~22 GiB nodes running the whole platform, a VM with a generous memory request plus Prometheus (3Gi) and Grafana (768Mi) is tight. Remediation: lower the VM's memory in git, or address node memory (`/runbooks/node-memory`). Check swap is genuinely off first — active swap makes the kubelet's accounting lie and eviction misbehave. ### 3. qemu is crashing The launcher log will carry the qemu error. Most often a device or firmware mismatch: a UEFI image needs `firmware.efi`, and `containerDisk` does not work here at all (it requires the ALPHA `ImageVolume` gate, off on k3s — use a CDI DataVolume). See `/runbooks/kubevirt-launch-failed`. ### 4. The disk is unavailable or degraded mid-flight A `longhorn-vm` volume that went degraded or faulted takes the guest with it. Cross-read `/runbooks/longhorn-degraded` and `/runbooks/longhorn-faulted`. Remember `longhorn-vm` is **Block mode only** and `migratable: true`; a VM on plain `longhorn` cannot live-migrate. ### 5. Node I/O pressure is starving the guest A guest whose virtual disk cannot complete writes will wedge and be restarted. `/runbooks/node-io` — and note the guest's writes land on the same partition as the embedded etcd WAL, so a thrashing VM is a control-plane risk, not just its own problem. ### 6. It is running under TCG `useEmulation: true` forces it on v1.9. Confirm with `/dev/kvm` presence and qemu CPU usage. Fix in git. ## When to escalate or stop - **Bound the damage first.** If the VM is thrashing the disk, its restarts are loading the node that etcd writes to. If `/runbooks/node-io` is also firing, stopping the VM (an operator action on the `VM` object, `spec.running: false` in git) is the right move — protecting the control plane beats keeping one VM trying. - **Do not delete the DataVolume or the PVC.** A re-import is hours of I/O on the shared partition. - **Do not `kubectl drain`** the node to relocate it. This cluster does not drain, and a VM live-migrates on drain **only** with per-VM `evictionStrategy: LiveMigrate`; a blanket setting would wedge drains for RWO-disk VMs. - Escalate if the guest needs in-guest repair — that is console work, and for the UniFi VM it is also home-network-visible. - If the alert is firing for a `virt-launcher` in a leaked test namespace, deleting that namespace is the fix; leaked TCG test namespaces have caused API blips here before. --- Source of truth: `runbooks/kubevirt-crashloop.md`. Index: `/runbooks/` Records: `DECISIONS-LOG.md` Session 7 (`useEmulation`, `firmware.efi`, `containerDisk`/ImageVolume); commit `da1f833` (cloud-init corrupted by envsubst); the 2026-09-07 swap-persistence finding.