fix(kubevirt): boot VMs via CDI DataVolume + EFI + real KVM (Phase 5)

VMs did not boot on the dev cluster. Three distinct live-verified causes,
all fixed GitOps-side:

1. containerDisk root failed at OCI init:
   exec "/container-disk-binary/usr/bin/container-disk": no such file.
   KubeVirt v1.9 supplies that binary to the volumerootdisk init container
   via a k8s-native `image` volume (VolumeSource.Image), gated by the ALPHA
   `ImageVolume` feature gate. k3s v1.31 has it DISABLED, so the apiserver
   silently prunes the image volume + its mount -> binary missing -> crash.
   (Proved: a server-side dry-run of a pod with an `image` volume drops it.)
   Fix: convert the VM roots in all three tests to CDI DataVolumes imported
   onto Longhorn PVCs (mounted disks bypass the container-disk/image-volume
   path). This is also the spec-required disk model (docs/06, R-STATE-01).
   vm-lifecycle now PASSES green live (VMI Running on start + restart).

2. Guest dropped to emergency mode: the containerdisks/fedora:41 image is
   UEFI-native (ESP + /boot mounted via systemd GPT auto-discovery); under
   KubeVirt's SeaBIOS default the ESP+/boot by-uuid mounts time out ->
   local-fs.target fails -> cloud-init runcmd/poweroff never runs.
   Fix: boot the VMs with EFI (OVMF, secureBoot:false) in all three tests
   and fixtures/vm.yaml. Confirmed live: emergency mode gone with EFI.

3. VMs ran under TCG software emulation (guests ~10x slow, saturating nodes
   until one went NotReady and VMs were evicted). Cause: useEmulation:true
   in the KubeVirt CR makes virt-controller NOT request devices.kubevirt.io/
   kvm, so /dev/kvm is never mounted and qemu falls back to TCG even though
   all nodes advertise kvm=1k / hypervisor=true. (Proved: `ls /dev/kvm` in a
   virt-launcher compute container = "No such file"; qemu accel = tcg.)
   Fix: useEmulation:false so the kvm device is requested and VMs run on real
   (nested) KVM. Requires Flux reconcile to take effect live (PM action).

The virt-handler clientcertificates tls.crt error is UNRELATED: node-labeller
(294 labels, hypervisor=true), device plugins, and VMI scheduling all work.

Also raised test exec timeouts to 30m (CDI import + boot) and, in
vm-default-deny, import the root BEFORE applying the default-deny NetworkPolicy
so the CDI importer's egress to quay.io is not severed by the policy.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-24 23:07:10 -05:00
1 parent 392f458d94
commit 3a59ef1918
5 files changed
+208 -44

No files matched your search

@@ -10,9 +10,15 @@
# exactly the container proof (data-survival), now for a KubeVirt VM
# (R-STATE-01 covers PaaS + IaaS).
#
# METHOD (virtctl-free, filesystem-agnostic, deterministic): a containerDisk-root
# VM (ephemeral) with a Longhorn BLOCK data disk. cloud-init writes a unique canary
# to the raw block device and powers the guest off; runStrategy RerunOnFailure means
# METHOD (virtctl-free, filesystem-agnostic, deterministic): a VM whose ROOT is a
# CDI DataVolume on Longhorn (a mounted PVC — NOT an ephemeral containerDisk; the
# containerDisk boot path is broken on this k3s cluster because KubeVirt v1.9 serves
# the container-disk binary through a k8s `image` volume gated by the ALPHA
# `ImageVolume` feature gate, which k3s v1.31 has disabled → the apiserver prunes the
# volume and the guest dies at OCI init. DataVolume disks bypass that path and are the
# spec-required disk model anyway — docs/06/R-STATE-01). The VM also has a SECOND
# Longhorn BLOCK data disk; cloud-init writes a unique canary to that raw block device
# and powers the guest off; runStrategy RerunOnFailure means
# a clean shutdown STAYS down (VMI -> Succeeded), so the write is known-complete and
# the disk is released. We then DELETE the VM (keeping the PVC) and bind a throwaway
# busybox reader pod to the SAME Longhorn PVC as a raw block device — if the bytes
@@ -38,9 +44,10 @@ metadata:
Destroy a VM entirely, keep its Longhorn PVC, and a fresh workload re-bound to
that PVC still reads the VM-written data. Skips cleanly until KubeVirt + longhorn exist.
spec:
# Boot a guest + run cloud-init + power off, then reattach a Longhorn RWO volume — far beyond 180s.
# CDI import onto Longhorn + boot a guest + run cloud-init + power off, then reattach a
# Longhorn RWO volume — far beyond 180s.
timeouts:
exec: 20m
exec: 30m
steps:
- name: vm-data-survives-destroy-and-rebuild
try:
@@ -77,7 +84,35 @@ spec:
storage: 1Gi
EOF
# -- The VM: containerDisk root (ephemeral) + the Longhorn block disk (/dev/vdb) + a
# -- The VM's ROOT disk: CDI imports Fedora onto a Longhorn PVC (bypasses the broken
# containerDisk/image-volume path; standalone DataVolume). Wait for the import. --
kubectl -n "$NS" apply -f - <<'EOF'
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
name: survivor-rootdisk
spec:
source:
registry:
url: "docker://quay.io/containerdisks/fedora:41"
storage:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 6Gi
EOF
dv_phase=""
for i in $(seq 1 120); do
dv_phase="$(kubectl -n "$NS" get dv survivor-rootdisk -o jsonpath='{.status.phase}' 2>/dev/null || true)"
[ "$dv_phase" = "Succeeded" ] && break
sleep 10
done
[ "$dv_phase" = "Succeeded" ] || { echo "FAIL: CDI import did not Succeed (last='$dv_phase')" >&2; exit 1; }
echo "CDI import complete: survivor-rootdisk Succeeded (Longhorn PVC)"
# -- The VM: DataVolume root (Longhorn) + the Longhorn block disk (/dev/vdb) + a
# cloud-init disk. cloud-init writes the canary to the raw device and powers off.
# RerunOnFailure => a clean poweroff STAYS Succeeded (not restarted), so the write is
# known-complete and the disk released. Only $CANARY expands (no other '$' in the body).
@@ -95,6 +130,14 @@ spec:
homelab.keithjackson.dev/egress-class: internet
spec:
domain:
# The containerdisks/fedora:41 image is UEFI-native (ESP + /boot mounted via
# systemd GPT auto-discovery). KubeVirt defaults to SeaBIOS, under which the guest
# boots to emergency mode (ESP + /boot by-uuid mounts time out) and cloud-init's
# runcmd/poweroff never runs. Boot with EFI (OVMF). secureBoot:false => no SMM needed.
firmware:
bootloader:
efi:
secureBoot: false
cpu: { cores: 1 }
memory: { guest: 1024Mi }
devices:
@@ -110,8 +153,8 @@ spec:
memory: 1024Mi
volumes:
- name: rootdisk
containerDisk:
image: quay.io/containerdisks/fedora:41
dataVolume:
name: survivor-rootdisk
- name: datadisk
persistentVolumeClaim:
claimName: vm-survivor
@@ -168,5 +211,6 @@ spec:
content: |
kubectl -n "$NAMESPACE" delete vm survivor --ignore-not-found --wait=false || true
kubectl -n "$NAMESPACE" delete pod reader --ignore-not-found --wait=false || true
kubectl -n "$NAMESPACE" delete dv survivor-rootdisk --ignore-not-found --wait=false || true
kubectl -n "$NAMESPACE" delete pvc vm-survivor --ignore-not-found --wait=false || true
echo "cleanup best-effort; chainsaw namespace teardown finalizes"