Files
home-network-k3s-impl/conformance/tests/l2-behavioral/vm-lifecycle/chainsaw-test.yaml
T
jakman85andClaude Opus 4.8 3a59ef1918 fix(kubevirt): boot VMs via CDI DataVolume + EFI + real KVM (Phase 5)
VMs did not boot on the dev cluster. Three distinct live-verified causes,
all fixed GitOps-side:

1. containerDisk root failed at OCI init:
   exec "/container-disk-binary/usr/bin/container-disk": no such file.
   KubeVirt v1.9 supplies that binary to the volumerootdisk init container
   via a k8s-native `image` volume (VolumeSource.Image), gated by the ALPHA
   `ImageVolume` feature gate. k3s v1.31 has it DISABLED, so the apiserver
   silently prunes the image volume + its mount -> binary missing -> crash.
   (Proved: a server-side dry-run of a pod with an `image` volume drops it.)
   Fix: convert the VM roots in all three tests to CDI DataVolumes imported
   onto Longhorn PVCs (mounted disks bypass the container-disk/image-volume
   path). This is also the spec-required disk model (docs/06, R-STATE-01).
   vm-lifecycle now PASSES green live (VMI Running on start + restart).

2. Guest dropped to emergency mode: the containerdisks/fedora:41 image is
   UEFI-native (ESP + /boot mounted via systemd GPT auto-discovery); under
   KubeVirt's SeaBIOS default the ESP+/boot by-uuid mounts time out ->
   local-fs.target fails -> cloud-init runcmd/poweroff never runs.
   Fix: boot the VMs with EFI (OVMF, secureBoot:false) in all three tests
   and fixtures/vm.yaml. Confirmed live: emergency mode gone with EFI.

3. VMs ran under TCG software emulation (guests ~10x slow, saturating nodes
   until one went NotReady and VMs were evicted). Cause: useEmulation:true
   in the KubeVirt CR makes virt-controller NOT request devices.kubevirt.io/
   kvm, so /dev/kvm is never mounted and qemu falls back to TCG even though
   all nodes advertise kvm=1k / hypervisor=true. (Proved: `ls /dev/kvm` in a
   virt-launcher compute container = "No such file"; qemu accel = tcg.)
   Fix: useEmulation:false so the kvm device is requested and VMs run on real
   (nested) KVM. Requires Flux reconcile to take effect live (PM action).

The virt-handler clientcertificates tls.crt error is UNRELATED: node-labeller
(294 labels, hypervisor=true), device plugins, and VMI scheduling all work.

Also raised test exec timeouts to 30m (CDI import + boot) and, in
vm-default-deny, import the root BEFORE applying the default-deny NetworkPolicy
so the CDI importer's egress to quay.io is not severed by the policy.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 23:07:10 -05:00

208 lines
9.9 KiB
YAML

# =============================================================================
# L2 BEHAVIORAL — VM lifecycle (R-PLAT-03, R-PLAT-04) [RUNNABLE / self-skips]
# Catalog: docs/14 "VM lifecycle (R-PLAT-03, L2/L3)"
# Spec: docs/06-vm-workloads-kubevirt.md, ADR-0005.
#
# GIVEN a `kind: vm` workload (KubeVirt VirtualMachine),
# WHEN it is created / started / stopped / restarted / destroyed via the SAME
# Kubernetes API + kubectl surface as a container,
# THEN each transition is honoured — the VMI reaches phase=Running on start,
# disappears on stop, comes back on restart, and is gone after delete.
# (R-PLAT-03 "VMs through the K8s API"; R-PLAT-04 "create/destroy/rebuild
# on demand".) The Longhorn-disk-survives-rebuild partner proof is the L3
# vm-data-survival test; the default-deny-applies-to-the-VM proof is the L2
# vm-default-deny test — both share this VM model.
#
# RUNNABLE / SELF-SKIPPING: KubeVirt (Phase 5) may not be installed when the
# broader suite runs. This test SKIPS CLEANLY (green) until BOTH the
# virtualmachineinstances.kubevirt.io CRD and a `longhorn` StorageClass exist, then
# runs the full lifecycle the moment they do — mirroring the data-survival self-skip.
# Everything lives in the chainsaw per-test namespace and is torn down with it
# (hermetic).
#
# DISK MODEL — CDI DataVolume on Longhorn (NOT an ephemeral containerDisk):
# KubeVirt v1.9 supplies the container-disk binary to the `volumerootdisk` init
# container via a Kubernetes-native `image` volume (VolumeSource.Image) mounted at
# /container-disk-binary. That k8s feature is gated by the ALPHA `ImageVolume`
# feature gate, which is DISABLED on this k3s v1.31 cluster — so the apiserver
# silently PRUNES the image volume + its mount, the binary is never present, and a
# containerDisk VM dies at OCI init with:
# exec: "/container-disk-binary/usr/bin/container-disk": no such file or directory
# A DataVolume-backed root is a mounted Longhorn PVC (CDI imports the same Fedora
# image into it out-of-band via its importer pod) and therefore does NOT touch the
# container-disk binary / image-volume path at all — it boots on this cluster. It is
# also the disk model the SPEC requires for VMs (docs/06, R-STATE-01: Longhorn-backed
# disks that survive rebuild), so this mirrors the representative fixtures/vm.yaml.
# Cost: a one-time CDI import per run (covered by the raised exec timeout below).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vm-lifecycle
labels:
conformance.home.k3s/R-PLAT-03: "true"
conformance.home.k3s/R-PLAT-04: "true"
annotations:
conformance.home.k3s/requirements: "R-PLAT-03, R-PLAT-04"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
A VM is created/started/stopped/restarted/destroyed via the K8s API; the VMI
honours each transition (Running on start, gone on delete). Skips cleanly
until the KubeVirt CRDs exist.
spec:
# CDI imports the Fedora image onto a Longhorn PVC, then the VM boots the guest —
# import + boot is well beyond the default 180s exec window.
timeouts:
exec: 30m
steps:
- name: vm-lifecycle-through-k8s-api
try:
- script:
content: |
set -euo pipefail
NS="$NAMESPACE"
# -- Gate: skip cleanly (green) until BOTH KubeVirt and longhorn are present. --
if ! kubectl get crd virtualmachineinstances.kubevirt.io >/dev/null 2>&1; then
echo "SKIP: KubeVirt CRDs absent — vm-lifecycle not applicable on this cluster yet"
echo VM_LIFECYCLE_SKIPPED_OK
exit 0
fi
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
echo "SKIP: 'longhorn' StorageClass absent — vm-lifecycle not applicable yet"
echo VM_LIFECYCLE_SKIPPED_OK
exit 0
fi
# -- GIVEN (disk): CDI imports the Fedora image onto a Longhorn PVC (standalone
# DataVolume — outlives VM start/stop/restart so the disk is reused, not re-imported). --
kubectl -n "$NS" apply -f - <<'EOF'
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
name: lc-rootdisk
spec:
source:
registry:
url: "docker://quay.io/containerdisks/fedora:41"
storage:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 6Gi
EOF
# Wait for the CDI import to complete before booting (bounded poll; import pulls quay.io).
dv_phase=""
for i in $(seq 1 120); do
dv_phase="$(kubectl -n "$NS" get dv lc-rootdisk -o jsonpath='{.status.phase}' 2>/dev/null || true)"
[ "$dv_phase" = "Succeeded" ] && break
sleep 10
done
[ "$dv_phase" = "Succeeded" ] || { echo "FAIL: CDI import did not Succeed (last='$dv_phase')" >&2; exit 1; }
echo "CDI import complete: DataVolume lc-rootdisk Succeeded (Longhorn PVC)"
# -- GIVEN (VM): a stopped VirtualMachine bound to that Longhorn DataVolume disk. --
kubectl -n "$NS" apply -f - <<'EOF'
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: lc
spec:
running: false
template:
metadata:
labels:
kubevirt.io/vm: lc
homelab.keithjackson.dev/egress-class: internet
spec:
domain:
# EFI (OVMF) boot: the containerdisks/fedora:41 image is UEFI-native; under the
# KubeVirt SeaBIOS default it boots to emergency mode (ESP + /boot mounts fail).
# Not strictly required for this test (it only asserts VMI phase transitions, not
# in-guest state) but kept consistent with the probe tests + fixtures/vm.yaml.
firmware:
bootloader:
efi:
secureBoot: false
cpu: { cores: 1 }
memory: { guest: 1024Mi }
devices:
disks:
- name: rootdisk
disk: { bus: virtio }
interfaces:
- name: default
masquerade: {}
resources:
requests:
memory: 1024Mi
networks:
- name: default
pod: {}
volumes:
- name: rootdisk
dataVolume:
name: lc-rootdisk
EOF
# No VMI should exist while running=false.
if kubectl -n "$NS" get vmi lc >/dev/null 2>&1; then
echo "FAIL: VMI exists though the VM is defined stopped" >&2; exit 1
fi
echo "created VM 'lc' (stopped)"
wait_phase() { # $1=expected phase, bounded poll
local want="$1" i phase
for i in $(seq 1 80); do
phase="$(kubectl -n "$NS" get vmi lc -o jsonpath='{.status.phase}' 2>/dev/null || true)"
[ "$phase" = "$want" ] && return 0
sleep 10
done
echo "FAIL: VMI did not reach phase=$want (last='$phase')" >&2; return 1
}
wait_gone() {
local i
for i in $(seq 1 30); do
kubectl -n "$NS" get vmi lc >/dev/null 2>&1 || return 0
sleep 5
done
echo "FAIL: VMI still present when it should be gone" >&2; return 1
}
# -- WHEN start (running=true) THEN VMI reaches Running. --
kubectl -n "$NS" patch vm lc --type merge -p '{"spec":{"running":true}}'
wait_phase Running
echo "STARTED: VMI phase=Running"
# -- WHEN stop (running=false) THEN the VMI is torn down. --
kubectl -n "$NS" patch vm lc --type merge -p '{"spec":{"running":false}}'
wait_gone
echo "STOPPED: VMI removed"
# -- WHEN restart (running=true again) THEN it comes back Running (rebuild-on-demand). --
kubectl -n "$NS" patch vm lc --type merge -p '{"spec":{"running":true}}'
wait_phase Running
echo "RESTARTED: VMI phase=Running"
# -- WHEN destroy (delete the VM) THEN both VM and VMI are gone. --
kubectl -n "$NS" delete vm lc --wait=true --timeout=300s
wait_gone
if kubectl -n "$NS" get vm lc >/dev/null 2>&1; then
echo "FAIL: VM object still present after delete" >&2; exit 1
fi
echo "DESTROYED: VM + VMI gone"
echo VM_LIFECYCLE_OK
check:
# green on a clean skip (no KubeVirt) OR a proven full lifecycle.
(contains($stdout, 'VM_LIFECYCLE_OK') || contains($stdout, 'VM_LIFECYCLE_SKIPPED_OK')): true
finally:
- script:
content: |
kubectl -n "$NAMESPACE" delete vm lc --ignore-not-found --wait=false || true
kubectl -n "$NAMESPACE" delete dv lc-rootdisk --ignore-not-found --wait=false || true
echo "cleanup best-effort; chainsaw namespace teardown finalizes"