VMs did not boot on the dev cluster. Three distinct live-verified causes, all fixed GitOps-side: 1. containerDisk root failed at OCI init: exec "/container-disk-binary/usr/bin/container-disk": no such file. KubeVirt v1.9 supplies that binary to the volumerootdisk init container via a k8s-native `image` volume (VolumeSource.Image), gated by the ALPHA `ImageVolume` feature gate. k3s v1.31 has it DISABLED, so the apiserver silently prunes the image volume + its mount -> binary missing -> crash. (Proved: a server-side dry-run of a pod with an `image` volume drops it.) Fix: convert the VM roots in all three tests to CDI DataVolumes imported onto Longhorn PVCs (mounted disks bypass the container-disk/image-volume path). This is also the spec-required disk model (docs/06, R-STATE-01). vm-lifecycle now PASSES green live (VMI Running on start + restart). 2. Guest dropped to emergency mode: the containerdisks/fedora:41 image is UEFI-native (ESP + /boot mounted via systemd GPT auto-discovery); under KubeVirt's SeaBIOS default the ESP+/boot by-uuid mounts time out -> local-fs.target fails -> cloud-init runcmd/poweroff never runs. Fix: boot the VMs with EFI (OVMF, secureBoot:false) in all three tests and fixtures/vm.yaml. Confirmed live: emergency mode gone with EFI. 3. VMs ran under TCG software emulation (guests ~10x slow, saturating nodes until one went NotReady and VMs were evicted). Cause: useEmulation:true in the KubeVirt CR makes virt-controller NOT request devices.kubevirt.io/ kvm, so /dev/kvm is never mounted and qemu falls back to TCG even though all nodes advertise kvm=1k / hypervisor=true. (Proved: `ls /dev/kvm` in a virt-launcher compute container = "No such file"; qemu accel = tcg.) Fix: useEmulation:false so the kvm device is requested and VMs run on real (nested) KVM. Requires Flux reconcile to take effect live (PM action). The virt-handler clientcertificates tls.crt error is UNRELATED: node-labeller (294 labels, hypervisor=true), device plugins, and VMI scheduling all work. Also raised test exec timeouts to 30m (CDI import + boot) and, in vm-default-deny, import the root BEFORE applying the default-deny NetworkPolicy so the CDI importer's egress to quay.io is not severed by the policy. Co-Authored-By: Claude Opus 4.8 <[email protected]>
208 lines
9.9 KiB
YAML
208 lines
9.9 KiB
YAML
# =============================================================================
|
|
# L2 BEHAVIORAL — VM lifecycle (R-PLAT-03, R-PLAT-04) [RUNNABLE / self-skips]
|
|
# Catalog: docs/14 "VM lifecycle (R-PLAT-03, L2/L3)"
|
|
# Spec: docs/06-vm-workloads-kubevirt.md, ADR-0005.
|
|
#
|
|
# GIVEN a `kind: vm` workload (KubeVirt VirtualMachine),
|
|
# WHEN it is created / started / stopped / restarted / destroyed via the SAME
|
|
# Kubernetes API + kubectl surface as a container,
|
|
# THEN each transition is honoured — the VMI reaches phase=Running on start,
|
|
# disappears on stop, comes back on restart, and is gone after delete.
|
|
# (R-PLAT-03 "VMs through the K8s API"; R-PLAT-04 "create/destroy/rebuild
|
|
# on demand".) The Longhorn-disk-survives-rebuild partner proof is the L3
|
|
# vm-data-survival test; the default-deny-applies-to-the-VM proof is the L2
|
|
# vm-default-deny test — both share this VM model.
|
|
#
|
|
# RUNNABLE / SELF-SKIPPING: KubeVirt (Phase 5) may not be installed when the
|
|
# broader suite runs. This test SKIPS CLEANLY (green) until BOTH the
|
|
# virtualmachineinstances.kubevirt.io CRD and a `longhorn` StorageClass exist, then
|
|
# runs the full lifecycle the moment they do — mirroring the data-survival self-skip.
|
|
# Everything lives in the chainsaw per-test namespace and is torn down with it
|
|
# (hermetic).
|
|
#
|
|
# DISK MODEL — CDI DataVolume on Longhorn (NOT an ephemeral containerDisk):
|
|
# KubeVirt v1.9 supplies the container-disk binary to the `volumerootdisk` init
|
|
# container via a Kubernetes-native `image` volume (VolumeSource.Image) mounted at
|
|
# /container-disk-binary. That k8s feature is gated by the ALPHA `ImageVolume`
|
|
# feature gate, which is DISABLED on this k3s v1.31 cluster — so the apiserver
|
|
# silently PRUNES the image volume + its mount, the binary is never present, and a
|
|
# containerDisk VM dies at OCI init with:
|
|
# exec: "/container-disk-binary/usr/bin/container-disk": no such file or directory
|
|
# A DataVolume-backed root is a mounted Longhorn PVC (CDI imports the same Fedora
|
|
# image into it out-of-band via its importer pod) and therefore does NOT touch the
|
|
# container-disk binary / image-volume path at all — it boots on this cluster. It is
|
|
# also the disk model the SPEC requires for VMs (docs/06, R-STATE-01: Longhorn-backed
|
|
# disks that survive rebuild), so this mirrors the representative fixtures/vm.yaml.
|
|
# Cost: a one-time CDI import per run (covered by the raised exec timeout below).
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: vm-lifecycle
|
|
labels:
|
|
conformance.home.k3s/R-PLAT-03: "true"
|
|
conformance.home.k3s/R-PLAT-04: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-PLAT-03, R-PLAT-04"
|
|
conformance.home.k3s/layer: "L2"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
A VM is created/started/stopped/restarted/destroyed via the K8s API; the VMI
|
|
honours each transition (Running on start, gone on delete). Skips cleanly
|
|
until the KubeVirt CRDs exist.
|
|
spec:
|
|
# CDI imports the Fedora image onto a Longhorn PVC, then the VM boots the guest —
|
|
# import + boot is well beyond the default 180s exec window.
|
|
timeouts:
|
|
exec: 30m
|
|
steps:
|
|
- name: vm-lifecycle-through-k8s-api
|
|
try:
|
|
- script:
|
|
content: |
|
|
set -euo pipefail
|
|
NS="$NAMESPACE"
|
|
|
|
# -- Gate: skip cleanly (green) until BOTH KubeVirt and longhorn are present. --
|
|
if ! kubectl get crd virtualmachineinstances.kubevirt.io >/dev/null 2>&1; then
|
|
echo "SKIP: KubeVirt CRDs absent — vm-lifecycle not applicable on this cluster yet"
|
|
echo VM_LIFECYCLE_SKIPPED_OK
|
|
exit 0
|
|
fi
|
|
if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
|
|
echo "SKIP: 'longhorn' StorageClass absent — vm-lifecycle not applicable yet"
|
|
echo VM_LIFECYCLE_SKIPPED_OK
|
|
exit 0
|
|
fi
|
|
|
|
# -- GIVEN (disk): CDI imports the Fedora image onto a Longhorn PVC (standalone
|
|
# DataVolume — outlives VM start/stop/restart so the disk is reused, not re-imported). --
|
|
kubectl -n "$NS" apply -f - <<'EOF'
|
|
apiVersion: cdi.kubevirt.io/v1beta1
|
|
kind: DataVolume
|
|
metadata:
|
|
name: lc-rootdisk
|
|
spec:
|
|
source:
|
|
registry:
|
|
url: "docker://quay.io/containerdisks/fedora:41"
|
|
storage:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
storageClassName: longhorn
|
|
resources:
|
|
requests:
|
|
storage: 6Gi
|
|
EOF
|
|
# Wait for the CDI import to complete before booting (bounded poll; import pulls quay.io).
|
|
dv_phase=""
|
|
for i in $(seq 1 120); do
|
|
dv_phase="$(kubectl -n "$NS" get dv lc-rootdisk -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
|
[ "$dv_phase" = "Succeeded" ] && break
|
|
sleep 10
|
|
done
|
|
[ "$dv_phase" = "Succeeded" ] || { echo "FAIL: CDI import did not Succeed (last='$dv_phase')" >&2; exit 1; }
|
|
echo "CDI import complete: DataVolume lc-rootdisk Succeeded (Longhorn PVC)"
|
|
|
|
# -- GIVEN (VM): a stopped VirtualMachine bound to that Longhorn DataVolume disk. --
|
|
kubectl -n "$NS" apply -f - <<'EOF'
|
|
apiVersion: kubevirt.io/v1
|
|
kind: VirtualMachine
|
|
metadata:
|
|
name: lc
|
|
spec:
|
|
running: false
|
|
template:
|
|
metadata:
|
|
labels:
|
|
kubevirt.io/vm: lc
|
|
homelab.keithjackson.dev/egress-class: internet
|
|
spec:
|
|
domain:
|
|
# EFI (OVMF) boot: the containerdisks/fedora:41 image is UEFI-native; under the
|
|
# KubeVirt SeaBIOS default it boots to emergency mode (ESP + /boot mounts fail).
|
|
# Not strictly required for this test (it only asserts VMI phase transitions, not
|
|
# in-guest state) but kept consistent with the probe tests + fixtures/vm.yaml.
|
|
firmware:
|
|
bootloader:
|
|
efi:
|
|
secureBoot: false
|
|
cpu: { cores: 1 }
|
|
memory: { guest: 1024Mi }
|
|
devices:
|
|
disks:
|
|
- name: rootdisk
|
|
disk: { bus: virtio }
|
|
interfaces:
|
|
- name: default
|
|
masquerade: {}
|
|
resources:
|
|
requests:
|
|
memory: 1024Mi
|
|
networks:
|
|
- name: default
|
|
pod: {}
|
|
volumes:
|
|
- name: rootdisk
|
|
dataVolume:
|
|
name: lc-rootdisk
|
|
EOF
|
|
# No VMI should exist while running=false.
|
|
if kubectl -n "$NS" get vmi lc >/dev/null 2>&1; then
|
|
echo "FAIL: VMI exists though the VM is defined stopped" >&2; exit 1
|
|
fi
|
|
echo "created VM 'lc' (stopped)"
|
|
|
|
wait_phase() { # $1=expected phase, bounded poll
|
|
local want="$1" i phase
|
|
for i in $(seq 1 80); do
|
|
phase="$(kubectl -n "$NS" get vmi lc -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
|
[ "$phase" = "$want" ] && return 0
|
|
sleep 10
|
|
done
|
|
echo "FAIL: VMI did not reach phase=$want (last='$phase')" >&2; return 1
|
|
}
|
|
wait_gone() {
|
|
local i
|
|
for i in $(seq 1 30); do
|
|
kubectl -n "$NS" get vmi lc >/dev/null 2>&1 || return 0
|
|
sleep 5
|
|
done
|
|
echo "FAIL: VMI still present when it should be gone" >&2; return 1
|
|
}
|
|
|
|
# -- WHEN start (running=true) THEN VMI reaches Running. --
|
|
kubectl -n "$NS" patch vm lc --type merge -p '{"spec":{"running":true}}'
|
|
wait_phase Running
|
|
echo "STARTED: VMI phase=Running"
|
|
|
|
# -- WHEN stop (running=false) THEN the VMI is torn down. --
|
|
kubectl -n "$NS" patch vm lc --type merge -p '{"spec":{"running":false}}'
|
|
wait_gone
|
|
echo "STOPPED: VMI removed"
|
|
|
|
# -- WHEN restart (running=true again) THEN it comes back Running (rebuild-on-demand). --
|
|
kubectl -n "$NS" patch vm lc --type merge -p '{"spec":{"running":true}}'
|
|
wait_phase Running
|
|
echo "RESTARTED: VMI phase=Running"
|
|
|
|
# -- WHEN destroy (delete the VM) THEN both VM and VMI are gone. --
|
|
kubectl -n "$NS" delete vm lc --wait=true --timeout=300s
|
|
wait_gone
|
|
if kubectl -n "$NS" get vm lc >/dev/null 2>&1; then
|
|
echo "FAIL: VM object still present after delete" >&2; exit 1
|
|
fi
|
|
echo "DESTROYED: VM + VMI gone"
|
|
|
|
echo VM_LIFECYCLE_OK
|
|
check:
|
|
# green on a clean skip (no KubeVirt) OR a proven full lifecycle.
|
|
(contains($stdout, 'VM_LIFECYCLE_OK') || contains($stdout, 'VM_LIFECYCLE_SKIPPED_OK')): true
|
|
finally:
|
|
- script:
|
|
content: |
|
|
kubectl -n "$NAMESPACE" delete vm lc --ignore-not-found --wait=false || true
|
|
kubectl -n "$NAMESPACE" delete dv lc-rootdisk --ignore-not-found --wait=false || true
|
|
echo "cleanup best-effort; chainsaw namespace teardown finalizes"
|