jakman85andClaude Opus 4.8
3a59ef1918
VMs did not boot on the dev cluster. Three distinct live-verified causes, all fixed GitOps-side: 1. containerDisk root failed at OCI init: exec "/container-disk-binary/usr/bin/container-disk": no such file. KubeVirt v1.9 supplies that binary to the volumerootdisk init container via a k8s-native `image` volume (VolumeSource.Image), gated by the ALPHA `ImageVolume` feature gate. k3s v1.31 has it DISABLED, so the apiserver silently prunes the image volume + its mount -> binary missing -> crash. (Proved: a server-side dry-run of a pod with an `image` volume drops it.) Fix: convert the VM roots in all three tests to CDI DataVolumes imported onto Longhorn PVCs (mounted disks bypass the container-disk/image-volume path). This is also the spec-required disk model (docs/06, R-STATE-01). vm-lifecycle now PASSES green live (VMI Running on start + restart). 2. Guest dropped to emergency mode: the containerdisks/fedora:41 image is UEFI-native (ESP + /boot mounted via systemd GPT auto-discovery); under KubeVirt's SeaBIOS default the ESP+/boot by-uuid mounts time out -> local-fs.target fails -> cloud-init runcmd/poweroff never runs. Fix: boot the VMs with EFI (OVMF, secureBoot:false) in all three tests and fixtures/vm.yaml. Confirmed live: emergency mode gone with EFI. 3. VMs ran under TCG software emulation (guests ~10x slow, saturating nodes until one went NotReady and VMs were evicted). Cause: useEmulation:true in the KubeVirt CR makes virt-controller NOT request devices.kubevirt.io/ kvm, so /dev/kvm is never mounted and qemu falls back to TCG even though all nodes advertise kvm=1k / hypervisor=true. (Proved: `ls /dev/kvm` in a virt-launcher compute container = "No such file"; qemu accel = tcg.) Fix: useEmulation:false so the kvm device is requested and VMs run on real (nested) KVM. Requires Flux reconcile to take effect live (PM action). The virt-handler clientcertificates tls.crt error is UNRELATED: node-labeller (294 labels, hypervisor=true), device plugins, and VMI scheduling all work. Also raised test exec timeouts to 30m (CDI import + boot) and, in vm-default-deny, import the root BEFORE applying the default-deny NetworkPolicy so the CDI importer's egress to quay.io is not severed by the policy. Co-Authored-By: Claude Opus 4.8 <[email protected]>
43 lines
2.8 KiB
YAML
43 lines
2.8 KiB
YAML
---
|
|
# KubeVirt CR — tunes the KubeVirt install (ADR-0005). CONSUMES the kubevirt.io CRD registered by the
|
|
# operator (platform/kubevirt/kustomization.yaml), so this is applied by the `platform-config` Flux
|
|
# Kustomization (dependsOn: platform) — see platform-config/kustomization.yaml + F-MON1.
|
|
#
|
|
# VIRT MODE — use real KVM (nested) on this cluster:
|
|
# developerConfiguration.useEmulation was previously `true` on the belief that it "does NOT force
|
|
# emulation, KubeVirt still prefers KVM when /dev/kvm is present." That belief is EMPIRICALLY WRONG
|
|
# on KubeVirt v1.9 (WS-VM verified live): when useEmulation=true, virt-controller does NOT add the
|
|
# `devices.kubevirt.io/kvm` resource to the virt-launcher pod, so /dev/kvm is never mounted into the
|
|
# compute container and qemu ALWAYS falls back to software emulation (TCG) — even on nodes that
|
|
# advertise `devices.kubevirt.io/kvm=1k` and are fully CPU-labelled (`hypervisor=true`). Proof: on
|
|
# the dev cluster with useEmulation=true, `ls /dev/kvm` inside a virt-launcher compute container
|
|
# returned "No such file or directory" and the qemu accelerator was `tcg`. TCG guests are ~10x slow,
|
|
# never finish the in-guest conformance probes in time, and saturate node CPUs (a node went NotReady
|
|
# and its VMs were evicted -> VMI Failed).
|
|
# * false (set here) -> virt-controller REQUIRES + requests the kvm device, /dev/kvm is mounted,
|
|
# and VMs run on real (nested) KVM. All three dev nodes have /dev/kvm
|
|
# (advertised =1k, vmx/svm present), and prod mini-PCs have native KVM, so
|
|
# this is safe cluster-wide. A node genuinely lacking /dev/kvm now fails LOUD
|
|
# (correct — we want to know, not silently run 10x-slow emulated VMs).
|
|
# * true -> the broken-on-1.9 behaviour above (silent TCG everywhere). Do NOT use while
|
|
# the nodes have KVM.
|
|
# RUNBOOK STEP 1 remains: confirm `/dev/kvm` on a node (`virt-host-validate` / `ls -l /dev/kvm`).
|
|
apiVersion: kubevirt.io/v1
|
|
kind: KubeVirt
|
|
metadata:
|
|
name: kubevirt
|
|
namespace: kubevirt
|
|
annotations:
|
|
homelab.keithjackson.dev/adr: ADR-0005
|
|
spec:
|
|
# Drain VMs off a cordoned node when possible; software-emulated dev VMs just Evict+restart.
|
|
# (Live migration across the 3 converged nodes is a nice-to-have per docs/06, not required.)
|
|
configuration:
|
|
developerConfiguration:
|
|
# Use real (nested) KVM. `false` makes virt-controller REQUEST the kvm device so /dev/kvm is
|
|
# mounted into virt-launcher (KVM acceleration). `true` is broken on v1.9 -> silent TCG. (See header.)
|
|
useEmulation: false
|
|
# Persistent EFI/TPM VM state (if a VM enables it) lands on the replicated Longhorn tier (ADR-0003),
|
|
# inheriting the same durability as every other stateful volume.
|
|
vmStateStorageClass: longhorn
|