fix(kubevirt): boot VMs via CDI DataVolume + EFI + real KVM (Phase 5)
VMs did not boot on the dev cluster. Three distinct live-verified causes, all fixed GitOps-side: 1. containerDisk root failed at OCI init: exec "/container-disk-binary/usr/bin/container-disk": no such file. KubeVirt v1.9 supplies that binary to the volumerootdisk init container via a k8s-native `image` volume (VolumeSource.Image), gated by the ALPHA `ImageVolume` feature gate. k3s v1.31 has it DISABLED, so the apiserver silently prunes the image volume + its mount -> binary missing -> crash. (Proved: a server-side dry-run of a pod with an `image` volume drops it.) Fix: convert the VM roots in all three tests to CDI DataVolumes imported onto Longhorn PVCs (mounted disks bypass the container-disk/image-volume path). This is also the spec-required disk model (docs/06, R-STATE-01). vm-lifecycle now PASSES green live (VMI Running on start + restart). 2. Guest dropped to emergency mode: the containerdisks/fedora:41 image is UEFI-native (ESP + /boot mounted via systemd GPT auto-discovery); under KubeVirt's SeaBIOS default the ESP+/boot by-uuid mounts time out -> local-fs.target fails -> cloud-init runcmd/poweroff never runs. Fix: boot the VMs with EFI (OVMF, secureBoot:false) in all three tests and fixtures/vm.yaml. Confirmed live: emergency mode gone with EFI. 3. VMs ran under TCG software emulation (guests ~10x slow, saturating nodes until one went NotReady and VMs were evicted). Cause: useEmulation:true in the KubeVirt CR makes virt-controller NOT request devices.kubevirt.io/ kvm, so /dev/kvm is never mounted and qemu falls back to TCG even though all nodes advertise kvm=1k / hypervisor=true. (Proved: `ls /dev/kvm` in a virt-launcher compute container = "No such file"; qemu accel = tcg.) Fix: useEmulation:false so the kvm device is requested and VMs run on real (nested) KVM. Requires Flux reconcile to take effect live (PM action). The virt-handler clientcertificates tls.crt error is UNRELATED: node-labeller (294 labels, hypervisor=true), device plugins, and VMI scheduling all work. Also raised test exec timeouts to 30m (CDI import + boot) and, in vm-default-deny, import the root BEFORE applying the default-deny NetworkPolicy so the CDI importer's egress to quay.io is not severed by the policy. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed
1 parent
392f458d94
commit
3a59ef1918
5 files changed
+208
-44
No files matched your search
@@ -12,9 +12,14 @@
|
||||
# own datapath (its traffic egresses via the virt-launcher pod netns, so a
|
||||
# podSelector:{} NetworkPolicy covers it). R-NET-01 for IaaS.
|
||||
#
|
||||
# METHOD (virtctl-free): cloud-init runs the two probes and writes a PASS/FAIL
|
||||
# verdict to a raw Longhorn block disk, then powers off (RerunOnFailure => stays
|
||||
# down). A busybox reader pod reads the verdict back. The container default-deny
|
||||
# METHOD (virtctl-free): the VM's ROOT is a CDI DataVolume on Longhorn (a mounted
|
||||
# PVC — NOT an ephemeral containerDisk, whose boot path is broken on this k3s cluster:
|
||||
# KubeVirt v1.9 serves the container-disk binary via a k8s `image` volume gated by the
|
||||
# ALPHA `ImageVolume` feature gate, disabled on k3s v1.31, so the apiserver prunes it
|
||||
# and the guest dies at OCI init. DataVolume roots bypass that path and are the
|
||||
# spec-required disk model — docs/06). cloud-init runs the two probes and writes a
|
||||
# PASS/FAIL verdict to a raw Longhorn block disk, then powers off (RerunOnFailure =>
|
||||
# stays down). A busybox reader pod reads the verdict back. The container default-deny
|
||||
# test proves the datapath fail-closed (default-deny-failclosed, L3); this asserts
|
||||
# the SAME baseline is enforced for a VM.
|
||||
#
|
||||
@@ -41,7 +46,7 @@ metadata:
|
||||
an undeclared public TCP dial is blocked. Skips cleanly until KubeVirt + longhorn exist.
|
||||
spec:
|
||||
timeouts:
|
||||
exec: 20m
|
||||
exec: 30m
|
||||
steps:
|
||||
- name: default-deny-applies-to-the-vm-datapath
|
||||
try:
|
||||
@@ -60,8 +65,42 @@ spec:
|
||||
|
||||
CANARY="vm-deny-$(date +%s%N)"
|
||||
|
||||
# -- GIVEN (disk): CDI imports the Fedora root onto a Longhorn PVC. This MUST happen
|
||||
# BEFORE the default-deny NetworkPolicy below: default-deny (podSelector {}) also
|
||||
# matches the CDI importer pod and would sever its egress to quay.io. The import is
|
||||
# a one-time provisioning step; once the PVC holds the disk the VM boots with NO
|
||||
# external egress, so applying default-deny AFTER the import still proves the VM's
|
||||
# own datapath is fail-closed (the runbook's app-namespace equivalent is a scoped
|
||||
# CiliumNetworkPolicy for the importer — see fixtures/vm.yaml). --
|
||||
kubectl -n "$NS" apply -f - <<'EOF'
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
kind: DataVolume
|
||||
metadata:
|
||||
name: prober-rootdisk
|
||||
spec:
|
||||
source:
|
||||
registry:
|
||||
url: "docker://quay.io/containerdisks/fedora:41"
|
||||
storage:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 6Gi
|
||||
EOF
|
||||
dv_phase=""
|
||||
for i in $(seq 1 120); do
|
||||
dv_phase="$(kubectl -n "$NS" get dv prober-rootdisk -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
[ "$dv_phase" = "Succeeded" ] && break
|
||||
sleep 10
|
||||
done
|
||||
[ "$dv_phase" = "Succeeded" ] || { echo "FAIL: CDI import did not Succeed (last='$dv_phase')" >&2; exit 1; }
|
||||
echo "CDI import complete: prober-rootdisk Succeeded (Longhorn PVC)"
|
||||
|
||||
# -- GIVEN: the default-deny baseline in this namespace (deny all, allow only DNS to
|
||||
# kube-system) — the exact shape Kyverno generates for app namespaces (docs/03). --
|
||||
# kube-system) — the exact shape Kyverno generates for app namespaces (docs/03).
|
||||
# Applied AFTER the import so it governs the running VM's datapath (the point of the test). --
|
||||
kubectl -n "$NS" apply -f - <<'EOF'
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
@@ -111,6 +150,15 @@ spec:
|
||||
homelab.keithjackson.dev/egress-class: internet
|
||||
spec:
|
||||
domain:
|
||||
# The containerdisks/fedora:41 image is UEFI-native (ESP + /boot mounted via
|
||||
# systemd GPT auto-discovery). KubeVirt defaults to SeaBIOS, under which the guest
|
||||
# boots to emergency mode (ESP + /boot by-uuid mounts time out -> local-fs.target
|
||||
# fails) and cloud-init's runcmd/poweroff never runs. Boot with EFI (OVMF) so the
|
||||
# guest reaches multi-user and the probe executes. secureBoot:false => no SMM needed.
|
||||
firmware:
|
||||
bootloader:
|
||||
efi:
|
||||
secureBoot: false
|
||||
cpu: { cores: 1 }
|
||||
memory: { guest: 1024Mi }
|
||||
devices:
|
||||
@@ -126,8 +174,8 @@ spec:
|
||||
memory: 1024Mi
|
||||
volumes:
|
||||
- name: rootdisk
|
||||
containerDisk:
|
||||
image: quay.io/containerdisks/fedora:41
|
||||
dataVolume:
|
||||
name: prober-rootdisk
|
||||
- name: verdict
|
||||
persistentVolumeClaim:
|
||||
claimName: vm-verdict
|
||||
@@ -185,5 +233,6 @@ spec:
|
||||
content: |
|
||||
kubectl -n "$NAMESPACE" delete vm prober --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete pod verdict-reader --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete dv prober-rootdisk --ignore-not-found --wait=false || true
|
||||
kubectl -n "$NAMESPACE" delete pvc vm-verdict --ignore-not-found --wait=false || true
|
||||
echo "cleanup best-effort; chainsaw namespace teardown finalizes"
|
||||
Reference in new issue
Block a user