jakman85andClaude Opus 4.8
3586769460
Adds the VM/CDI storage tier approved by the PM (RWX for live-migration): - platform/external-snapshotter (NEW, controllers pass): the cluster-singleton snapshot.storage.k8s.io control plane (CRDs + snapshot-controller in kube-system), pinned v8.2.0 (Kubernetes deps aligned to 1.31 since v8.1.0; this cluster is k3s v1.31.5). These CRDs were absent — only snapshots.longhorn.io and kubevirt.io snapshot CRDs existed. - platform/longhorn/volumesnapshotclass.yaml (NEW, platform-config): the driver.longhorn.io VolumeSnapshotClass (type: snap, in-cluster). - platform/kubevirt/cdi-storageprofiles.yaml (NEW, platform-config): StorageProfile longhorn -> cloneStrategy: snapshot; StorageProfile longhorn-vm -> snapshot clone + claimPropertySets RWX/Block. Non-degraded default virt class clears CDIDefaultStorageClassDegraded. - platform/longhorn/storageclasses.yaml: NEW longhorn-vm StorageClass — migratable: "true" (Longhorn's KubeVirt-live-migration knob; RWX/Block-only on 1.7.3), is-default-virt-class (NOT the k8s default; longhorn stays RWO default). - platform/kubevirt/kubevirt-cr.yaml: comment only — LiveMigration is GA on v1.9 (no feature gate); no cluster-wide evictionStrategy (would wedge RWO-disk drains); live-migration stays per-VM opt-in on a longhorn-vm disk. Co-Authored-By: Claude Opus 4.8 <[email protected]>
53 lines
3.7 KiB
YAML
53 lines
3.7 KiB
YAML
---
|
|
# KubeVirt CR — tunes the KubeVirt install (ADR-0005). CONSUMES the kubevirt.io CRD registered by the
|
|
# operator (platform/kubevirt/kustomization.yaml), so this is applied by the `platform-config` Flux
|
|
# Kustomization (dependsOn: platform) — see platform-config/kustomization.yaml + F-MON1.
|
|
#
|
|
# VIRT MODE — use real KVM (nested) on this cluster:
|
|
# developerConfiguration.useEmulation was previously `true` on the belief that it "does NOT force
|
|
# emulation, KubeVirt still prefers KVM when /dev/kvm is present." That belief is EMPIRICALLY WRONG
|
|
# on KubeVirt v1.9 (WS-VM verified live): when useEmulation=true, virt-controller does NOT add the
|
|
# `devices.kubevirt.io/kvm` resource to the virt-launcher pod, so /dev/kvm is never mounted into the
|
|
# compute container and qemu ALWAYS falls back to software emulation (TCG) — even on nodes that
|
|
# advertise `devices.kubevirt.io/kvm=1k` and are fully CPU-labelled (`hypervisor=true`). Proof: on
|
|
# the dev cluster with useEmulation=true, `ls /dev/kvm` inside a virt-launcher compute container
|
|
# returned "No such file or directory" and the qemu accelerator was `tcg`. TCG guests are ~10x slow,
|
|
# never finish the in-guest conformance probes in time, and saturate node CPUs (a node went NotReady
|
|
# and its VMs were evicted -> VMI Failed).
|
|
# * false (set here) -> virt-controller REQUIRES + requests the kvm device, /dev/kvm is mounted,
|
|
# and VMs run on real (nested) KVM. All three dev nodes have /dev/kvm
|
|
# (advertised =1k, vmx/svm present), and prod mini-PCs have native KVM, so
|
|
# this is safe cluster-wide. A node genuinely lacking /dev/kvm now fails LOUD
|
|
# (correct — we want to know, not silently run 10x-slow emulated VMs).
|
|
# * true -> the broken-on-1.9 behaviour above (silent TCG everywhere). Do NOT use while
|
|
# the nodes have KVM.
|
|
# RUNBOOK STEP 1 remains: confirm `/dev/kvm` on a node (`virt-host-validate` / `ls -l /dev/kvm`).
|
|
apiVersion: kubevirt.io/v1
|
|
kind: KubeVirt
|
|
metadata:
|
|
name: kubevirt
|
|
namespace: kubevirt
|
|
annotations:
|
|
homelab.keithjackson.dev/adr: ADR-0005
|
|
spec:
|
|
# LIVE MIGRATION (docs/06): GA in KubeVirt since v0.43 — on v1.9 (this install) it is enabled by
|
|
# default and needs NO feature gate. developerConfiguration.featureGates is intentionally left
|
|
# EMPTY: the `LiveMigration` gate no longer exists as an alpha/beta toggle, and adding it would be
|
|
# a no-op (worse, an unknown gate name can be rejected). No spec.configuration.migrations tuning is
|
|
# added either — the defaults (parallelMigrationsPerCluster=5, parallelOutboundMigrationsPerNode=2,
|
|
# 64Mi/s bandwidth cap) are correct for a 3-node dev cluster; revisit only under WS-VM load tests.
|
|
#
|
|
# A VM live-migrates on node drain ONLY if it opts in per-VM with `evictionStrategy: LiveMigrate`
|
|
# AND its disks are on a RWX-migratable StorageClass — `longhorn-vm` (platform/longhorn/
|
|
# storageclasses.yaml: migratable=true, RWX/Block). We do NOT set a cluster-wide
|
|
# spec.configuration.evictionStrategy: a blanket LiveMigrate would WEDGE drains for VMs whose disks
|
|
# are on the RWO `longhorn` class (unmigratable). VMs without opt-in still Evict+restart on drain.
|
|
configuration:
|
|
developerConfiguration:
|
|
# Use real (nested) KVM. `false` makes virt-controller REQUEST the kvm device so /dev/kvm is
|
|
# mounted into virt-launcher (KVM acceleration). `true` is broken on v1.9 -> silent TCG. (See header.)
|
|
useEmulation: false
|
|
# Persistent EFI/TPM VM state (if a VM enables it) lands on the replicated Longhorn tier (ADR-0003),
|
|
# inheriting the same durability as every other stateful volume.
|
|
vmStateStorageClass: longhorn
|