home-network-k3s/docs/06-vm-workloads-kubevirt.md

06 — VM Workloads (IaaS via KubeVirt)

Serves: R-PLAT-03, R-PLAT-04, R-STATE-01. Related: ADR-0005 (KubeVirt).

Why this exists

"Not everything fits in a container." Some workloads need a full OS, a specific kernel, non-container software, or a lift-and-shift of an existing VM. The requirement (R-PLAT-03) is that these VMs are managed through the same Kubernetes API, CLI, and YAML as containers — one control plane, one GitOps flow, one network/storage/observability model. KubeVirt delivers exactly this: VMs as Kubernetes custom resources.

Do not confuse the two VM layers

• Layer 1 — cluster nodes are VMs only in dev/test (libvirt) and bare metal in prod. Managed by OpenTofu/Ansible (02). Not KubeVirt.
• Layer 2 — application VMs run inside the cluster via KubeVirt and are the subject of this spec. Managed by Kubernetes/GitOps like any other workload.

What KubeVirt provides

• VirtualMachine / VirtualMachineInstance custom resources — declare a VM in YAML, reconcile via Flux, manage via kubectl (and virtctl for console/start/stop).
• CDI (Containerized Data Importer) to import/clone disk images into Longhorn PVCs — VM disks get the same replication and backup story as container volumes (04).
• VMs sit on the same Cilium network — so default-deny isolation and egress classes apply to VMs too (R-NET-*). A KubeVirt VM can be egress: vpn just like a container.
• VMs are exposed through the same Gateway API / Service model and monitored through the same stack.

The unified contract

A VM onboarded via a ServiceDefinition (kind: vm) gets, from the same declarative surface:

• Compute (vCPU/RAM), an OS image (via CDI), and Longhorn-backed disks.
• Default-deny networking + explicit allow rules + egress class (R-NET-*).
• TLS + DNS + ingress if it exposes an HTTP(S) service (R-TLS/R-DNS).
• Metrics/logs and alerting (R-OBS-*) — node-exporter-class metrics from inside the guest plus KubeVirt/virt metrics from the platform.
• Backup/DR of its disks (R-STATE-03) and rebuildability (R-STATE-01): destroy the VM, keep the disk PVC, recreate from the declaration.

Update policy for VMs (R-UPD-*)

VM "updates" differ from container image tag bumps:

• Guest OS package updates happen inside the guest on a schedule (unattended-upgrades / dnf- automatic style), configured via the VM's cloud-init/declaration — no human action.
• VM image/template updates (new base image) are handled as declarative changes to the VM definition; where a golden-image pipeline exists, new images can be promoted automatically per policy.
• The requirement "latest stable without human/AI/GitOps change" is met for guest packages via in- guest automation; base-image rebuilds follow the same automated-promotion philosophy as containers where feasible. Record any per-VM exceptions explicitly.

When to choose VM vs container

Prefer containers (PaaS) by default. Choose a KubeVirt VM (IaaS) when:

• The software isn't reasonably containerizable (kernel modules, full desktop, licensing tied to a VM, appliance images).
• Lifting an existing VM is faster/safer than re-platforming.
• Strong OS-level isolation is required beyond namespace/network policy.

For the initial catalog, all 13 named services are container-first; KubeVirt is the escape hatch and must be proven in dev/test with at least one representative VM workload so the IaaS path is real, not theoretical.

Prerequisites & constraints

• Nodes must expose virtualization extensions (nested virt in dev/test KVMs; native on prod mini- PCs) — captured in the node baseline (02).
• Live migration across 3 converged nodes is a nice-to-have, not a requirement; document whether it's enabled.
• Resource pressure: VMs are heavier than pods — capacity planning must account for them, particularly on the 3-node prod cluster.

Acceptance gates

• Create, start, stop, and destroy a VM entirely through Kubernetes YAML + kubectl/virtctl.
• The VM's disk is Longhorn-backed; destroy/rebuild preserves data (R-STATE-01).
• Default-deny applies: the VM can reach only what its allow rules permit.
• An egress: vpn VM egresses via the VPN with a working kill-switch (R-NET-06).
• Guest OS auto-updates on schedule with no manual step (R-UPD-*).