jakman85andClaude Opus 5.5
250fb54154
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
5.4 KiB
06 — VM Workloads (IaaS via KubeVirt)
Serves: R-PLAT-03, R-PLAT-04, R-STATE-01. Related: ADR-0005 (KubeVirt).
Why this exists
"Not everything fits in a container." Some workloads need a full OS, a specific kernel, non-container software, or a lift-and-shift of an existing VM. The requirement (R-PLAT-03) is that these VMs are managed through the same Kubernetes API, CLI, and YAML as containers — one control plane, one GitOps flow, one network/storage/observability model. KubeVirt delivers exactly this: VMs as Kubernetes custom resources.
Do not confuse the two VM layers
- Layer 1 — cluster nodes are bare-metal mini-PCs, baselined and managed as code (02). Not KubeVirt.
- Layer 2 — application VMs run inside the cluster via KubeVirt and are the subject of this spec. Managed by Kubernetes/GitOps like any other workload.
What KubeVirt provides
VirtualMachine/VirtualMachineInstancecustom resources — declare a VM in YAML, reconcile via Flux, manage viakubectl(andvirtctlfor console/start/stop).- CDI (Containerized Data Importer) to import/clone disk images into Longhorn PVCs — VM disks get the same replication as container volumes and are backed up by default (04).
- VMs sit on the same Cilium network — so default-deny isolation and egress classes apply to
VMs too (R-NET-*). A KubeVirt VM can be
egress: vpnjust like a container. - VMs are exposed through the same Gateway API / Service model and monitored through the same stack.
The unified contract
A VM onboarded via a ServiceDefinition (kind: vm) gets, from the same declarative surface:
- Compute (vCPU/RAM), an OS image (via CDI), and Longhorn-backed disks.
- Default-deny networking + explicit allow rules + egress class (R-NET-*).
- TLS + DNS + ingress if it exposes an HTTP(S) service (R-TLS/R-DNS).
- Metrics/logs and alerting (R-OBS-*) — node-exporter-class metrics from inside the guest plus KubeVirt/virt metrics from the platform.
- Backup/DR of its disks (R-STATE-03) and rebuildability (R-STATE-01): destroy the VM, keep the disk PVC, recreate from the declaration.
kind: vm is the catalog path. A VM with a custom install flow MAY instead be declared directly in
git under the same controls: Flux reconciliation, default-deny networking, Longhorn disks and
platform monitoring, in its own VM tier that never gates the application waves.
Recorded per-VM exception: UniFi's disk is excluded from volume backup; its Kubernetes resources are captured, and the design relies on UniFi's own configuration export for application state (04).
Update policy for VMs (R-UPD-*)
VM "updates" differ from container image tag bumps:
- Guest OS package updates happen inside the guest on a schedule (unattended-upgrades / dnf- automatic style), configured via the VM's cloud-init/declaration — no human action.
- VM image/template updates (new base image) are handled as declarative changes to the VM definition; where a golden-image pipeline exists, new images can be promoted automatically per policy.
- The requirement "latest stable without human/AI/GitOps change" is met for guest packages via in- guest automation; base-image rebuilds follow the same automated-promotion philosophy as containers where feasible. Record any per-VM exceptions explicitly.
When to choose VM vs container
Prefer containers (PaaS) by default. Choose a KubeVirt VM (IaaS) when:
- The software isn't reasonably containerizable (kernel modules, full desktop, licensing tied to a VM, appliance images).
- Lifting an existing VM is faster/safer than re-platforming.
- Strong OS-level isolation is required beyond namespace/network policy.
The catalog is container-first and KubeVirt is the escape hatch. UniFi is the production VM
workload: UniFi OS Server is an installer that expects a full Ubuntu host and runs its own container
stack and embedded datastore, so it is an appliance image in all but name. It is hand-authored in git
under the rule above. The catalog's vm-demo entry is the kind: vm reference and conformance
fixture and is not deployed. The IaaS path must be proven with at least one representative VM
workload so it is real, not theoretical.
Prerequisites & constraints
- Nodes must expose virtualization extensions (native on the mini-PCs) — captured in the node baseline (02). VMs never fall back silently to software emulation: a node without KVM fails loudly.
- Live migration is enabled and opt-in per VM: the VM sets a live-migrate eviction strategy and keeps its disks on the migratable Longhorn class. There is no cluster-wide policy, because forcing it would wedge node drains for VMs on non-migratable disks. Other VMs restart on drain.
- Resource pressure: VMs are heavier than pods — capacity planning must account for them, particularly on the 3-node cluster.
Acceptance gates
- Create, start, stop, and destroy a VM entirely through Kubernetes YAML +
kubectl/virtctl. - The VM's disk is Longhorn-backed; destroy/rebuild preserves data (R-STATE-01).
- Default-deny applies: the VM can reach only what its allow rules permit.
- An
egress: vpnVM egresses via the VPN with a working kill-switch (R-NET-06). - Guest OS auto-updates on schedule with no manual step (R-UPD-*).