jakman85andClaude Opus 5.5
250fb54154
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
94 lines
5.4 KiB
Markdown
94 lines
5.4 KiB
Markdown
# 06 — VM Workloads (IaaS via KubeVirt)
|
|
|
|
Serves: **R-PLAT-03**, **R-PLAT-04**, **R-STATE-01**. Related: [ADR-0005 (KubeVirt)](../decisions/adr-0005-vm-workloads-kubevirt.md).
|
|
|
|
## Why this exists
|
|
|
|
"Not everything fits in a container." Some workloads need a full OS, a specific kernel, non-container
|
|
software, or a lift-and-shift of an existing VM. The requirement (R-PLAT-03) is that these VMs are
|
|
managed through the **same Kubernetes API, CLI, and YAML** as containers — one control plane, one
|
|
GitOps flow, one network/storage/observability model. **KubeVirt** delivers exactly this: VMs as
|
|
Kubernetes custom resources.
|
|
|
|
## Do not confuse the two VM layers
|
|
|
|
- **Layer 1 — cluster nodes** are bare-metal mini-PCs, baselined and managed as code
|
|
([02](02-environments-and-provisioning.md)). **Not** KubeVirt.
|
|
- **Layer 2 — application VMs** run *inside* the cluster via KubeVirt and are the subject of this
|
|
spec. Managed by Kubernetes/GitOps like any other workload.
|
|
|
|
## What KubeVirt provides
|
|
|
|
- `VirtualMachine` / `VirtualMachineInstance` custom resources — declare a VM in YAML, reconcile via
|
|
Flux, manage via `kubectl` (and `virtctl` for console/start/stop).
|
|
- **CDI (Containerized Data Importer)** to import/clone disk images into **Longhorn** PVCs — VM disks
|
|
get the same replication as container volumes and are backed up by default
|
|
([04](04-storage-and-state.md)).
|
|
- VMs sit on the **same Cilium network** — so **default-deny isolation and egress classes apply to
|
|
VMs too** (R-NET-*). A KubeVirt VM can be `egress: vpn` just like a container.
|
|
- VMs are exposed through the same **Gateway API / Service** model and monitored through the same
|
|
stack.
|
|
|
|
## The unified contract
|
|
|
|
A VM onboarded via a ServiceDefinition (`kind: vm`) gets, from the *same* declarative surface:
|
|
- Compute (vCPU/RAM), an OS image (via CDI), and Longhorn-backed disks.
|
|
- Default-deny networking + explicit allow rules + egress class (R-NET-*).
|
|
- TLS + DNS + ingress if it exposes an HTTP(S) service (R-TLS/R-DNS).
|
|
- Metrics/logs and alerting (R-OBS-*) — node-exporter-class metrics from inside the guest plus
|
|
KubeVirt/virt metrics from the platform.
|
|
- Backup/DR of its disks (R-STATE-03) and rebuildability (R-STATE-01): destroy the VM, keep the
|
|
disk PVC, recreate from the declaration.
|
|
|
|
`kind: vm` is the catalog path. A VM with a custom install flow MAY instead be declared directly in
|
|
git under the same controls: Flux reconciliation, default-deny networking, Longhorn disks and
|
|
platform monitoring, in its own VM tier that never gates the application waves.
|
|
|
|
**Recorded per-VM exception:** UniFi's disk is excluded from volume backup; its Kubernetes resources
|
|
are captured, and the design relies on UniFi's own configuration export for application state
|
|
([04](04-storage-and-state.md)).
|
|
|
|
## Update policy for VMs (R-UPD-*)
|
|
|
|
VM "updates" differ from container image tag bumps:
|
|
- **Guest OS package updates** happen *inside* the guest on a schedule (unattended-upgrades / dnf-
|
|
automatic style), configured via the VM's cloud-init/declaration — no human action.
|
|
- **VM image/template updates** (new base image) are handled as declarative changes to the VM
|
|
definition; where a golden-image pipeline exists, new images can be promoted automatically per
|
|
policy.
|
|
- The requirement "latest stable without human/AI/GitOps change" is met for guest packages via in-
|
|
guest automation; base-image rebuilds follow the same automated-promotion philosophy as containers
|
|
where feasible. Record any per-VM exceptions explicitly.
|
|
|
|
## When to choose VM vs container
|
|
|
|
Prefer containers (PaaS) by default. Choose a KubeVirt VM (IaaS) when:
|
|
- The software isn't reasonably containerizable (kernel modules, full desktop, licensing tied to a
|
|
VM, appliance images).
|
|
- Lifting an existing VM is faster/safer than re-platforming.
|
|
- Strong OS-level isolation is required beyond namespace/network policy.
|
|
|
|
The catalog is container-first and KubeVirt is the escape hatch. **UniFi is the production VM
|
|
workload**: UniFi OS Server is an installer that expects a full Ubuntu host and runs its own container
|
|
stack and embedded datastore, so it is an appliance image in all but name. It is hand-authored in git
|
|
under the rule above. The catalog's `vm-demo` entry is the `kind: vm` reference and conformance
|
|
fixture and is not deployed. The IaaS path must be **proven** with at least one representative VM
|
|
workload so it is real, not theoretical.
|
|
|
|
## Prerequisites & constraints
|
|
- Nodes must expose virtualization extensions (native on the mini-PCs) — captured in the node baseline
|
|
([02](02-environments-and-provisioning.md)). VMs never fall back silently to software emulation: a
|
|
node without KVM fails loudly.
|
|
- Live migration is **enabled and opt-in per VM**: the VM sets a live-migrate eviction strategy and
|
|
keeps its disks on the migratable Longhorn class. There is no cluster-wide policy, because forcing
|
|
it would wedge node drains for VMs on non-migratable disks. Other VMs restart on drain.
|
|
- Resource pressure: VMs are heavier than pods — capacity planning must account for them,
|
|
particularly on the 3-node cluster.
|
|
|
|
## Acceptance gates
|
|
- Create, start, stop, and destroy a VM entirely through Kubernetes YAML + `kubectl`/`virtctl`.
|
|
- The VM's disk is Longhorn-backed; destroy/rebuild preserves data (R-STATE-01).
|
|
- Default-deny applies: the VM can reach only what its allow rules permit.
|
|
- An `egress: vpn` VM egresses via the VPN with a working kill-switch (R-NET-06).
|
|
- Guest OS auto-updates on schedule with no manual step (R-UPD-*).
|