jakman85andClaude Opus 4.8
e8c555cfeb
Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision & requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap (phases + workstreams), and AI-agent coordination (roles, protocol, definition-of-done, task board). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
80 lines
4.4 KiB
Markdown
80 lines
4.4 KiB
Markdown
# 06 — VM Workloads (IaaS via KubeVirt)
|
|
|
|
Serves: **R-PLAT-03**, **R-PLAT-04**, **R-STATE-01**. Related: [ADR-0005 (KubeVirt)](../decisions/adr-0005-vm-workloads-kubevirt.md).
|
|
|
|
## Why this exists
|
|
|
|
"Not everything fits in a container." Some workloads need a full OS, a specific kernel, non-container
|
|
software, or a lift-and-shift of an existing VM. The requirement (R-PLAT-03) is that these VMs are
|
|
managed through the **same Kubernetes API, CLI, and YAML** as containers — one control plane, one
|
|
GitOps flow, one network/storage/observability model. **KubeVirt** delivers exactly this: VMs as
|
|
Kubernetes custom resources.
|
|
|
|
## Do not confuse the two VM layers
|
|
|
|
- **Layer 1 — cluster nodes** are VMs only in dev/test (libvirt) and bare metal in prod. Managed by
|
|
OpenTofu/Ansible ([02](02-environments-and-provisioning.md)). **Not** KubeVirt.
|
|
- **Layer 2 — application VMs** run *inside* the cluster via KubeVirt and are the subject of this
|
|
spec. Managed by Kubernetes/GitOps like any other workload.
|
|
|
|
## What KubeVirt provides
|
|
|
|
- `VirtualMachine` / `VirtualMachineInstance` custom resources — declare a VM in YAML, reconcile via
|
|
Flux, manage via `kubectl` (and `virtctl` for console/start/stop).
|
|
- **CDI (Containerized Data Importer)** to import/clone disk images into **Longhorn** PVCs — VM disks
|
|
get the same replication and backup story as container volumes ([04](04-storage-and-state.md)).
|
|
- VMs sit on the **same Cilium network** — so **default-deny isolation and egress classes apply to
|
|
VMs too** (R-NET-*). A KubeVirt VM can be `egress: vpn` just like a container.
|
|
- VMs are exposed through the same **Gateway API / Service** model and monitored through the same
|
|
stack.
|
|
|
|
## The unified contract
|
|
|
|
A VM onboarded via a ServiceDefinition (`kind: vm`) gets, from the *same* declarative surface:
|
|
- Compute (vCPU/RAM), an OS image (via CDI), and Longhorn-backed disks.
|
|
- Default-deny networking + explicit allow rules + egress class (R-NET-*).
|
|
- TLS + DNS + ingress if it exposes an HTTP(S) service (R-TLS/R-DNS).
|
|
- Metrics/logs and alerting (R-OBS-*) — node-exporter-class metrics from inside the guest plus
|
|
KubeVirt/virt metrics from the platform.
|
|
- Backup/DR of its disks (R-STATE-03) and rebuildability (R-STATE-01): destroy the VM, keep the
|
|
disk PVC, recreate from the declaration.
|
|
|
|
## Update policy for VMs (R-UPD-*)
|
|
|
|
VM "updates" differ from container image tag bumps:
|
|
- **Guest OS package updates** happen *inside* the guest on a schedule (unattended-upgrades / dnf-
|
|
automatic style), configured via the VM's cloud-init/declaration — no human action.
|
|
- **VM image/template updates** (new base image) are handled as declarative changes to the VM
|
|
definition; where a golden-image pipeline exists, new images can be promoted automatically per
|
|
policy.
|
|
- The requirement "latest stable without human/AI/GitOps change" is met for guest packages via in-
|
|
guest automation; base-image rebuilds follow the same automated-promotion philosophy as containers
|
|
where feasible. Record any per-VM exceptions explicitly.
|
|
|
|
## When to choose VM vs container
|
|
|
|
Prefer containers (PaaS) by default. Choose a KubeVirt VM (IaaS) when:
|
|
- The software isn't reasonably containerizable (kernel modules, full desktop, licensing tied to a
|
|
VM, appliance images).
|
|
- Lifting an existing VM is faster/safer than re-platforming.
|
|
- Strong OS-level isolation is required beyond namespace/network policy.
|
|
|
|
For the initial catalog, all 13 named services are container-first; KubeVirt is the escape hatch and
|
|
must be **proven** in dev/test with at least one representative VM workload so the IaaS path is real,
|
|
not theoretical.
|
|
|
|
## Prerequisites & constraints
|
|
- Nodes must expose virtualization extensions (nested virt in dev/test KVMs; native on prod mini-
|
|
PCs) — captured in the node baseline ([02](02-environments-and-provisioning.md)).
|
|
- Live migration across 3 converged nodes is a nice-to-have, not a requirement; document whether it's
|
|
enabled.
|
|
- Resource pressure: VMs are heavier than pods — capacity planning must account for them,
|
|
particularly on the 3-node prod cluster.
|
|
|
|
## Acceptance gates
|
|
- Create, start, stop, and destroy a VM entirely through Kubernetes YAML + `kubectl`/`virtctl`.
|
|
- The VM's disk is Longhorn-backed; destroy/rebuild preserves data (R-STATE-01).
|
|
- Default-deny applies: the VM can reach only what its allow rules permit.
|
|
- An `egress: vpn` VM egresses via the VPN with a working kill-switch (R-NET-06).
|
|
- Guest OS auto-updates on schedule with no manual step (R-UPD-*).
|