0 Fork 0

Files

T

jakman85andClaude Opus 5.5 250fb54154 docs: bring the specification in line with the accepted implementation

The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

2026-11-08 22:46:00 -06:00

5.4 KiB

Raw Blame History

06 — VM Workloads (IaaS via KubeVirt)

Serves: R-PLAT-03, R-PLAT-04, R-STATE-01. Related: ADR-0005 (KubeVirt).

Why this exists

"Not everything fits in a container." Some workloads need a full OS, a specific kernel, non-container software, or a lift-and-shift of an existing VM. The requirement (R-PLAT-03) is that these VMs are managed through the same Kubernetes API, CLI, and YAML as containers — one control plane, one GitOps flow, one network/storage/observability model. KubeVirt delivers exactly this: VMs as Kubernetes custom resources.

Do not confuse the two VM layers

Layer 1 — cluster nodes are bare-metal mini-PCs, baselined and managed as code (02). Not KubeVirt.
Layer 2 — application VMs run inside the cluster via KubeVirt and are the subject of this spec. Managed by Kubernetes/GitOps like any other workload.

What KubeVirt provides

VirtualMachine / VirtualMachineInstance custom resources — declare a VM in YAML, reconcile via Flux, manage via kubectl (and virtctl for console/start/stop).
CDI (Containerized Data Importer) to import/clone disk images into Longhorn PVCs — VM disks get the same replication as container volumes and are backed up by default (04).
VMs sit on the same Cilium network — so default-deny isolation and egress classes apply to VMs too (R-NET-*). A KubeVirt VM can be egress: vpn just like a container.
VMs are exposed through the same Gateway API / Service model and monitored through the same stack.

The unified contract

A VM onboarded via a ServiceDefinition (kind: vm) gets, from the same declarative surface:

Compute (vCPU/RAM), an OS image (via CDI), and Longhorn-backed disks.
Default-deny networking + explicit allow rules + egress class (R-NET-*).
TLS + DNS + ingress if it exposes an HTTP(S) service (R-TLS/R-DNS).
Metrics/logs and alerting (R-OBS-*) — node-exporter-class metrics from inside the guest plus KubeVirt/virt metrics from the platform.
Backup/DR of its disks (R-STATE-03) and rebuildability (R-STATE-01): destroy the VM, keep the disk PVC, recreate from the declaration.

kind: vm is the catalog path. A VM with a custom install flow MAY instead be declared directly in git under the same controls: Flux reconciliation, default-deny networking, Longhorn disks and platform monitoring, in its own VM tier that never gates the application waves.

Recorded per-VM exception: UniFi's disk is excluded from volume backup; its Kubernetes resources are captured, and the design relies on UniFi's own configuration export for application state (04).

Update policy for VMs (R-UPD-*)

VM "updates" differ from container image tag bumps:

Guest OS package updates happen inside the guest on a schedule (unattended-upgrades / dnf- automatic style), configured via the VM's cloud-init/declaration — no human action.
VM image/template updates (new base image) are handled as declarative changes to the VM definition; where a golden-image pipeline exists, new images can be promoted automatically per policy.
The requirement "latest stable without human/AI/GitOps change" is met for guest packages via in- guest automation; base-image rebuilds follow the same automated-promotion philosophy as containers where feasible. Record any per-VM exceptions explicitly.

When to choose VM vs container

Prefer containers (PaaS) by default. Choose a KubeVirt VM (IaaS) when:

The software isn't reasonably containerizable (kernel modules, full desktop, licensing tied to a VM, appliance images).
Lifting an existing VM is faster/safer than re-platforming.
Strong OS-level isolation is required beyond namespace/network policy.

The catalog is container-first and KubeVirt is the escape hatch. UniFi is the production VM workload: UniFi OS Server is an installer that expects a full Ubuntu host and runs its own container stack and embedded datastore, so it is an appliance image in all but name. It is hand-authored in git under the rule above. The catalog's vm-demo entry is the kind: vm reference and conformance fixture and is not deployed. The IaaS path must be proven with at least one representative VM workload so it is real, not theoretical.

Prerequisites & constraints

Nodes must expose virtualization extensions (native on the mini-PCs) — captured in the node baseline (02). VMs never fall back silently to software emulation: a node without KVM fails loudly.
Live migration is enabled and opt-in per VM: the VM sets a live-migrate eviction strategy and keeps its disks on the migratable Longhorn class. There is no cluster-wide policy, because forcing it would wedge node drains for VMs on non-migratable disks. Other VMs restart on drain.
Resource pressure: VMs are heavier than pods — capacity planning must account for them, particularly on the 3-node cluster.

Acceptance gates

Create, start, stop, and destroy a VM entirely through Kubernetes YAML + kubectl/virtctl.
The VM's disk is Longhorn-backed; destroy/rebuild preserves data (R-STATE-01).
Default-deny applies: the VM can reach only what its allow rules permit.
An egress: vpn VM egresses via the VPN with a working kill-switch (R-NET-06).
Guest OS auto-updates on schedule with no manual step (R-UPD-*).

5.4 KiB Raw Blame History