docs: bring the specification in line with the accepted implementation

The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-11-08 22:46:00 -06:00
1 parent 1ddc14b918
commit 250fb54154
43 files changed
+1672 -575

No files matched your search

+25 -11
View File
@@ -12,8 +12,8 @@ Kubernetes custom resources.
## Do not confuse the two VM layers
- **Layer 1 — cluster nodes** are VMs only in dev/test (libvirt) and bare metal in prod. Managed by
OpenTofu/Ansible ([02](02-environments-and-provisioning.md)). **Not** KubeVirt.
- **Layer 1 — cluster nodes** are bare-metal mini-PCs, baselined and managed as code
([02](02-environments-and-provisioning.md)). **Not** KubeVirt.
- **Layer 2 — application VMs** run *inside* the cluster via KubeVirt and are the subject of this
spec. Managed by Kubernetes/GitOps like any other workload.
@@ -22,7 +22,8 @@ Kubernetes custom resources.
- `VirtualMachine` / `VirtualMachineInstance` custom resources — declare a VM in YAML, reconcile via
Flux, manage via `kubectl` (and `virtctl` for console/start/stop).
- **CDI (Containerized Data Importer)** to import/clone disk images into **Longhorn** PVCs — VM disks
get the same replication and backup story as container volumes ([04](04-storage-and-state.md)).
get the same replication as container volumes and are backed up by default
([04](04-storage-and-state.md)).
- VMs sit on the **same Cilium network** — so **default-deny isolation and egress classes apply to
VMs too** (R-NET-*). A KubeVirt VM can be `egress: vpn` just like a container.
- VMs are exposed through the same **Gateway API / Service** model and monitored through the same
@@ -39,6 +40,14 @@ A VM onboarded via a ServiceDefinition (`kind: vm`) gets, from the *same* declar
- Backup/DR of its disks (R-STATE-03) and rebuildability (R-STATE-01): destroy the VM, keep the
disk PVC, recreate from the declaration.
`kind: vm` is the catalog path. A VM with a custom install flow MAY instead be declared directly in
git under the same controls: Flux reconciliation, default-deny networking, Longhorn disks and
platform monitoring, in its own VM tier that never gates the application waves.
**Recorded per-VM exception:** UniFi's disk is excluded from volume backup; its Kubernetes resources
are captured, and the design relies on UniFi's own configuration export for application state
([04](04-storage-and-state.md)).
## Update policy for VMs (R-UPD-*)
VM "updates" differ from container image tag bumps:
@@ -59,17 +68,22 @@ Prefer containers (PaaS) by default. Choose a KubeVirt VM (IaaS) when:
- Lifting an existing VM is faster/safer than re-platforming.
- Strong OS-level isolation is required beyond namespace/network policy.
For the initial catalog, all 13 named services are container-first; KubeVirt is the escape hatch and
must be **proven** in dev/test with at least one representative VM workload so the IaaS path is real,
not theoretical.
The catalog is container-first and KubeVirt is the escape hatch. **UniFi is the production VM
workload**: UniFi OS Server is an installer that expects a full Ubuntu host and runs its own container
stack and embedded datastore, so it is an appliance image in all but name. It is hand-authored in git
under the rule above. The catalog's `vm-demo` entry is the `kind: vm` reference and conformance
fixture and is not deployed. The IaaS path must be **proven** with at least one representative VM
workload so it is real, not theoretical.
## Prerequisites & constraints
- Nodes must expose virtualization extensions (nested virt in dev/test KVMs; native on prod mini-
PCs) — captured in the node baseline ([02](02-environments-and-provisioning.md)).
- Live migration across 3 converged nodes is a nice-to-have, not a requirement; document whether it's
enabled.
- Nodes must expose virtualization extensions (native on the mini-PCs) — captured in the node baseline
([02](02-environments-and-provisioning.md)). VMs never fall back silently to software emulation: a
node without KVM fails loudly.
- Live migration is **enabled and opt-in per VM**: the VM sets a live-migrate eviction strategy and
keeps its disks on the migratable Longhorn class. There is no cluster-wide policy, because forcing
it would wedge node drains for VMs on non-migratable disks. Other VMs restart on drain.
- Resource pressure: VMs are heavier than pods — capacity planning must account for them,
particularly on the 3-node prod cluster.
particularly on the 3-node cluster.
## Acceptance gates
- Create, start, stop, and destroy a VM entirely through Kubernetes YAML + `kubectl`/`virtctl`.