docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
1ddc14b918
commit
250fb54154
43 files changed
+1672
-575
No files matched your search
@@ -12,8 +12,8 @@ Kubernetes custom resources.
|
||||
|
||||
## Do not confuse the two VM layers
|
||||
|
||||
- **Layer 1 — cluster nodes** are VMs only in dev/test (libvirt) and bare metal in prod. Managed by
|
||||
OpenTofu/Ansible ([02](02-environments-and-provisioning.md)). **Not** KubeVirt.
|
||||
- **Layer 1 — cluster nodes** are bare-metal mini-PCs, baselined and managed as code
|
||||
([02](02-environments-and-provisioning.md)). **Not** KubeVirt.
|
||||
- **Layer 2 — application VMs** run *inside* the cluster via KubeVirt and are the subject of this
|
||||
spec. Managed by Kubernetes/GitOps like any other workload.
|
||||
|
||||
@@ -22,7 +22,8 @@ Kubernetes custom resources.
|
||||
- `VirtualMachine` / `VirtualMachineInstance` custom resources — declare a VM in YAML, reconcile via
|
||||
Flux, manage via `kubectl` (and `virtctl` for console/start/stop).
|
||||
- **CDI (Containerized Data Importer)** to import/clone disk images into **Longhorn** PVCs — VM disks
|
||||
get the same replication and backup story as container volumes ([04](04-storage-and-state.md)).
|
||||
get the same replication as container volumes and are backed up by default
|
||||
([04](04-storage-and-state.md)).
|
||||
- VMs sit on the **same Cilium network** — so **default-deny isolation and egress classes apply to
|
||||
VMs too** (R-NET-*). A KubeVirt VM can be `egress: vpn` just like a container.
|
||||
- VMs are exposed through the same **Gateway API / Service** model and monitored through the same
|
||||
@@ -39,6 +40,14 @@ A VM onboarded via a ServiceDefinition (`kind: vm`) gets, from the *same* declar
|
||||
- Backup/DR of its disks (R-STATE-03) and rebuildability (R-STATE-01): destroy the VM, keep the
|
||||
disk PVC, recreate from the declaration.
|
||||
|
||||
`kind: vm` is the catalog path. A VM with a custom install flow MAY instead be declared directly in
|
||||
git under the same controls: Flux reconciliation, default-deny networking, Longhorn disks and
|
||||
platform monitoring, in its own VM tier that never gates the application waves.
|
||||
|
||||
**Recorded per-VM exception:** UniFi's disk is excluded from volume backup; its Kubernetes resources
|
||||
are captured, and the design relies on UniFi's own configuration export for application state
|
||||
([04](04-storage-and-state.md)).
|
||||
|
||||
## Update policy for VMs (R-UPD-*)
|
||||
|
||||
VM "updates" differ from container image tag bumps:
|
||||
@@ -59,17 +68,22 @@ Prefer containers (PaaS) by default. Choose a KubeVirt VM (IaaS) when:
|
||||
- Lifting an existing VM is faster/safer than re-platforming.
|
||||
- Strong OS-level isolation is required beyond namespace/network policy.
|
||||
|
||||
For the initial catalog, all 13 named services are container-first; KubeVirt is the escape hatch and
|
||||
must be **proven** in dev/test with at least one representative VM workload so the IaaS path is real,
|
||||
not theoretical.
|
||||
The catalog is container-first and KubeVirt is the escape hatch. **UniFi is the production VM
|
||||
workload**: UniFi OS Server is an installer that expects a full Ubuntu host and runs its own container
|
||||
stack and embedded datastore, so it is an appliance image in all but name. It is hand-authored in git
|
||||
under the rule above. The catalog's `vm-demo` entry is the `kind: vm` reference and conformance
|
||||
fixture and is not deployed. The IaaS path must be **proven** with at least one representative VM
|
||||
workload so it is real, not theoretical.
|
||||
|
||||
## Prerequisites & constraints
|
||||
- Nodes must expose virtualization extensions (nested virt in dev/test KVMs; native on prod mini-
|
||||
PCs) — captured in the node baseline ([02](02-environments-and-provisioning.md)).
|
||||
- Live migration across 3 converged nodes is a nice-to-have, not a requirement; document whether it's
|
||||
enabled.
|
||||
- Nodes must expose virtualization extensions (native on the mini-PCs) — captured in the node baseline
|
||||
([02](02-environments-and-provisioning.md)). VMs never fall back silently to software emulation: a
|
||||
node without KVM fails loudly.
|
||||
- Live migration is **enabled and opt-in per VM**: the VM sets a live-migrate eviction strategy and
|
||||
keeps its disks on the migratable Longhorn class. There is no cluster-wide policy, because forcing
|
||||
it would wedge node drains for VMs on non-migratable disks. Other VMs restart on drain.
|
||||
- Resource pressure: VMs are heavier than pods — capacity planning must account for them,
|
||||
particularly on the 3-node prod cluster.
|
||||
particularly on the 3-node cluster.
|
||||
|
||||
## Acceptance gates
|
||||
- Create, start, stop, and destroy a VM entirely through Kubernetes YAML + `kubectl`/`virtctl`.
|
||||
|
||||
Reference in new issue
Block a user