Initial baseline: home platform (k3s + Longhorn) specification repository

Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision &
requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap
(phases + workstreams), and AI-agent coordination (roles, protocol,
definition-of-done, task board).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

docs/02-environments-and-provisioning.md

@@ -0,0 +1,93 @@
+# 02 — Environments & Provisioning
+
+Serves: **R-ENV-01/02/03**, **R-PLAT-01**. Related: [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md).
+
+## Two environments, one configuration
+
+| Aspect | dev/test | prod |
+|---|---|---|
+| Substrate | libvirt/KVM VMs on the operator workstation | 3× mini-PC bare metal |
+| Node count | 3 VMs (mirror prod topology) | 3 physical nodes |
+| Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd (k3s server on all 3) |
+| Storage | virtual disks presented to Longhorn | local SSD/NVMe presented to Longhorn |
+| Status | **active build target** | **design-only until roadmap flips it** |
+
+**R-ENV-02 is the golden constraint:** the GitOps configuration is authored once. Environments
+differ only through scoped overrides (Flux `Kustomization` overlays / cluster-scoped variables):
+node counts, disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits.
+Never fork the base config per environment.
+
+## Infrastructure provisioning (Layer 1 — the cluster nodes)
+
+> These are the machines that *run* k3s. Distinct from KubeVirt application VMs (Layer 2).
+
+### Toolchain
+- **OpenTofu** (Terraform-compatible) with the **libvirt** provider defines dev/test node VMs as
+  code: CPU, RAM, disks (a dedicated data disk for Longhorn), NICs, and a **cloud-init** seed.
+- **cloud-init** performs first-boot baseline: hostname, users, SSH keys, package baseline, kernel
+  modules/sysctls required by k3s/Longhorn/KubeVirt, and marks the node ready for k3s.
+- **Ansible** performs cluster bootstrap: installs k3s in HA mode, joins nodes, lays down cluster-
+  wide prerequisites (Cilium is installed as the CNI, not flannel), and hands off to Flux.
+- For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation step differs
+  (bare-metal image/cloud-init via USB/PXE instead of libvirt).
+
+### Node lifecycle stages (R-ENV-01 — "created, baselined, cluster-joined, managed as code")
+1. **Create** — OpenTofu instantiates the VM (dev) / operator images the mini-PC (prod).
+2. **Baseline** — cloud-init applies OS baseline and prerequisites.
+3. **Cluster-join** — Ansible installs/joins k3s with Cilium as CNI; disables k3s bundled
+   flannel/traefik/servicelb/kube-proxy as required by the Cilium ADR.
+4. **Manage** — from here the cluster is GitOps-managed by Flux; node config changes are made via
+   IaC and re-applied, never hand-edited on the box.
+
+### k3s topology decisions
+- HA control plane with **embedded etcd** across all 3 nodes (no external datastore).
+- Bundled components that conflict with our stack are **disabled at install**: flannel (→ Cilium),
+  kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb where
+  a different LB strategy is chosen. Exact flags live in the Ansible bootstrap; the *intent* is
+  fixed here and in ADR-0001/0002.
+- Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not
+  isolate control-plane nodes.
+
+### Bare-metal specifics (prod, design-only for now)
+- Each mini-PC contributes local NVMe/SSD to Longhorn; document the disk layout (OS vs. Longhorn
+  data device) before prod build.
+- BIOS/firmware, virtualization extensions (for KubeVirt), and wake/boot behavior are captured as a
+  provisioning checklist under the prod-readiness workstream.
+- Edge networking (multi-WAN, VPN uplink) is largely an **edge-router** concern — see
+  [`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md) and OQ-1.
+
+## Environment promotion model
+
+```
+   git repo (single source of truth)
+        │
+   base configuration  ──────────────┐
+        │                            │
+   overlay: dev/test            overlay: prod
+        │                            │
+   Flux on dev cluster          Flux on prod cluster (later)
+```
+
+- A change proves out in **dev/test** against the acceptance gates ([`AGENTS.md` §7](../AGENTS.md)).
+- Promotion to prod is a change to the prod overlay only (no base edits), gated by the roadmap's
+  prod-readiness phase.
+- **Never develop against prod.** Prod hardware is a target, not a sandbox (R-ENV-03).
+
+## Statelessness of the nodes themselves
+
+Nodes are cattle. A node can be destroyed and rebuilt from IaC + cloud-init + Ansible; cluster and
+application state lives on **Longhorn** (replicated) and in **git** (desired state), so node loss is
+a re-provision, not a data-loss event (supports R-STATE-01/02).
+
+## Deliverables for this spec (tracked on the task board)
+- OpenTofu module for dev/test node VMs (libvirt) with a dedicated Longhorn data disk.
+- cloud-init baseline template (users, SSH, sysctls, kernel modules for Longhorn/KubeVirt).
+- Ansible role(s): k3s HA install with Cilium, component disabling, Flux bootstrap handoff.
+- Documented prod bare-metal provisioning checklist (design-only; do not execute).
+- Proof: destroy a dev node VM and rebuild it; cluster re-converges with no data loss.
+
+## Open questions
+- **OQ-1** (edge router / WAN count) — blocks final multi-WAN design, not node provisioning.
+- Do we want **Cluster API** later to unify dev VM and prod bare-metal lifecycle? Deferred; OpenTofu
+  + Ansible is sufficient and lower-complexity for 3 nodes now. Revisit via ADR if prod scaling
+  demands it.