# 02 — Environments & Provisioning

Serves: **R-ENV-01/02/03**, **R-PLAT-01**. Related: [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md).

## Two environments, one configuration

| Aspect | dev/test | prod |
|---|---|---|
| Substrate | libvirt/KVM VMs on the operator workstation | 3× mini-PC bare metal |
| Node count | 3 VMs (mirror prod topology) | 3 physical nodes |
| Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd (k3s server on all 3) |
| Storage | virtual disks presented to Longhorn | local SSD/NVMe presented to Longhorn |
| Status | **active build target** | **design-only until roadmap flips it** |

**R-ENV-02 is the golden constraint:** the GitOps configuration is authored once. Environments
differ only through scoped overrides (Flux `Kustomization` overlays / cluster-scoped variables):
node counts, disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits.
Never fork the base config per environment.

## Infrastructure provisioning (Layer 1 — the cluster nodes)

> These are the machines that *run* k3s. Distinct from KubeVirt application VMs (Layer 2).

### Toolchain
- **OpenTofu** (Terraform-compatible) with the **libvirt** provider defines dev/test node VMs as
  code: CPU, RAM, disks (a dedicated data disk for Longhorn), NICs, and a **cloud-init** seed.
- **cloud-init** performs first-boot baseline: hostname, users, SSH keys, package baseline, kernel
  modules/sysctls required by k3s/Longhorn/KubeVirt, and marks the node ready for k3s.
- **Ansible** performs cluster bootstrap: installs k3s in HA mode, joins nodes, lays down cluster-
  wide prerequisites (Cilium is installed as the CNI, not flannel), and hands off to Flux.
- For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation step differs
  (bare-metal image/cloud-init via USB/PXE instead of libvirt).

### Node lifecycle stages (R-ENV-01 — "created, baselined, cluster-joined, managed as code")
1. **Create** — OpenTofu instantiates the VM (dev) / operator images the mini-PC (prod).
2. **Baseline** — cloud-init applies OS baseline and prerequisites.
3. **Cluster-join** — Ansible installs/joins k3s with Cilium as CNI; disables k3s bundled
   flannel/traefik/servicelb/kube-proxy as required by the Cilium ADR.
4. **Manage** — from here the cluster is GitOps-managed by Flux; node config changes are made via
   IaC and re-applied, never hand-edited on the box.

### k3s topology decisions
- HA control plane with **embedded etcd** across all 3 nodes (no external datastore).
- Bundled components that conflict with our stack are **disabled at install**: flannel (→ Cilium),
  kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb where
  a different LB strategy is chosen. Exact flags live in the Ansible bootstrap; the *intent* is
  fixed here and in ADR-0001/0002.
- Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not
  isolate control-plane nodes.

### Bare-metal specifics (prod, design-only for now)
- Each mini-PC contributes local NVMe/SSD to Longhorn; document the disk layout (OS vs. Longhorn
  data device) before prod build.
- BIOS/firmware, virtualization extensions (for KubeVirt), and wake/boot behavior are captured as a
  provisioning checklist under the prod-readiness workstream.
- Edge networking (multi-WAN, VPN uplink) is largely an **edge-router** concern — see
  [`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md) and OQ-1.

## Environment promotion model

```
   git repo (single source of truth)
        │
   base configuration  ──────────────┐
        │                            │
   overlay: dev/test            overlay: prod
        │                            │
   Flux on dev cluster          Flux on prod cluster (later)
```

- A change proves out in **dev/test** against the acceptance gates ([`AGENTS.md` §7](../AGENTS.md)).
- Promotion to prod is a change to the prod overlay only (no base edits), gated by the roadmap's
  prod-readiness phase.
- **Never develop against prod.** Prod hardware is a target, not a sandbox (R-ENV-03).

## Statelessness of the nodes themselves

Nodes are cattle. A node can be destroyed and rebuilt from IaC + cloud-init + Ansible; cluster and
application state lives on **Longhorn** (replicated) and in **git** (desired state), so node loss is
a re-provision, not a data-loss event (supports R-STATE-01/02).

## Deliverables for this spec (tracked on the task board)
- OpenTofu module for dev/test node VMs (libvirt) with a dedicated Longhorn data disk.
- cloud-init baseline template (users, SSH, sysctls, kernel modules for Longhorn/KubeVirt).
- Ansible role(s): k3s HA install with Cilium, component disabling, Flux bootstrap handoff.
- Documented prod bare-metal provisioning checklist (design-only; do not execute).
- Proof: destroy a dev node VM and rebuild it; cluster re-converges with no data loss.

## Open questions
- **OQ-1** (edge router / WAN count) — blocks final multi-WAN design, not node provisioning.
- Do we want **Cluster API** later to unify dev VM and prod bare-metal lifecycle? Deferred; OpenTofu
  + Ansible is sufficient and lower-complexity for 3 nodes now. Revisit via ADR if prod scaling
  demands it.