home-network-k3s/docs/02-environments-and-provisioning.md

02 — Environments & Provisioning

Serves: R-ENV-01/02/03, R-PLAT-01. Related: ADR-0001.

Two environments, one configuration

Aspect	dev/test	prod
Substrate	libvirt/KVM VMs on the operator workstation	3× mini-PC bare metal
Node count	3 VMs (mirror prod topology)	3 physical nodes
Control plane	HA embedded etcd (k3s server on all 3)	HA embedded etcd (k3s server on all 3)
Storage	virtual disks presented to Longhorn	local SSD/NVMe presented to Longhorn
Status	active build target	design-only until roadmap flips it

R-ENV-02 is the golden constraint: the GitOps configuration is authored once. Environments differ only through scoped overrides (Flux Kustomization overlays / cluster-scoped variables): node counts, disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits. Never fork the base config per environment.

Infrastructure provisioning (Layer 1 — the cluster nodes)

These are the machines that run k3s. Distinct from KubeVirt application VMs (Layer 2).

Toolchain

• OpenTofu (Terraform-compatible) with the libvirt provider defines dev/test node VMs as code: CPU, RAM, disks (a dedicated data disk for Longhorn), NICs, and a cloud-init seed.
• cloud-init performs first-boot baseline: hostname, users, SSH keys, package baseline, kernel modules/sysctls required by k3s/Longhorn/KubeVirt, and marks the node ready for k3s.
• Ansible performs cluster bootstrap: installs k3s in HA mode, joins nodes, lays down cluster- wide prerequisites (Cilium is installed as the CNI, not flannel), and hands off to Flux.
• For prod bare-metal, the same Ansible bootstrap runs; only the machine-creation step differs (bare-metal image/cloud-init via USB/PXE instead of libvirt).

Node lifecycle stages (R-ENV-01 — "created, baselined, cluster-joined, managed as code")

1. Create — OpenTofu instantiates the VM (dev) / operator images the mini-PC (prod).
2. Baseline — cloud-init applies OS baseline and prerequisites.
3. Cluster-join — Ansible installs/joins k3s with Cilium as CNI; disables k3s bundled flannel/traefik/servicelb/kube-proxy as required by the Cilium ADR.
4. Manage — from here the cluster is GitOps-managed by Flux; node config changes are made via IaC and re-applied, never hand-edited on the box.

k3s topology decisions

• HA control plane with embedded etcd across all 3 nodes (no external datastore).
• Bundled components that conflict with our stack are disabled at install: flannel (→ Cilium), kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb where a different LB strategy is chosen. Exact flags live in the Ansible bootstrap; the intent is fixed here and in ADR-0001/0002.
• Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not isolate control-plane nodes.

Bare-metal specifics (prod, design-only for now)

• Each mini-PC contributes local NVMe/SSD to Longhorn; document the disk layout (OS vs. Longhorn data device) before prod build.
• BIOS/firmware, virtualization extensions (for KubeVirt), and wake/boot behavior are captured as a provisioning checklist under the prod-readiness workstream.
• Edge networking (multi-WAN, VPN uplink) is largely an edge-router concern — see 03-networking-isolation-and-egress.md and OQ-1.

Environment promotion model

   git repo (single source of truth)
        │
   base configuration  ──────────────┐
        │                            │
   overlay: dev/test            overlay: prod
        │                            │
   Flux on dev cluster          Flux on prod cluster (later)

• A change proves out in dev/test against the acceptance gates (AGENTS.md §7).
• Promotion to prod is a change to the prod overlay only (no base edits), gated by the roadmap's prod-readiness phase.
• Never develop against prod. Prod hardware is a target, not a sandbox (R-ENV-03).

Statelessness of the nodes themselves

Nodes are cattle. A node can be destroyed and rebuilt from IaC + cloud-init + Ansible; cluster and application state lives on Longhorn (replicated) and in git (desired state), so node loss is a re-provision, not a data-loss event (supports R-STATE-01/02).

Deliverables for this spec (tracked on the task board)

• OpenTofu module for dev/test node VMs (libvirt) with a dedicated Longhorn data disk.
• cloud-init baseline template (users, SSH, sysctls, kernel modules for Longhorn/KubeVirt).
• Ansible role(s): k3s HA install with Cilium, component disabling, Flux bootstrap handoff.
• Documented prod bare-metal provisioning checklist (design-only; do not execute).
• Proof: destroy a dev node VM and rebuild it; cluster re-converges with no data loss.

Open questions

• OQ-1 (edge router / WAN count) — blocks final multi-WAN design, not node provisioning.
• Do we want Cluster API later to unify dev VM and prod bare-metal lifecycle? Deferred; OpenTofu
  • Ansible is sufficient and lower-complexity for 3 nodes now. Revisit via ADR if prod scaling demands it.