home-network-k3s/docs/02-environments-and-provisioning.md

02 — Environments & Provisioning

Serves: R-ENV-01/02/03, R-PLAT-01. Related: ADR-0001, ADR-0017.

One live environment, one configuration

Aspect	prod
Substrate	3× mini-PC bare metal
Node count	3 physical nodes
Control plane	HA embedded etcd (k3s server on all 3)
Storage	local SSD/NVMe presented to Longhorn; bulk media on an off-cluster NAS
Status	live

A dev/test cluster proved the platform and was decommissioned in 2026-09. Prod is the only environment.

R-ENV-02 is the golden constraint: the GitOps configuration is authored once, as a base plus an environment-scoped overlay (Flux Kustomization overlays / cluster-scoped variables): node counts, disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits. Never fork the base config per environment. Any future environment needs overrides only.

Infrastructure provisioning (Layer 1 — the cluster nodes)

These are the machines that run k3s. Distinct from KubeVirt application VMs (Layer 2).

Toolchain

• Bare-metal imaging is an operator step driven by a checklist. There is no hypervisor API on bare metal to automate machine creation.
• Ansible performs the node baseline: users, package floor, kernel, kernel modules/sysctls required by k3s/Longhorn/KubeVirt, and the data partition.
• Ansible performs cluster bootstrap: installs k3s in HA mode, joins nodes, lays down cluster- wide prerequisites (Cilium is installed as the CNI, not flannel), and hands off to Flux.

Node lifecycle stages (R-ENV-01 — "baselined, cluster-joined, managed as code")

1. Create — the operator images the mini-PC from the provisioning checklist.
2. Baseline — Ansible applies the OS baseline and prerequisites.
3. Cluster-join — Ansible installs/joins k3s with Cilium as CNI; disables k3s bundled flannel/traefik/servicelb/kube-proxy as required by the Cilium ADR.
4. Manage — from here the cluster is GitOps-managed by Flux; node config changes are made via IaC and re-applied, never hand-edited on the box.

k3s topology decisions

• HA control plane with embedded etcd across all 3 nodes (no external datastore).
• The Kubernetes API has one HA endpoint: a kube-vip virtual IP. Steady-state consumers use the virtual IP. A node joining during a cold bootstrap uses the founding server's own address, because the virtual IP does not exist until a cluster does.
• Bundled components that conflict with our stack are disabled at install: flannel (→ Cilium), kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb (→ Cilium L2 announcements and LB-IPAM), local-path storage (→ longhorn as the sole default). Exact flags live in the Ansible bootstrap; the intent is fixed here and in ADR-0001/0002.
• Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not isolate control-plane nodes.

Cilium install ownership (Ansible bootstrap ↔ Flux)

There is a deliberate hand-off between the two control layers: Ansible installs Cilium at cluster bootstrap (the CNI must exist before Flux can run any workload), and thereafter the Flux HelmRelease adopts and manages Cilium's configuration (05). This resolves the apparent overlap between "Ansible installs the CNI" (here) and "Flux manages platform/" (docs/05). The contract: the bootstrap values and the Flux HelmRelease values MUST stay in parity for the settings that define cluster networking — notably kubeProxyReplacement and k8sServiceHost/k8sServicePort — or the two layers will fight over the CNI config.

Bare-metal specifics

• The nodes are 3 HP EliteDesk Mini desktops with VT-x. Their drives differ (SATA SSD and NVMe); the disk layout is identical across all 3. The node inventory, BIOS/firmware settings, and the provisioning checklist are infra/PROD-PROVISIONING.md in the implementation repo.
• Each node has one physical disk: an OS area and one data partition. The data partition holds both the Longhorn store and the k3s data directory (embedded etcd and the container image store).
• Nodes run a newer kernel series than the distribution default, because the firmware publishes a defective EFI memory table that older kernels mishandle. This is a standing, deliberate deviation.
• Guardrail and accepted risk (etcd and Longhorn share a device). With no second device in the chassis, etcd shares each node's disk with Longhorn replica I/O. Every node MUST be SSD or NVMe, and etcd takes I/O priority over other work on the device. The risk is accepted knowingly. Its exit condition is that etcd write latency stays within budget during a Longhorn rebuild, or each node gains a dedicated etcd device (see known gaps).
• The edge is a single-WAN SOHO router — see 03-networking-isolation-and-egress.md and OQ-1.

Off-host storage

Bulk media lives on a standalone TrueNAS NFS server on the production LAN, directly reachable by the nodes (ADR-0018, 04). It is storage only and runs no platform compute.

Node maintenance

OS patching is automatic and security-only, excluding the kernel. Kernel upgrades are deliberate operator actions, run one node at a time. A reboot daemon carries out the mechanics of a needed reboot and takes a single cluster-wide lock, so at most one node reboots at a time, and the next waits for Longhorn volumes to recover. A hung node holds the lock, stopping the cycle until a human intervenes: on 3 etcd members, losing a second node is the failure to avoid.

Environment promotion model

   git repo (single source of truth)
        │
   base configuration
        │
   overlay: prod
        │
   Flux on the prod cluster

• A change proves out against the acceptance gates (AGENTS.md §7) before it merges.
• A future environment is a new overlay only (no base edits).
• Changes reach prod only through the PR gate and Flux (R-ENV-03). Fault-injecting tests against prod are deliberate, scheduled exercises.

Statelessness of the nodes themselves

Nodes are cattle. A node can be destroyed and rebuilt by imaging and Ansible; cluster and application state lives on Longhorn (replicated) and in git (desired state), so node loss is a re-provision, not a data-loss event (supports R-STATE-01/02).

Deliverables for this spec

• Ansible role(s): node baseline, k3s HA install with Cilium, component disabling, Flux bootstrap handoff.
• Documented bare-metal provisioning checklist.
• Proof: lose a node and rebuild it; the cluster re-converges with no data loss.

Open questions

• Do we want Cluster API later to automate bare-metal lifecycle? Deferred; imaging + Ansible is sufficient and lower-complexity for 3 nodes. Revisit via ADR if scaling demands it.