home-network-k3s/docs/02-environments-and-provisioning.md

# 02 — Environments & Provisioning

Serves: **R-ENV-01/02/03**, **R-PLAT-01**. Related: [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md),
[ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md).

## One live environment, one configuration

| Aspect | prod |
|---|---|
| Substrate | 3× mini-PC bare metal |
| Node count | 3 physical nodes |
| Control plane | HA embedded etcd (k3s server on all 3) |
| Storage | local SSD/NVMe presented to Longhorn; bulk media on an off-cluster NAS |
| Status | **live** |

A dev/test cluster proved the platform and was decommissioned in 2026-09. Prod is the only
environment.

**R-ENV-02 is the golden constraint:** the GitOps configuration is authored once, as a base plus an
environment-scoped overlay (Flux `Kustomization` overlays / cluster-scoped variables): node counts,
disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits. Never fork the base
config per environment. Any future environment needs overrides only.

## Infrastructure provisioning (Layer 1 — the cluster nodes)

> These are the machines that *run* k3s. Distinct from KubeVirt application VMs (Layer 2).

### Toolchain
- **Bare-metal imaging** is an operator step driven by a checklist. There is no hypervisor API on
  bare metal to automate machine creation.
- **Ansible** performs the node baseline: users, package floor, kernel, kernel modules/sysctls
  required by k3s/Longhorn/KubeVirt, and the data partition.
- **Ansible** performs cluster bootstrap: installs k3s in HA mode, joins nodes, lays down cluster-
  wide prerequisites (Cilium is installed as the CNI, not flannel), and hands off to Flux.

### Node lifecycle stages (R-ENV-01 — "baselined, cluster-joined, managed as code")
1. **Create** — the operator images the mini-PC from the provisioning checklist.
2. **Baseline** — Ansible applies the OS baseline and prerequisites.
3. **Cluster-join** — Ansible installs/joins k3s with Cilium as CNI; disables k3s bundled
   flannel/traefik/servicelb/kube-proxy as required by the Cilium ADR.
4. **Manage** — from here the cluster is GitOps-managed by Flux; node config changes are made via
   IaC and re-applied, never hand-edited on the box.

### k3s topology decisions
- HA control plane with **embedded etcd** across all 3 nodes (no external datastore).
- The Kubernetes API has one HA endpoint: a **kube-vip** virtual IP. Steady-state consumers use the
  virtual IP. A node joining during a cold bootstrap uses the founding server's own address, because
  the virtual IP does not exist until a cluster does.
- Bundled components that conflict with our stack are **disabled at install**: flannel (→ Cilium),
  kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb
  (→ Cilium L2 announcements and LB-IPAM), local-path storage (→ `longhorn` as the sole default).
  Exact flags live in the Ansible bootstrap; the *intent* is fixed here and in ADR-0001/0002.
- Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not
  isolate control-plane nodes.

### Cilium install ownership (Ansible bootstrap ↔ Flux)
There is a deliberate hand-off between the two control layers: **Ansible installs Cilium at cluster
bootstrap** (the CNI must exist before Flux can run any workload), and thereafter the **Flux
HelmRelease adopts and manages Cilium's configuration** ([05](05-gitops-and-delivery.md)). This
resolves the apparent overlap between "Ansible installs the CNI" (here) and "Flux manages
`platform/`" (docs/05). The contract: the bootstrap values and the Flux HelmRelease values MUST stay
in **parity** for the settings that define cluster networking — notably `kubeProxyReplacement` and
`k8sServiceHost`/`k8sServicePort` — or the two layers will fight over the CNI config.

### Bare-metal specifics
- The nodes are 3 HP EliteDesk Mini desktops with VT-x. Their drives differ (SATA SSD and NVMe); the
  disk *layout* is identical across all 3. The node inventory, BIOS/firmware settings, and the
  provisioning checklist are `infra/PROD-PROVISIONING.md` in the
  [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).
- Each node has **one physical disk**: an OS area and one data partition. The data partition holds
  both the Longhorn store and the k3s data directory (embedded etcd and the container image store).
- Nodes run a newer kernel series than the distribution default, because the firmware publishes a
  defective EFI memory table that older kernels mishandle. This is a standing, deliberate deviation.
- **Guardrail and accepted risk (etcd and Longhorn share a device).** With no second device in the
  chassis, etcd shares each node's disk with Longhorn replica I/O. Every node MUST be SSD or NVMe,
  and etcd takes I/O priority over other work on the device. The risk is accepted knowingly. Its
  exit condition is that etcd write latency stays within budget during a Longhorn rebuild, or each
  node gains a dedicated etcd device (see [known gaps](16-known-gaps.md)).
- The edge is a single-WAN SOHO router — see
  [`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md) and OQ-1.

### Off-host storage
Bulk media lives on a standalone TrueNAS NFS server on the production LAN, directly reachable by the
nodes (ADR-0018, [04](04-storage-and-state.md)). It is storage only and runs no platform compute.

### Node maintenance
OS patching is automatic and security-only, excluding the kernel. Kernel upgrades are deliberate
operator actions, run one node at a time. A reboot daemon carries out the mechanics of a needed
reboot and takes a single cluster-wide lock, so at most one node reboots at a time, and the next
waits for Longhorn volumes to recover. A hung node holds the lock, stopping the cycle until a human
intervenes: on 3 etcd members, losing a second node is the failure to avoid.

## Environment promotion model

```
   git repo (single source of truth)
        │
   base configuration
        │
   overlay: prod
        │
   Flux on the prod cluster
```

- A change proves out against the acceptance gates ([`AGENTS.md` §7](../AGENTS.md)) before it merges.
- A future environment is a new overlay only (no base edits).
- **Changes reach prod only through the PR gate and Flux** (R-ENV-03). Fault-injecting tests against
  prod are deliberate, scheduled exercises.

## Statelessness of the nodes themselves

Nodes are cattle. A node can be destroyed and rebuilt by imaging and Ansible; cluster and
application state lives on **Longhorn** (replicated) and in **git** (desired state), so node loss is
a re-provision, not a data-loss event (supports R-STATE-01/02).

## Deliverables for this spec
- Ansible role(s): node baseline, k3s HA install with Cilium, component disabling, Flux bootstrap
  handoff.
- Documented bare-metal provisioning checklist.
- Proof: lose a node and rebuild it; the cluster re-converges with no data loss.

## Open questions
- Do we want **Cluster API** later to automate bare-metal lifecycle? Deferred; imaging + Ansible is
  sufficient and lower-complexity for 3 nodes. Revisit via ADR if scaling demands it.