jakman85andClaude Opus 4.8
70a98675ac
- docs/07: correct plaintext-secret enforcement — CI-time ciphertext check vs
admission SOPS-provenance (Flux decrypts before the API server, so an admitted
Secret carries no ciphertext; a 'reject plaintext' admission rule is impossible).
- docs/13: add Contract clarifications (v1.1) — source exactly-one-of; longhorn⇒size
(L0 policy); backup is a {policy} ref; update.channel free-string/stable default;
auth.groups required for oidc|forward-auth; public⇒auth.mode≠none (schema);
irreplaceable⇒off-cluster-backup (L0 policy).
- docs/02: Cilium install ownership — Ansible bootstraps, Flux adopts config, value-parity contract.
- docs/03: platform label-key convention (homelab.keithjackson.dev/{egress-class,auth-mode,...}).
- docs/14: ServiceDefinition validation is two complementary L0 gates (JSON Schema shape + Kyverno invariants).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
6.0 KiB
02 — Environments & Provisioning
Serves: R-ENV-01/02/03, R-PLAT-01. Related: ADR-0001.
Two environments, one configuration
| Aspect | dev/test | prod |
|---|---|---|
| Substrate | libvirt/KVM VMs on the operator workstation | 3× mini-PC bare metal |
| Node count | 3 VMs (mirror prod topology) | 3 physical nodes |
| Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd (k3s server on all 3) |
| Storage | virtual disks presented to Longhorn | local SSD/NVMe presented to Longhorn |
| Status | active build target | design-only until roadmap flips it |
R-ENV-02 is the golden constraint: the GitOps configuration is authored once. Environments
differ only through scoped overrides (Flux Kustomization overlays / cluster-scoped variables):
node counts, disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits.
Never fork the base config per environment.
Infrastructure provisioning (Layer 1 — the cluster nodes)
These are the machines that run k3s. Distinct from KubeVirt application VMs (Layer 2).
Toolchain
- OpenTofu (Terraform-compatible) with the libvirt provider defines dev/test node VMs as code: CPU, RAM, disks (a dedicated data disk for Longhorn), NICs, and a cloud-init seed.
- cloud-init performs first-boot baseline: hostname, users, SSH keys, package baseline, kernel modules/sysctls required by k3s/Longhorn/KubeVirt, and marks the node ready for k3s.
- Ansible performs cluster bootstrap: installs k3s in HA mode, joins nodes, lays down cluster- wide prerequisites (Cilium is installed as the CNI, not flannel), and hands off to Flux.
- For prod bare-metal, the same Ansible bootstrap runs; only the machine-creation step differs (bare-metal image/cloud-init via USB/PXE instead of libvirt).
Node lifecycle stages (R-ENV-01 — "created, baselined, cluster-joined, managed as code")
- Create — OpenTofu instantiates the VM (dev) / operator images the mini-PC (prod).
- Baseline — cloud-init applies OS baseline and prerequisites.
- Cluster-join — Ansible installs/joins k3s with Cilium as CNI; disables k3s bundled flannel/traefik/servicelb/kube-proxy as required by the Cilium ADR.
- Manage — from here the cluster is GitOps-managed by Flux; node config changes are made via IaC and re-applied, never hand-edited on the box.
k3s topology decisions
- HA control plane with embedded etcd across all 3 nodes (no external datastore).
- Bundled components that conflict with our stack are disabled at install: flannel (→ Cilium), kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb where a different LB strategy is chosen. Exact flags live in the Ansible bootstrap; the intent is fixed here and in ADR-0001/0002.
- Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not isolate control-plane nodes.
Cilium install ownership (Ansible bootstrap ↔ Flux)
There is a deliberate hand-off between the two control layers: Ansible installs Cilium at cluster
bootstrap (the CNI must exist before Flux can run any workload), and thereafter the Flux
HelmRelease adopts and manages Cilium's configuration (05). This
resolves the apparent overlap between "Ansible installs the CNI" (here) and "Flux manages
platform/" (docs/05). The contract: the bootstrap values and the Flux HelmRelease values MUST stay
in parity for the settings that define cluster networking — notably kubeProxyReplacement and
k8sServiceHost/k8sServicePort — or the two layers will fight over the CNI config.
Bare-metal specifics (prod, design-only for now)
- Each mini-PC contributes local NVMe/SSD to Longhorn; document the disk layout (OS vs. Longhorn data device) before prod build.
- BIOS/firmware, virtualization extensions (for KubeVirt), and wake/boot behavior are captured as a provisioning checklist under the prod-readiness workstream.
- Edge networking (multi-WAN, VPN uplink) is largely an edge-router concern — see
03-networking-isolation-and-egress.mdand OQ-1.
Environment promotion model
git repo (single source of truth)
│
base configuration ──────────────┐
│ │
overlay: dev/test overlay: prod
│ │
Flux on dev cluster Flux on prod cluster (later)
- A change proves out in dev/test against the acceptance gates (
AGENTS.md§7). - Promotion to prod is a change to the prod overlay only (no base edits), gated by the roadmap's prod-readiness phase.
- Never develop against prod. Prod hardware is a target, not a sandbox (R-ENV-03).
Statelessness of the nodes themselves
Nodes are cattle. A node can be destroyed and rebuilt from IaC + cloud-init + Ansible; cluster and application state lives on Longhorn (replicated) and in git (desired state), so node loss is a re-provision, not a data-loss event (supports R-STATE-01/02).
Deliverables for this spec (tracked on the task board)
- OpenTofu module for dev/test node VMs (libvirt) with a dedicated Longhorn data disk.
- cloud-init baseline template (users, SSH, sysctls, kernel modules for Longhorn/KubeVirt).
- Ansible role(s): k3s HA install with Cilium, component disabling, Flux bootstrap handoff.
- Documented prod bare-metal provisioning checklist (design-only; do not execute).
- Proof: destroy a dev node VM and rebuild it; cluster re-converges with no data loss.
Open questions
- OQ-1 (edge router / WAN count) — blocks final multi-WAN design, not node provisioning.
- Do we want Cluster API later to unify dev VM and prod bare-metal lifecycle? Deferred; OpenTofu
- Ansible is sufficient and lower-complexity for 3 nodes now. Revisit via ADR if prod scaling demands it.