jakman85andClaude Opus 4.8
e8c555cfeb
Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision & requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap (phases + workstreams), and AI-agent coordination (roles, protocol, definition-of-done, task board). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
94 lines
5.2 KiB
Markdown
94 lines
5.2 KiB
Markdown
# 02 — Environments & Provisioning
|
||
|
||
Serves: **R-ENV-01/02/03**, **R-PLAT-01**. Related: [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md).
|
||
|
||
## Two environments, one configuration
|
||
|
||
| Aspect | dev/test | prod |
|
||
|---|---|---|
|
||
| Substrate | libvirt/KVM VMs on the operator workstation | 3× mini-PC bare metal |
|
||
| Node count | 3 VMs (mirror prod topology) | 3 physical nodes |
|
||
| Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd (k3s server on all 3) |
|
||
| Storage | virtual disks presented to Longhorn | local SSD/NVMe presented to Longhorn |
|
||
| Status | **active build target** | **design-only until roadmap flips it** |
|
||
|
||
**R-ENV-02 is the golden constraint:** the GitOps configuration is authored once. Environments
|
||
differ only through scoped overrides (Flux `Kustomization` overlays / cluster-scoped variables):
|
||
node counts, disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits.
|
||
Never fork the base config per environment.
|
||
|
||
## Infrastructure provisioning (Layer 1 — the cluster nodes)
|
||
|
||
> These are the machines that *run* k3s. Distinct from KubeVirt application VMs (Layer 2).
|
||
|
||
### Toolchain
|
||
- **OpenTofu** (Terraform-compatible) with the **libvirt** provider defines dev/test node VMs as
|
||
code: CPU, RAM, disks (a dedicated data disk for Longhorn), NICs, and a **cloud-init** seed.
|
||
- **cloud-init** performs first-boot baseline: hostname, users, SSH keys, package baseline, kernel
|
||
modules/sysctls required by k3s/Longhorn/KubeVirt, and marks the node ready for k3s.
|
||
- **Ansible** performs cluster bootstrap: installs k3s in HA mode, joins nodes, lays down cluster-
|
||
wide prerequisites (Cilium is installed as the CNI, not flannel), and hands off to Flux.
|
||
- For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation step differs
|
||
(bare-metal image/cloud-init via USB/PXE instead of libvirt).
|
||
|
||
### Node lifecycle stages (R-ENV-01 — "created, baselined, cluster-joined, managed as code")
|
||
1. **Create** — OpenTofu instantiates the VM (dev) / operator images the mini-PC (prod).
|
||
2. **Baseline** — cloud-init applies OS baseline and prerequisites.
|
||
3. **Cluster-join** — Ansible installs/joins k3s with Cilium as CNI; disables k3s bundled
|
||
flannel/traefik/servicelb/kube-proxy as required by the Cilium ADR.
|
||
4. **Manage** — from here the cluster is GitOps-managed by Flux; node config changes are made via
|
||
IaC and re-applied, never hand-edited on the box.
|
||
|
||
### k3s topology decisions
|
||
- HA control plane with **embedded etcd** across all 3 nodes (no external datastore).
|
||
- Bundled components that conflict with our stack are **disabled at install**: flannel (→ Cilium),
|
||
kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb where
|
||
a different LB strategy is chosen. Exact flags live in the Ansible bootstrap; the *intent* is
|
||
fixed here and in ADR-0001/0002.
|
||
- Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not
|
||
isolate control-plane nodes.
|
||
|
||
### Bare-metal specifics (prod, design-only for now)
|
||
- Each mini-PC contributes local NVMe/SSD to Longhorn; document the disk layout (OS vs. Longhorn
|
||
data device) before prod build.
|
||
- BIOS/firmware, virtualization extensions (for KubeVirt), and wake/boot behavior are captured as a
|
||
provisioning checklist under the prod-readiness workstream.
|
||
- Edge networking (multi-WAN, VPN uplink) is largely an **edge-router** concern — see
|
||
[`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md) and OQ-1.
|
||
|
||
## Environment promotion model
|
||
|
||
```
|
||
git repo (single source of truth)
|
||
│
|
||
base configuration ──────────────┐
|
||
│ │
|
||
overlay: dev/test overlay: prod
|
||
│ │
|
||
Flux on dev cluster Flux on prod cluster (later)
|
||
```
|
||
|
||
- A change proves out in **dev/test** against the acceptance gates ([`AGENTS.md` §7](../AGENTS.md)).
|
||
- Promotion to prod is a change to the prod overlay only (no base edits), gated by the roadmap's
|
||
prod-readiness phase.
|
||
- **Never develop against prod.** Prod hardware is a target, not a sandbox (R-ENV-03).
|
||
|
||
## Statelessness of the nodes themselves
|
||
|
||
Nodes are cattle. A node can be destroyed and rebuilt from IaC + cloud-init + Ansible; cluster and
|
||
application state lives on **Longhorn** (replicated) and in **git** (desired state), so node loss is
|
||
a re-provision, not a data-loss event (supports R-STATE-01/02).
|
||
|
||
## Deliverables for this spec (tracked on the task board)
|
||
- OpenTofu module for dev/test node VMs (libvirt) with a dedicated Longhorn data disk.
|
||
- cloud-init baseline template (users, SSH, sysctls, kernel modules for Longhorn/KubeVirt).
|
||
- Ansible role(s): k3s HA install with Cilium, component disabling, Flux bootstrap handoff.
|
||
- Documented prod bare-metal provisioning checklist (design-only; do not execute).
|
||
- Proof: destroy a dev node VM and rebuild it; cluster re-converges with no data loss.
|
||
|
||
## Open questions
|
||
- **OQ-1** (edge router / WAN count) — blocks final multi-WAN design, not node provisioning.
|
||
- Do we want **Cluster API** later to unify dev VM and prod bare-metal lifecycle? Deferred; OpenTofu
|
||
+ Ansible is sufficient and lower-complexity for 3 nodes now. Revisit via ADR if prod scaling
|
||
demands it.
|