jakman85andClaude Opus 4.8
70a98675ac
- docs/07: correct plaintext-secret enforcement — CI-time ciphertext check vs
admission SOPS-provenance (Flux decrypts before the API server, so an admitted
Secret carries no ciphertext; a 'reject plaintext' admission rule is impossible).
- docs/13: add Contract clarifications (v1.1) — source exactly-one-of; longhorn⇒size
(L0 policy); backup is a {policy} ref; update.channel free-string/stable default;
auth.groups required for oidc|forward-auth; public⇒auth.mode≠none (schema);
irreplaceable⇒off-cluster-backup (L0 policy).
- docs/02: Cilium install ownership — Ansible bootstraps, Flux adopts config, value-parity contract.
- docs/03: platform label-key convention (homelab.keithjackson.dev/{egress-class,auth-mode,...}).
- docs/14: ServiceDefinition validation is two complementary L0 gates (JSON Schema shape + Kyverno invariants).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
103 lines
6.0 KiB
Markdown
103 lines
6.0 KiB
Markdown
# 02 — Environments & Provisioning
|
||
|
||
Serves: **R-ENV-01/02/03**, **R-PLAT-01**. Related: [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md).
|
||
|
||
## Two environments, one configuration
|
||
|
||
| Aspect | dev/test | prod |
|
||
|---|---|---|
|
||
| Substrate | libvirt/KVM VMs on the operator workstation | 3× mini-PC bare metal |
|
||
| Node count | 3 VMs (mirror prod topology) | 3 physical nodes |
|
||
| Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd (k3s server on all 3) |
|
||
| Storage | virtual disks presented to Longhorn | local SSD/NVMe presented to Longhorn |
|
||
| Status | **active build target** | **design-only until roadmap flips it** |
|
||
|
||
**R-ENV-02 is the golden constraint:** the GitOps configuration is authored once. Environments
|
||
differ only through scoped overrides (Flux `Kustomization` overlays / cluster-scoped variables):
|
||
node counts, disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits.
|
||
Never fork the base config per environment.
|
||
|
||
## Infrastructure provisioning (Layer 1 — the cluster nodes)
|
||
|
||
> These are the machines that *run* k3s. Distinct from KubeVirt application VMs (Layer 2).
|
||
|
||
### Toolchain
|
||
- **OpenTofu** (Terraform-compatible) with the **libvirt** provider defines dev/test node VMs as
|
||
code: CPU, RAM, disks (a dedicated data disk for Longhorn), NICs, and a **cloud-init** seed.
|
||
- **cloud-init** performs first-boot baseline: hostname, users, SSH keys, package baseline, kernel
|
||
modules/sysctls required by k3s/Longhorn/KubeVirt, and marks the node ready for k3s.
|
||
- **Ansible** performs cluster bootstrap: installs k3s in HA mode, joins nodes, lays down cluster-
|
||
wide prerequisites (Cilium is installed as the CNI, not flannel), and hands off to Flux.
|
||
- For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation step differs
|
||
(bare-metal image/cloud-init via USB/PXE instead of libvirt).
|
||
|
||
### Node lifecycle stages (R-ENV-01 — "created, baselined, cluster-joined, managed as code")
|
||
1. **Create** — OpenTofu instantiates the VM (dev) / operator images the mini-PC (prod).
|
||
2. **Baseline** — cloud-init applies OS baseline and prerequisites.
|
||
3. **Cluster-join** — Ansible installs/joins k3s with Cilium as CNI; disables k3s bundled
|
||
flannel/traefik/servicelb/kube-proxy as required by the Cilium ADR.
|
||
4. **Manage** — from here the cluster is GitOps-managed by Flux; node config changes are made via
|
||
IaC and re-applied, never hand-edited on the box.
|
||
|
||
### k3s topology decisions
|
||
- HA control plane with **embedded etcd** across all 3 nodes (no external datastore).
|
||
- Bundled components that conflict with our stack are **disabled at install**: flannel (→ Cilium),
|
||
kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb where
|
||
a different LB strategy is chosen. Exact flags live in the Ansible bootstrap; the *intent* is
|
||
fixed here and in ADR-0001/0002.
|
||
- Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not
|
||
isolate control-plane nodes.
|
||
|
||
### Cilium install ownership (Ansible bootstrap ↔ Flux)
|
||
There is a deliberate hand-off between the two control layers: **Ansible installs Cilium at cluster
|
||
bootstrap** (the CNI must exist before Flux can run any workload), and thereafter the **Flux
|
||
HelmRelease adopts and manages Cilium's configuration** ([05](05-gitops-and-delivery.md)). This
|
||
resolves the apparent overlap between "Ansible installs the CNI" (here) and "Flux manages
|
||
`platform/`" (docs/05). The contract: the bootstrap values and the Flux HelmRelease values MUST stay
|
||
in **parity** for the settings that define cluster networking — notably `kubeProxyReplacement` and
|
||
`k8sServiceHost`/`k8sServicePort` — or the two layers will fight over the CNI config.
|
||
|
||
### Bare-metal specifics (prod, design-only for now)
|
||
- Each mini-PC contributes local NVMe/SSD to Longhorn; document the disk layout (OS vs. Longhorn
|
||
data device) before prod build.
|
||
- BIOS/firmware, virtualization extensions (for KubeVirt), and wake/boot behavior are captured as a
|
||
provisioning checklist under the prod-readiness workstream.
|
||
- Edge networking (multi-WAN, VPN uplink) is largely an **edge-router** concern — see
|
||
[`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md) and OQ-1.
|
||
|
||
## Environment promotion model
|
||
|
||
```
|
||
git repo (single source of truth)
|
||
│
|
||
base configuration ──────────────┐
|
||
│ │
|
||
overlay: dev/test overlay: prod
|
||
│ │
|
||
Flux on dev cluster Flux on prod cluster (later)
|
||
```
|
||
|
||
- A change proves out in **dev/test** against the acceptance gates ([`AGENTS.md` §7](../AGENTS.md)).
|
||
- Promotion to prod is a change to the prod overlay only (no base edits), gated by the roadmap's
|
||
prod-readiness phase.
|
||
- **Never develop against prod.** Prod hardware is a target, not a sandbox (R-ENV-03).
|
||
|
||
## Statelessness of the nodes themselves
|
||
|
||
Nodes are cattle. A node can be destroyed and rebuilt from IaC + cloud-init + Ansible; cluster and
|
||
application state lives on **Longhorn** (replicated) and in **git** (desired state), so node loss is
|
||
a re-provision, not a data-loss event (supports R-STATE-01/02).
|
||
|
||
## Deliverables for this spec (tracked on the task board)
|
||
- OpenTofu module for dev/test node VMs (libvirt) with a dedicated Longhorn data disk.
|
||
- cloud-init baseline template (users, SSH, sysctls, kernel modules for Longhorn/KubeVirt).
|
||
- Ansible role(s): k3s HA install with Cilium, component disabling, Flux bootstrap handoff.
|
||
- Documented prod bare-metal provisioning checklist (design-only; do not execute).
|
||
- Proof: destroy a dev node VM and rebuild it; cluster re-converges with no data loss.
|
||
|
||
## Open questions
|
||
- **OQ-1** (edge router / WAN count) — blocks final multi-WAN design, not node provisioning.
|
||
- Do we want **Cluster API** later to unify dev VM and prod bare-metal lifecycle? Deferred; OpenTofu
|
||
+ Ansible is sufficient and lower-complexity for 3 nodes now. Revisit via ADR if prod scaling
|
||
demands it.
|