docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
1ddc14b918
commit
250fb54154
43 files changed
+1672
-575
No files matched your search
@@ -1,39 +1,41 @@
|
||||
# 02 — Environments & Provisioning
|
||||
|
||||
Serves: **R-ENV-01/02/03**, **R-PLAT-01**. Related: [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md).
|
||||
Serves: **R-ENV-01/02/03**, **R-PLAT-01**. Related: [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md),
|
||||
[ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md).
|
||||
|
||||
## Two environments, one configuration
|
||||
## One live environment, one configuration
|
||||
|
||||
| Aspect | dev/test | prod |
|
||||
|---|---|---|
|
||||
| Substrate | libvirt/KVM VMs on the operator workstation | 3× mini-PC bare metal |
|
||||
| Node count | 3 VMs (mirror prod topology) | 3 physical nodes |
|
||||
| Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd (k3s server on all 3) |
|
||||
| Storage | virtual disks presented to Longhorn | local SSD/NVMe presented to Longhorn |
|
||||
| Status | **active build target** | **design-only until roadmap flips it** |
|
||||
| Aspect | prod |
|
||||
|---|---|
|
||||
| Substrate | 3× mini-PC bare metal |
|
||||
| Node count | 3 physical nodes |
|
||||
| Control plane | HA embedded etcd (k3s server on all 3) |
|
||||
| Storage | local SSD/NVMe presented to Longhorn; bulk media on an off-cluster NAS |
|
||||
| Status | **live** |
|
||||
|
||||
**R-ENV-02 is the golden constraint:** the GitOps configuration is authored once. Environments
|
||||
differ only through scoped overrides (Flux `Kustomization` overlays / cluster-scoped variables):
|
||||
node counts, disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits.
|
||||
Never fork the base config per environment.
|
||||
A dev/test cluster proved the platform and was decommissioned in 2026-09. Prod is the only
|
||||
environment.
|
||||
|
||||
**R-ENV-02 is the golden constraint:** the GitOps configuration is authored once, as a base plus an
|
||||
environment-scoped overlay (Flux `Kustomization` overlays / cluster-scoped variables): node counts,
|
||||
disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits. Never fork the base
|
||||
config per environment. Any future environment needs overrides only.
|
||||
|
||||
## Infrastructure provisioning (Layer 1 — the cluster nodes)
|
||||
|
||||
> These are the machines that *run* k3s. Distinct from KubeVirt application VMs (Layer 2).
|
||||
|
||||
### Toolchain
|
||||
- **OpenTofu** (Terraform-compatible) with the **libvirt** provider defines dev/test node VMs as
|
||||
code: CPU, RAM, disks (a dedicated data disk for Longhorn), NICs, and a **cloud-init** seed.
|
||||
- **cloud-init** performs first-boot baseline: hostname, users, SSH keys, package baseline, kernel
|
||||
modules/sysctls required by k3s/Longhorn/KubeVirt, and marks the node ready for k3s.
|
||||
- **Bare-metal imaging** is an operator step driven by a checklist. There is no hypervisor API on
|
||||
bare metal to automate machine creation.
|
||||
- **Ansible** performs the node baseline: users, package floor, kernel, kernel modules/sysctls
|
||||
required by k3s/Longhorn/KubeVirt, and the data partition.
|
||||
- **Ansible** performs cluster bootstrap: installs k3s in HA mode, joins nodes, lays down cluster-
|
||||
wide prerequisites (Cilium is installed as the CNI, not flannel), and hands off to Flux.
|
||||
- For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation step differs
|
||||
(bare-metal image/cloud-init via USB/PXE instead of libvirt).
|
||||
|
||||
### Node lifecycle stages (R-ENV-01 — "created, baselined, cluster-joined, managed as code")
|
||||
1. **Create** — OpenTofu instantiates the VM (dev) / operator images the mini-PC (prod).
|
||||
2. **Baseline** — cloud-init applies OS baseline and prerequisites.
|
||||
### Node lifecycle stages (R-ENV-01 — "baselined, cluster-joined, managed as code")
|
||||
1. **Create** — the operator images the mini-PC from the provisioning checklist.
|
||||
2. **Baseline** — Ansible applies the OS baseline and prerequisites.
|
||||
3. **Cluster-join** — Ansible installs/joins k3s with Cilium as CNI; disables k3s bundled
|
||||
flannel/traefik/servicelb/kube-proxy as required by the Cilium ADR.
|
||||
4. **Manage** — from here the cluster is GitOps-managed by Flux; node config changes are made via
|
||||
@@ -41,10 +43,13 @@ Never fork the base config per environment.
|
||||
|
||||
### k3s topology decisions
|
||||
- HA control plane with **embedded etcd** across all 3 nodes (no external datastore).
|
||||
- The Kubernetes API has one HA endpoint: a **kube-vip** virtual IP. Steady-state consumers use the
|
||||
virtual IP. A node joining during a cold bootstrap uses the founding server's own address, because
|
||||
the virtual IP does not exist until a cluster does.
|
||||
- Bundled components that conflict with our stack are **disabled at install**: flannel (→ Cilium),
|
||||
kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb where
|
||||
a different LB strategy is chosen. Exact flags live in the Ansible bootstrap; the *intent* is
|
||||
fixed here and in ADR-0001/0002.
|
||||
kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb
|
||||
(→ Cilium L2 announcements and LB-IPAM), local-path storage (→ `longhorn` as the sole default).
|
||||
Exact flags live in the Ansible bootstrap; the *intent* is fixed here and in ADR-0001/0002.
|
||||
- Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not
|
||||
isolate control-plane nodes.
|
||||
|
||||
@@ -57,46 +62,63 @@ resolves the apparent overlap between "Ansible installs the CNI" (here) and "Flu
|
||||
in **parity** for the settings that define cluster networking — notably `kubeProxyReplacement` and
|
||||
`k8sServiceHost`/`k8sServicePort` — or the two layers will fight over the CNI config.
|
||||
|
||||
### Bare-metal specifics (prod, design-only for now)
|
||||
- Each mini-PC contributes local NVMe/SSD to Longhorn; document the disk layout (OS vs. Longhorn
|
||||
data device) before prod build.
|
||||
- BIOS/firmware, virtualization extensions (for KubeVirt), and wake/boot behavior are captured as a
|
||||
provisioning checklist under the prod-readiness workstream.
|
||||
- Edge networking (multi-WAN, VPN uplink) is largely an **edge-router** concern — see
|
||||
### Bare-metal specifics
|
||||
- The nodes are 3 HP EliteDesk Mini desktops with VT-x. Their drives differ (SATA SSD and NVMe); the
|
||||
disk *layout* is identical across all 3. The node inventory, BIOS/firmware settings, and the
|
||||
provisioning checklist are `infra/PROD-PROVISIONING.md` in the
|
||||
[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).
|
||||
- Each node has **one physical disk**: an OS area and one data partition. The data partition holds
|
||||
both the Longhorn store and the k3s data directory (embedded etcd and the container image store).
|
||||
- Nodes run a newer kernel series than the distribution default, because the firmware publishes a
|
||||
defective EFI memory table that older kernels mishandle. This is a standing, deliberate deviation.
|
||||
- **Guardrail and accepted risk (etcd and Longhorn share a device).** With no second device in the
|
||||
chassis, etcd shares each node's disk with Longhorn replica I/O. Every node MUST be SSD or NVMe,
|
||||
and etcd takes I/O priority over other work on the device. The risk is accepted knowingly. Its
|
||||
exit condition is that etcd write latency stays within budget during a Longhorn rebuild, or each
|
||||
node gains a dedicated etcd device (see [known gaps](16-known-gaps.md)).
|
||||
- The edge is a single-WAN SOHO router — see
|
||||
[`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md) and OQ-1.
|
||||
|
||||
### Off-host storage
|
||||
Bulk media lives on a standalone TrueNAS NFS server on the production LAN, directly reachable by the
|
||||
nodes (ADR-0018, [04](04-storage-and-state.md)). It is storage only and runs no platform compute.
|
||||
|
||||
### Node maintenance
|
||||
OS patching is automatic and security-only, excluding the kernel. Kernel upgrades are deliberate
|
||||
operator actions, run one node at a time. A reboot daemon carries out the mechanics of a needed
|
||||
reboot and takes a single cluster-wide lock, so at most one node reboots at a time, and the next
|
||||
waits for Longhorn volumes to recover. A hung node holds the lock, stopping the cycle until a human
|
||||
intervenes: on 3 etcd members, losing a second node is the failure to avoid.
|
||||
|
||||
## Environment promotion model
|
||||
|
||||
```
|
||||
git repo (single source of truth)
|
||||
│
|
||||
base configuration ──────────────┐
|
||||
│ │
|
||||
overlay: dev/test overlay: prod
|
||||
│ │
|
||||
Flux on dev cluster Flux on prod cluster (later)
|
||||
base configuration
|
||||
│
|
||||
overlay: prod
|
||||
│
|
||||
Flux on the prod cluster
|
||||
```
|
||||
|
||||
- A change proves out in **dev/test** against the acceptance gates ([`AGENTS.md` §7](../AGENTS.md)).
|
||||
- Promotion to prod is a change to the prod overlay only (no base edits), gated by the roadmap's
|
||||
prod-readiness phase.
|
||||
- **Never develop against prod.** Prod hardware is a target, not a sandbox (R-ENV-03).
|
||||
- A change proves out against the acceptance gates ([`AGENTS.md` §7](../AGENTS.md)) before it merges.
|
||||
- A future environment is a new overlay only (no base edits).
|
||||
- **Changes reach prod only through the PR gate and Flux** (R-ENV-03). Fault-injecting tests against
|
||||
prod are deliberate, scheduled exercises.
|
||||
|
||||
## Statelessness of the nodes themselves
|
||||
|
||||
Nodes are cattle. A node can be destroyed and rebuilt from IaC + cloud-init + Ansible; cluster and
|
||||
Nodes are cattle. A node can be destroyed and rebuilt by imaging and Ansible; cluster and
|
||||
application state lives on **Longhorn** (replicated) and in **git** (desired state), so node loss is
|
||||
a re-provision, not a data-loss event (supports R-STATE-01/02).
|
||||
|
||||
## Deliverables for this spec (tracked on the task board)
|
||||
- OpenTofu module for dev/test node VMs (libvirt) with a dedicated Longhorn data disk.
|
||||
- cloud-init baseline template (users, SSH, sysctls, kernel modules for Longhorn/KubeVirt).
|
||||
- Ansible role(s): k3s HA install with Cilium, component disabling, Flux bootstrap handoff.
|
||||
- Documented prod bare-metal provisioning checklist (design-only; do not execute).
|
||||
- Proof: destroy a dev node VM and rebuild it; cluster re-converges with no data loss.
|
||||
## Deliverables for this spec
|
||||
- Ansible role(s): node baseline, k3s HA install with Cilium, component disabling, Flux bootstrap
|
||||
handoff.
|
||||
- Documented bare-metal provisioning checklist.
|
||||
- Proof: lose a node and rebuild it; the cluster re-converges with no data loss.
|
||||
|
||||
## Open questions
|
||||
- **OQ-1** (edge router / WAN count) — blocks final multi-WAN design, not node provisioning.
|
||||
- Do we want **Cluster API** later to unify dev VM and prod bare-metal lifecycle? Deferred; OpenTofu
|
||||
+ Ansible is sufficient and lower-complexity for 3 nodes now. Revisit via ADR if prod scaling
|
||||
demands it.
|
||||
- Do we want **Cluster API** later to automate bare-metal lifecycle? Deferred; imaging + Ansible is
|
||||
sufficient and lower-complexity for 3 nodes. Revisit via ADR if scaling demands it.
|
||||
Reference in new issue
Block a user