jakman85andClaude Opus 4.8
5d2b248b68
Per operator rule: OpenTofu/age/sops/flux are Go binaries, so bake the toolchain into a pinned Fedora-44 dev container (matches host libvirt 12) instead of a venv. infra/devcontainer/: Containerfile + versions.env (tofu 1.9.0, age 1.2.1, sops 3.9.4, flux 2.4.0, kubectl 1.32.0, helm 3.16.0) + dev.sh (runs with host libvirt socket + repo + ssh mounted) + README. Built & validated: all tools resolve; virsh reaches host libvirt from inside the container. Also record resolved Phase 1 decisions in PHASE1-SCOPE.md (6GiB/4vCPU x3; k3s-dev NAT 10.73.0.0/24; container toolchain). Gate green. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
100 lines
6.7 KiB
Markdown
100 lines
6.7 KiB
Markdown
# Phase 1 Scope — dev/test Cluster Bring-up (local KVM)
|
||
|
||
**Owner:** Program Manager. **Spec refs (read-only):** `../home-network-k3s/roadmap/phases.md`
|
||
(Phase 1), `docs/02-environments-and-provisioning.md`, ADRs 0001 (k3s), 0002 (Cilium), 0004 (Flux),
|
||
0007 (SOPS). **Status:** scoped, awaiting operator go / prerequisite installs.
|
||
|
||
## Objective & exit criteria (roadmap Phase 1)
|
||
A **3-node k3s HA cluster** on local libvirt/KVM VMs, provisioned entirely by IaC
|
||
(**OpenTofu + cloud-init + Ansible**), with **Cilium** CNI, **Flux** reconciling from `clusters/dev`,
|
||
and the **SOPS age key** bootstrapped. Exit gates:
|
||
1. 3-node k3s HA (embedded etcd) **Ready**; Cilium healthy (Hubble up); no flannel/kube-proxy/Traefik.
|
||
2. Flux reconciling `clusters/dev` from git; drift corrected.
|
||
3. A node VM **destroyed and rebuilt** from `tofu apply` re-joins; cluster reconverges, **no data loss**.
|
||
4. Flip `CLUSTER_AVAILABLE=true` → the conformance **L2/L3 behavior tests begin running** (first real
|
||
default-deny / data-survival green — the gate stops skipping).
|
||
|
||
## Environment (verified 2026-08-16)
|
||
| Resource | Finding | Implication |
|
||
|---|---|---|
|
||
| CPU | 24 cores, AMD-V (`svm`), `kvm_amd` loaded | ample; nested-virt for KubeVirt (Phase 5) to be confirmed later |
|
||
| RAM | 31 GiB total, **~22 GiB available** | **binding constraint** — see sizing |
|
||
| Disk | 137 GiB free on `/` | tight; use thin qcow2, keep dev volumes small |
|
||
| libvirt | 12.0.0, functional (existing VMs, `default` NAT net, pools) | reuse or add a dedicated net |
|
||
| Present | ansible 2.20, virt-install, qemu-img, kubectl, helm, genisoimage | ok |
|
||
| **Missing (prereqs)** | **OpenTofu(+libvirt provider), age, sops, flux CLI**, cloud-localds | must install before execution |
|
||
|
||
## Toolchain — pinned dev container (decided)
|
||
The prereqs are **not all Python** (OpenTofu/age/sops/flux are Go binaries), so per the operator's
|
||
rule they are delivered as a **pinned dev container**, not host installs (and not a venv). See
|
||
[`devcontainer/`](devcontainer/): OpenTofu + `dmacvicar/libvirt` deps, age, sops, flux, ansible,
|
||
kubectl, helm, libvirt-client — baked into a Fedora-44 image (matches host libvirt 12). Run with the
|
||
host libvirt socket + repo mounted:
|
||
```
|
||
cd infra/devcontainer && ./dev.sh build && ./dev.sh
|
||
```
|
||
This isolates the toolchain, keeps the host clean, and gives every worker the same baseline.
|
||
|
||
## Resource plan (constrained by ~22 GiB RAM / 137 GiB disk)
|
||
Per node: **4 vCPU · 6 GiB RAM · 20 GiB OS disk + 40 GiB thin data disk** (Longhorn).
|
||
Totals: 12 vCPU · **18 GiB RAM** (~4 GiB host headroom) · ~180 GiB thin-allocated (sparse).
|
||
- ⚠️ **RAM is tight.** The *full* platform (Longhorn + Cilium + kube-prometheus-stack + Loki +
|
||
Keycloak/Postgres + cert-manager + …) across 18 GiB is a squeeze. In dev we **trim non-essential
|
||
components** (e.g. shorter Prometheus/Loki retention, single-replica where safe) or the operator
|
||
frees more RAM. This workstation cannot carry prod-scale — which is exactly why prod is 3 dedicated
|
||
mini-PCs.
|
||
- ⚠️ **Disk is tight** for media; keep Longhorn volumes minimal in dev — bulk media is external anyway
|
||
(OQ-2). Thin qcow2 means allocation ≫ real usage initially.
|
||
|
||
## Network plan
|
||
- A **dedicated libvirt NAT network** for the cluster (proposed `k3s-dev`, `10.73.0.0/24`) — isolates
|
||
it from the operator's other VMs, host-reachable. (Alternative: reuse `default` 192.168.122/24.)
|
||
- **Static IPs**: nodes `.11 / .12 / .13`; reserve `.240–.250` for **Cilium L2-announcement VIPs**
|
||
(ingress, per docs/09).
|
||
- Services are reachable **from the host** over the NAT subnet; split-horizon DNS in dev via the
|
||
network's dnsmasq or `/etc/hosts` (PiHole arrives in Phase 6). Public/edge port-forwarding is a prod
|
||
concern — N/A on the dev NAT.
|
||
|
||
## Work breakdown (task-board T-1.1 … T-1.5)
|
||
| Task | Deliverable | Location (proposed) |
|
||
|---|---|---|
|
||
| **T-1.1** OpenTofu libvirt module | 3 VMs + OS/data disks + NAT net + cloud-init seed | `infra/tofu/` |
|
||
| **T-1.2** cloud-init baseline | users, SSH key, sysctls + kernel modules for Cilium/Longhorn | `infra/cloud-init/` |
|
||
| **T-1.3** Ansible k3s + Cilium | HA k3s (3 servers, embedded etcd); Cilium (kube-proxy replacement, Gateway API); disable flannel/kube-proxy/Traefik/servicelb | `infra/ansible/` |
|
||
| **T-1.4** Flux + SOPS bootstrap | `flux bootstrap` → `clusters/dev`; generate age key, install as cluster Secret (private key **never** committed) | `bootstrap/` |
|
||
| **T-1.5** Destroy/rebuild proof | kill a node VM, `tofu apply` rebuild, re-join, no data loss; flip `CLUSTER_AVAILABLE=true`; first L2 conformance | `conformance/` (QA) |
|
||
|
||
Proposed Phase-1 impl tree: `infra/{tofu,ansible,cloud-init}/` and `bootstrap/{sops,flux}/`.
|
||
|
||
## Sequencing
|
||
`tofu apply` (VMs up) → cloud-init baseline → Ansible (k3s + Cilium) → Flux + SOPS bootstrap →
|
||
verify reconcile → destroy/rebuild proof → `CLUSTER_AVAILABLE=true` → conformance L2/L3 begins.
|
||
|
||
## Risks & mitigations
|
||
- **libvirt perms** — use `qemu:///system`; ensure the socket (virtqemud/libvirtd socket-activated) is
|
||
up and the user is in the `libvirt` group. `virsh` already works here, so this is largely satisfied.
|
||
- **k3s+Cilium install order** — install k3s with `--flannel-backend=none --disable-kube-proxy
|
||
--disable=traefik,servicelb`; the cluster is **NotReady until Cilium is up** (expected). Honor the
|
||
spec's Cilium-ownership contract (docs/02): Ansible installs, Flux HelmRelease adopts config, with
|
||
value-parity on `kubeProxyReplacement` / `k8sServiceHost/Port`.
|
||
- **Flux ↔ SOPS chicken-egg** — bootstrap Flux, create the age-key Secret out-of-band, then Flux
|
||
decrypts. Private age key handled outside git (`.gitignore` already guards `age.key`).
|
||
- **RAM ceiling** — trim the dev component set; monitor; don't co-schedule heavy workloads during
|
||
bring-up.
|
||
- **Nested virt (Phase 5, not now)** — verify `/sys/module/kvm_amd/parameters/nested=1` before
|
||
KubeVirt.
|
||
|
||
## Operator decisions / hand-offs — RESOLVED (2026-08-16)
|
||
1. **VM sizing:** ✅ **6 GiB / 4 vCPU × 3** (18 GiB; ~4 GiB host headroom). Full stack in later phases
|
||
will need trimming or more RAM.
|
||
2. **Toolchain:** ✅ **pinned dev container** (`infra/devcontainer/`) — prereqs baked in, not host-
|
||
installed.
|
||
3. **Network:** ✅ **dedicated NAT `k3s-dev` (10.73.0.0/24)**; nodes .11/.12/.13; VIPs .240–.250.
|
||
4. Cloudflare token / PIA creds remain **Phase 2/3** — not needed for Phase 1.
|
||
|
||
## Worker dispatch (on execution go)
|
||
- **Infra/Cluster Engineer** — T-1.1/1.2/1.3 (tofu + cloud-init + Ansible k3s/Cilium).
|
||
- **Security Engineer** — T-1.4 SOPS/age bootstrap.
|
||
- **GitOps Engineer** — T-1.4 Flux bootstrap → `clusters/dev`.
|
||
- **QA/Conformance** — T-1.5 destroy/rebuild test, flip `CLUSTER_AVAILABLE`, first L2 runs.
|