@@ -0,0 +1,92 @@
# Phase 1 Scope — dev/test Cluster Bring-up (local KVM)
**Owner: ** Program Manager. **Spec refs (read-only): ** `../home-network-k3s/roadmap/phases.md`
(Phase 1), `docs/02-environments-and-provisioning.md` , ADRs 0001 (k3s), 0002 (Cilium), 0004 (Flux),
0007 (SOPS). **Status: ** scoped, awaiting operator go / prerequisite installs.
## Objective & exit criteria (roadmap Phase 1)
A **3-node k3s HA cluster ** on local libvirt/KVM VMs, provisioned entirely by IaC
(**OpenTofu + cloud-init + Ansible**), with **Cilium ** CNI, **Flux ** reconciling from `clusters/dev` ,
and the **SOPS age key ** bootstrapped. Exit gates:
1. 3-node k3s HA (embedded etcd) **Ready ** ; Cilium healthy (Hubble up); no flannel/kube-proxy/Traefik.
2. Flux reconciling `clusters/dev` from git; drift corrected.
3. A node VM **destroyed and rebuilt ** from `tofu apply` re-joins; cluster reconverges, **no data loss ** .
4. Flip `CLUSTER_AVAILABLE=true` → the conformance **L2/L3 behavior tests begin running ** (first real
default-deny / data-survival green — the gate stops skipping).
## Environment (verified 2026-08-16)
| Resource | Finding | Implication |
|---|---|---|
| CPU | 24 cores, AMD-V (`svm` ), `kvm_amd` loaded | ample; nested-virt for KubeVirt (Phase 5) to be confirmed later |
| RAM | 31 GiB total, * * ~22 GiB available** | **binding constraint ** — see sizing |
| Disk | 137 GiB free on `/` | tight; use thin qcow2, keep dev volumes small |
| libvirt | 12.0.0, functional (existing VMs, `default` NAT net, pools) | reuse or add a dedicated net |
| Present | ansible 2.20, virt-install, qemu-img, kubectl, helm, genisoimage | ok |
| **Missing (prereqs) ** | **OpenTofu(+libvirt provider), age, sops, flux CLI ** , cloud-localds | must install before execution |
## Prerequisites to install (before execution)
- **OpenTofu** + the * * `dmacvicar/libvirt` ** provider — node-VM IaC.
- **age** + **sops ** — the SOPS secret bootstrap (ADR-0007).
- **flux** CLI — Flux bootstrap.
- (`cloud-localds` optional — the tofu libvirt provider builds cloud-init disks natively.)
## Resource plan (constrained by ~22 GiB RAM / 137 GiB disk)
Per node: **4 vCPU · 6 GiB RAM · 20 GiB OS disk + 40 GiB thin data disk ** (Longhorn).
Totals: 12 vCPU · **18 GiB RAM ** (~4 GiB host headroom) · ~180 GiB thin-allocated (sparse).
- ⚠️ **RAM is tight. ** The * full * platform (Longhorn + Cilium + kube-prometheus-stack + Loki +
Keycloak/Postgres + cert-manager + …) across 18 GiB is a squeeze. In dev we **trim non-essential
components** (e.g. shorter Prometheus/Loki retention, single-replica where safe) or the operator
frees more RAM. This workstation cannot carry prod-scale — which is exactly why prod is 3 dedicated
mini-PCs.
- ⚠️ **Disk is tight ** for media; keep Longhorn volumes minimal in dev — bulk media is external anyway
(OQ-2). Thin qcow2 means allocation ≫ real usage initially.
## Network plan
- A **dedicated libvirt NAT network ** for the cluster (proposed `k3s-dev` , `10.73.0.0/24` ) — isolates
it from the operator's other VMs, host-reachable. (Alternative: reuse `default` 192.168.122/24.)
- **Static IPs**: nodes `.11 / .12 / .13` ; reserve `.240– for **Cilium L2-announcement VIPs **
(ingress, per docs/09).
- Services are reachable **from the host ** over the NAT subnet; split-horizon DNS in dev via the
network's dnsmasq or `/etc/hosts` (PiHole arrives in Phase 6). Public/edge port-forwarding is a prod
concern — N/A on the dev NAT.
## Work breakdown (task-board T-1.1 … T-1.5)
| Task | Deliverable | Location (proposed) |
|---|---|---|
| **T-1.1 ** OpenTofu libvirt module | 3 VMs + OS/data disks + NAT net + cloud-init seed | `infra/tofu/` |
| **T-1.2 ** cloud-init baseline | users, SSH key, sysctls + kernel modules for Cilium/Longhorn | `infra/cloud-init/` |
| **T-1.3 ** Ansible k3s + Cilium | HA k3s (3 servers, embedded etcd); Cilium (kube-proxy replacement, Gateway API); disable flannel/kube-proxy/Traefik/servicelb | `infra/ansible/` |
| **T-1.4 ** Flux + SOPS bootstrap | `flux bootstrap` → `clusters/dev` ; generate age key, install as cluster Secret (private key **never ** committed) | `bootstrap/` |
| **T-1.5 ** Destroy/rebuild proof | kill a node VM, `tofu apply` rebuild, re-join, no data loss; flip `CLUSTER_AVAILABLE=true` ; first L2 conformance | `conformance/` (QA) |
Proposed Phase-1 impl tree: `infra/{tofu,ansible,cloud-init}/` and `bootstrap/{sops,flux}/` .
## Sequencing
`tofu apply` (VMs up) → cloud-init baseline → Ansible (k3s + Cilium) → Flux + SOPS bootstrap →
verify reconcile → destroy/rebuild proof → `CLUSTER_AVAILABLE=true` → conformance L2/L3 begins.
## Risks & mitigations
- **libvirt perms** — use `qemu:///system` ; ensure the socket (virtqemud/libvirtd socket-activated) is
up and the user is in the `libvirt` group. `virsh` already works here, so this is largely satisfied.
- **k3s+Cilium install order** — install k3s with `--flannel-backend=none --disable-kube-proxy
--disable=traefik,servicelb`; the cluster is **NotReady until Cilium is up** (expected). Honor the
spec's Cilium-ownership contract (docs/02): Ansible installs, Flux HelmRelease adopts config, with
value-parity on ` kubeProxyReplacement` / ` k8sServiceHost/Port`.
- **Flux ↔ SOPS chicken-egg** — bootstrap Flux, create the age-key Secret out-of-band, then Flux
decrypts. Private age key handled outside git (` .gitignore` already guards ` age.key`).
- **RAM ceiling** — trim the dev component set; monitor; don't co-schedule heavy workloads during
bring-up.
- **Nested virt (Phase 5, not now)** — verify ` /sys/module/kvm_amd/parameters/nested=1` before
KubeVirt.
## Operator decisions / hand-offs needed
1. **Approve VM sizing** (6 GiB / 4 vCPU ×
2. **Approve installing the prereqs** (OpenTofu + libvirt provider, age, sops, flux).
3. **Network**: dedicated NAT ` k3s-dev` (recommended) vs reuse ` default`.
4. Cloudflare token / PIA creds are **Phase 2/3** — not needed for Phase 1.
## Worker dispatch (on execution go)
- **Infra/Cluster Engineer** — T-1.1/1.2/1.3 (tofu + cloud-init + Ansible k3s/Cilium).
- **Security Engineer** — T-1.4 SOPS/age bootstrap.
- **GitOps Engineer** — T-1.4 Flux bootstrap → ` clusters/dev`.
- **QA/Conformance** — T-1.5 destroy/rebuild test, flip ` CLUSTER_AVAILABLE`, first L2 runs.
