From 5ba9238cc1ce190201cab723e2e1f465f5ae168f Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 16 Aug 2026 00:48:48 -0500 Subject: [PATCH] Scope Phase 1: dev/test cluster bring-up on local KVM Add infra/PHASE1-SCOPE.md (grounded in local recon: 24c/22GiB-avail/137G, AMD-V+libvirt ready; OpenTofu/age/sops/flux missing) + infra/ and bootstrap/ structure stubs. Maps T-1.1..T-1.5; resource/network plan; risks; acceptance gates; operator decisions. No spec edits; gate green. Co-Authored-By: Claude Opus 4.8 (1M context) --- bootstrap/README.md | 4 ++ bootstrap/flux/README.md | 3 ++ bootstrap/sops/README.md | 3 ++ infra/PHASE1-SCOPE.md | 92 ++++++++++++++++++++++++++++++++++++++ infra/README.md | 4 ++ infra/ansible/README.md | 3 ++ infra/cloud-init/README.md | 3 ++ infra/tofu/README.md | 3 ++ 8 files changed, 115 insertions(+) create mode 100644 bootstrap/README.md create mode 100644 bootstrap/flux/README.md create mode 100644 bootstrap/sops/README.md create mode 100644 infra/PHASE1-SCOPE.md create mode 100644 infra/README.md create mode 100644 infra/ansible/README.md create mode 100644 infra/cloud-init/README.md create mode 100644 infra/tofu/README.md diff --git a/bootstrap/README.md b/bootstrap/README.md new file mode 100644 index 0000000..7de6c26 --- /dev/null +++ b/bootstrap/README.md @@ -0,0 +1,4 @@ +# bootstrap/ — Flux + SOPS age-key bootstrap (Phase 1 T-1.4) + +sops/: age key generation + Flux SOPS decryption wiring (private key NEVER committed). +flux/: `flux bootstrap` config pointing at clusters/dev. Spec: ADR-0004, ADR-0007. diff --git a/bootstrap/flux/README.md b/bootstrap/flux/README.md new file mode 100644 index 0000000..37f222e --- /dev/null +++ b/bootstrap/flux/README.md @@ -0,0 +1,3 @@ +# bootstrap/flux + +_Phase 1 (T-1.x) — to be authored on execution go. See infra/PHASE1-SCOPE.md._ diff --git a/bootstrap/sops/README.md b/bootstrap/sops/README.md new file mode 100644 index 0000000..5ece3a6 --- /dev/null +++ b/bootstrap/sops/README.md @@ -0,0 +1,3 @@ +# bootstrap/sops + +_Phase 1 (T-1.x) — to be authored on execution go. See infra/PHASE1-SCOPE.md._ diff --git a/infra/PHASE1-SCOPE.md b/infra/PHASE1-SCOPE.md new file mode 100644 index 0000000..1e36c29 --- /dev/null +++ b/infra/PHASE1-SCOPE.md @@ -0,0 +1,92 @@ +# Phase 1 Scope — dev/test Cluster Bring-up (local KVM) + +**Owner:** Program Manager. **Spec refs (read-only):** `../home-network-k3s/roadmap/phases.md` +(Phase 1), `docs/02-environments-and-provisioning.md`, ADRs 0001 (k3s), 0002 (Cilium), 0004 (Flux), +0007 (SOPS). **Status:** scoped, awaiting operator go / prerequisite installs. + +## Objective & exit criteria (roadmap Phase 1) +A **3-node k3s HA cluster** on local libvirt/KVM VMs, provisioned entirely by IaC +(**OpenTofu + cloud-init + Ansible**), with **Cilium** CNI, **Flux** reconciling from `clusters/dev`, +and the **SOPS age key** bootstrapped. Exit gates: +1. 3-node k3s HA (embedded etcd) **Ready**; Cilium healthy (Hubble up); no flannel/kube-proxy/Traefik. +2. Flux reconciling `clusters/dev` from git; drift corrected. +3. A node VM **destroyed and rebuilt** from `tofu apply` re-joins; cluster reconverges, **no data loss**. +4. Flip `CLUSTER_AVAILABLE=true` → the conformance **L2/L3 behavior tests begin running** (first real + default-deny / data-survival green — the gate stops skipping). + +## Environment (verified 2026-08-16) +| Resource | Finding | Implication | +|---|---|---| +| CPU | 24 cores, AMD-V (`svm`), `kvm_amd` loaded | ample; nested-virt for KubeVirt (Phase 5) to be confirmed later | +| RAM | 31 GiB total, **~22 GiB available** | **binding constraint** — see sizing | +| Disk | 137 GiB free on `/` | tight; use thin qcow2, keep dev volumes small | +| libvirt | 12.0.0, functional (existing VMs, `default` NAT net, pools) | reuse or add a dedicated net | +| Present | ansible 2.20, virt-install, qemu-img, kubectl, helm, genisoimage | ok | +| **Missing (prereqs)** | **OpenTofu(+libvirt provider), age, sops, flux CLI**, cloud-localds | must install before execution | + +## Prerequisites to install (before execution) +- **OpenTofu** + the **`dmacvicar/libvirt`** provider — node-VM IaC. +- **age** + **sops** — the SOPS secret bootstrap (ADR-0007). +- **flux** CLI — Flux bootstrap. +- (`cloud-localds` optional — the tofu libvirt provider builds cloud-init disks natively.) + +## Resource plan (constrained by ~22 GiB RAM / 137 GiB disk) +Per node: **4 vCPU · 6 GiB RAM · 20 GiB OS disk + 40 GiB thin data disk** (Longhorn). +Totals: 12 vCPU · **18 GiB RAM** (~4 GiB host headroom) · ~180 GiB thin-allocated (sparse). +- ⚠️ **RAM is tight.** The *full* platform (Longhorn + Cilium + kube-prometheus-stack + Loki + + Keycloak/Postgres + cert-manager + …) across 18 GiB is a squeeze. In dev we **trim non-essential + components** (e.g. shorter Prometheus/Loki retention, single-replica where safe) or the operator + frees more RAM. This workstation cannot carry prod-scale — which is exactly why prod is 3 dedicated + mini-PCs. +- ⚠️ **Disk is tight** for media; keep Longhorn volumes minimal in dev — bulk media is external anyway + (OQ-2). Thin qcow2 means allocation ≫ real usage initially. + +## Network plan +- A **dedicated libvirt NAT network** for the cluster (proposed `k3s-dev`, `10.73.0.0/24`) — isolates + it from the operator's other VMs, host-reachable. (Alternative: reuse `default` 192.168.122/24.) +- **Static IPs**: nodes `.11 / .12 / .13`; reserve `.240–.250` for **Cilium L2-announcement VIPs** + (ingress, per docs/09). +- Services are reachable **from the host** over the NAT subnet; split-horizon DNS in dev via the + network's dnsmasq or `/etc/hosts` (PiHole arrives in Phase 6). Public/edge port-forwarding is a prod + concern — N/A on the dev NAT. + +## Work breakdown (task-board T-1.1 … T-1.5) +| Task | Deliverable | Location (proposed) | +|---|---|---| +| **T-1.1** OpenTofu libvirt module | 3 VMs + OS/data disks + NAT net + cloud-init seed | `infra/tofu/` | +| **T-1.2** cloud-init baseline | users, SSH key, sysctls + kernel modules for Cilium/Longhorn | `infra/cloud-init/` | +| **T-1.3** Ansible k3s + Cilium | HA k3s (3 servers, embedded etcd); Cilium (kube-proxy replacement, Gateway API); disable flannel/kube-proxy/Traefik/servicelb | `infra/ansible/` | +| **T-1.4** Flux + SOPS bootstrap | `flux bootstrap` → `clusters/dev`; generate age key, install as cluster Secret (private key **never** committed) | `bootstrap/` | +| **T-1.5** Destroy/rebuild proof | kill a node VM, `tofu apply` rebuild, re-join, no data loss; flip `CLUSTER_AVAILABLE=true`; first L2 conformance | `conformance/` (QA) | + +Proposed Phase-1 impl tree: `infra/{tofu,ansible,cloud-init}/` and `bootstrap/{sops,flux}/`. + +## Sequencing +`tofu apply` (VMs up) → cloud-init baseline → Ansible (k3s + Cilium) → Flux + SOPS bootstrap → +verify reconcile → destroy/rebuild proof → `CLUSTER_AVAILABLE=true` → conformance L2/L3 begins. + +## Risks & mitigations +- **libvirt perms** — use `qemu:///system`; ensure the socket (virtqemud/libvirtd socket-activated) is + up and the user is in the `libvirt` group. `virsh` already works here, so this is largely satisfied. +- **k3s+Cilium install order** — install k3s with `--flannel-backend=none --disable-kube-proxy + --disable=traefik,servicelb`; the cluster is **NotReady until Cilium is up** (expected). Honor the + spec's Cilium-ownership contract (docs/02): Ansible installs, Flux HelmRelease adopts config, with + value-parity on `kubeProxyReplacement` / `k8sServiceHost/Port`. +- **Flux ↔ SOPS chicken-egg** — bootstrap Flux, create the age-key Secret out-of-band, then Flux + decrypts. Private age key handled outside git (`.gitignore` already guards `age.key`). +- **RAM ceiling** — trim the dev component set; monitor; don't co-schedule heavy workloads during + bring-up. +- **Nested virt (Phase 5, not now)** — verify `/sys/module/kvm_amd/parameters/nested=1` before + KubeVirt. + +## Operator decisions / hand-offs needed +1. **Approve VM sizing** (6 GiB / 4 vCPU × 3) given the RAM ceiling — or free more RAM for larger. +2. **Approve installing the prereqs** (OpenTofu + libvirt provider, age, sops, flux). +3. **Network**: dedicated NAT `k3s-dev` (recommended) vs reuse `default`. +4. Cloudflare token / PIA creds are **Phase 2/3** — not needed for Phase 1. + +## Worker dispatch (on execution go) +- **Infra/Cluster Engineer** — T-1.1/1.2/1.3 (tofu + cloud-init + Ansible k3s/Cilium). +- **Security Engineer** — T-1.4 SOPS/age bootstrap. +- **GitOps Engineer** — T-1.4 Flux bootstrap → `clusters/dev`. +- **QA/Conformance** — T-1.5 destroy/rebuild test, flip `CLUSTER_AVAILABLE`, first L2 runs. diff --git a/infra/README.md b/infra/README.md new file mode 100644 index 0000000..a0fbc5b --- /dev/null +++ b/infra/README.md @@ -0,0 +1,4 @@ +# infra/ — Phase 1 node provisioning (local KVM) + +OpenTofu (libvirt) + cloud-init + Ansible bring up the 3-node k3s/Cilium cluster. +See PHASE1-SCOPE.md. Spec: ../home-network-k3s/docs/02-environments-and-provisioning.md. diff --git a/infra/ansible/README.md b/infra/ansible/README.md new file mode 100644 index 0000000..d4a2eb8 --- /dev/null +++ b/infra/ansible/README.md @@ -0,0 +1,3 @@ +# infra/ansible + +_Phase 1 (T-1.x) — to be authored on execution go. See infra/PHASE1-SCOPE.md._ diff --git a/infra/cloud-init/README.md b/infra/cloud-init/README.md new file mode 100644 index 0000000..c860e7e --- /dev/null +++ b/infra/cloud-init/README.md @@ -0,0 +1,3 @@ +# infra/cloud-init + +_Phase 1 (T-1.x) — to be authored on execution go. See infra/PHASE1-SCOPE.md._ diff --git a/infra/tofu/README.md b/infra/tofu/README.md new file mode 100644 index 0000000..9671c55 --- /dev/null +++ b/infra/tofu/README.md @@ -0,0 +1,3 @@ +# infra/tofu + +_Phase 1 (T-1.x) — to be authored on execution go. See infra/PHASE1-SCOPE.md._