home-network-k3s-impl/infra/PHASE1-SCOPE.md

Phase 1 Scope — dev/test Cluster Bring-up (local KVM)

Owner: Program Manager. Spec refs (read-only): ../home-network-k3s/roadmap/phases.md (Phase 1), docs/02-environments-and-provisioning.md, ADRs 0001 (k3s), 0002 (Cilium), 0004 (Flux), 0007 (SOPS). Status: scoped, awaiting operator go / prerequisite installs.

Objective & exit criteria (roadmap Phase 1)

A 3-node k3s HA cluster on local libvirt/KVM VMs, provisioned entirely by IaC (OpenTofu + cloud-init + Ansible), with Cilium CNI, Flux reconciling from clusters/dev, and the SOPS age key bootstrapped. Exit gates:

1. 3-node k3s HA (embedded etcd) Ready; Cilium healthy (Hubble up); no flannel/kube-proxy/Traefik.
2. Flux reconciling clusters/dev from git; drift corrected.
3. A node VM destroyed and rebuilt from tofu apply re-joins; cluster reconverges, no data loss.
4. Flip CLUSTER_AVAILABLE=true → the conformance L2/L3 behavior tests begin running (first real default-deny / data-survival green — the gate stops skipping).

Environment (verified 2026-08-16)

Resource	Finding	Implication
CPU	24 cores, AMD-V (svm), kvm_amd loaded	ample; nested-virt for KubeVirt (Phase 5) to be confirmed later
RAM	31 GiB total, ~22 GiB available	binding constraint — see sizing
Disk	137 GiB free on /	tight; use thin qcow2, keep dev volumes small
libvirt	12.0.0, functional (existing VMs, default NAT net, pools)	reuse or add a dedicated net
Present	ansible 2.20, virt-install, qemu-img, kubectl, helm, genisoimage	ok
Missing (prereqs)	OpenTofu(+libvirt provider), age, sops, flux CLI, cloud-localds	must install before execution

Toolchain — pinned dev container (decided)

The prereqs are not all Python (OpenTofu/age/sops/flux are Go binaries), so per the operator's rule they are delivered as a pinned dev container, not host installs (and not a venv). See devcontainer/: OpenTofu + dmacvicar/libvirt deps, age, sops, flux, ansible, kubectl, helm, libvirt-client — baked into a Fedora-44 image (matches host libvirt 12). Run with the host libvirt socket + repo mounted:

cd infra/devcontainer && ./dev.sh build && ./dev.sh

This isolates the toolchain, keeps the host clean, and gives every worker the same baseline.

Resource plan (constrained by ~22 GiB RAM / 137 GiB disk)

Per node: 4 vCPU · 6 GiB RAM · 20 GiB OS disk + 40 GiB thin data disk (Longhorn). Totals: 12 vCPU · 18 GiB RAM (~4 GiB host headroom) · ~180 GiB thin-allocated (sparse).

• ⚠️ RAM is tight. The full platform (Longhorn + Cilium + kube-prometheus-stack + Loki + Keycloak/Postgres + cert-manager + …) across 18 GiB is a squeeze. In dev we trim non-essential components (e.g. shorter Prometheus/Loki retention, single-replica where safe) or the operator frees more RAM. This workstation cannot carry prod-scale — which is exactly why prod is 3 dedicated mini-PCs.
• ⚠️ Disk is tight for media; keep Longhorn volumes minimal in dev — bulk media is external anyway (OQ-2). Thin qcow2 means allocation ≫ real usage initially.

Network plan

• A dedicated libvirt NAT network for the cluster (proposed k3s-dev, 10.73.0.0/24) — isolates it from the operator's other VMs, host-reachable. (Alternative: reuse default 192.168.122/24.)
• Static IPs: nodes .11 / .12 / .13; reserve .240–.250 for Cilium L2-announcement VIPs (ingress, per docs/09).
• Services are reachable from the host over the NAT subnet; split-horizon DNS in dev via the network's dnsmasq or /etc/hosts (PiHole arrives in Phase 6). Public/edge port-forwarding is a prod concern — N/A on the dev NAT.

Work breakdown (task-board T-1.1 … T-1.5)

Task	Deliverable	Location (proposed)
T-1.1 OpenTofu libvirt module	3 VMs + OS/data disks + NAT net + cloud-init seed	infra/tofu/
T-1.2 cloud-init baseline	users, SSH key, sysctls + kernel modules for Cilium/Longhorn	infra/cloud-init/
T-1.3 Ansible k3s + Cilium	HA k3s (3 servers, embedded etcd); Cilium (kube-proxy replacement, Gateway API); disable flannel/kube-proxy/Traefik/servicelb	infra/ansible/
T-1.4 Flux + SOPS bootstrap	flux bootstrap → clusters/dev; generate age key, install as cluster Secret (private key never committed)	bootstrap/
T-1.5 Destroy/rebuild proof	kill a node VM, tofu apply rebuild, re-join, no data loss; flip CLUSTER_AVAILABLE=true; first L2 conformance	conformance/ (QA)

Proposed Phase-1 impl tree: infra/{tofu,ansible,cloud-init}/ and bootstrap/{sops,flux}/.

Sequencing

Start host Gitea (bootstrap/gitea/) + push repo → tofu apply (VMs up) → cloud-init baseline → Ansible (k3s + Cilium) → Flux + SOPS bootstrap (Flux pulls from the host Gitea) → verify reconcile → destroy/rebuild proof → CLUSTER_AVAILABLE=true → conformance L2/L3 begins.

Bootstrap source — RESOLVED: a single Gitea container on the host (bootstrap/gitea/), reachable from the cluster at http://10.73.0.1:3000/… (NAT gateway), public repo → Flux needs no creds. Migrates to in-cluster Gitea in Phase 6. Pins CONFIRMED: k3s v1.31.5+k3s1, Cilium chart 1.16.5, single-node k8sServiceHost (dev-acceptable SPOF).

Risks & mitigations

• libvirt perms — use qemu:///system; ensure the socket (virtqemud/libvirtd socket-activated) is up and the user is in the libvirt group. virsh already works here, so this is largely satisfied.
• k3s+Cilium install order — install k3s with --flannel-backend=none --disable-kube-proxy --disable=traefik,servicelb; the cluster is NotReady until Cilium is up (expected). Honor the spec's Cilium-ownership contract (docs/02): Ansible installs, Flux HelmRelease adopts config, with value-parity on kubeProxyReplacement / k8sServiceHost/Port.
• Flux ↔ SOPS chicken-egg — bootstrap Flux, create the age-key Secret out-of-band, then Flux decrypts. Private age key handled outside git (.gitignore already guards age.key).
• RAM ceiling — trim the dev component set; monitor; don't co-schedule heavy workloads during bring-up.
• Nested virt (Phase 5, not now) — verify /sys/module/kvm_amd/parameters/nested=1 before KubeVirt.

Operator decisions / hand-offs — RESOLVED (2026-08-16)

1. VM sizing: ✅ 6 GiB / 4 vCPU × 3 (18 GiB; ~4 GiB host headroom). Full stack in later phases will need trimming or more RAM.
2. Toolchain: ✅ pinned dev container (infra/devcontainer/) — prereqs baked in, not host- installed.
3. Network: ✅ dedicated NAT k3s-dev (10.73.0.0/24); nodes .11/.12/.13; VIPs .240–.250.
4. Cloudflare token / PIA creds remain Phase 2/3 — not needed for Phase 1.

Worker dispatch (on execution go)

• Infra/Cluster Engineer — T-1.1/1.2/1.3 (tofu + cloud-init + Ansible k3s/Cilium).
• Security Engineer — T-1.4 SOPS/age bootstrap.
• GitOps Engineer — T-1.4 Flux bootstrap → clusters/dev.
• QA/Conformance — T-1.5 destroy/rebuild test, flip CLUSTER_AVAILABLE, first L2 runs.