home-network-k3s-impl/infra/PHASE1-SCOPE.md

Phase 1 Scope — dev/test Cluster Bring-up (local KVM)

Owner: Program Manager. Spec refs (read-only): ../home-network-k3s/roadmap/phases.md (Phase 1), docs/02-environments-and-provisioning.md, ADRs 0001 (k3s), 0002 (Cilium), 0004 (Flux), 0007 (SOPS). Status: scoped, awaiting operator go / prerequisite installs.

Objective & exit criteria (roadmap Phase 1)

A 3-node k3s HA cluster on local libvirt/KVM VMs, provisioned entirely by IaC (OpenTofu + cloud-init + Ansible), with Cilium CNI, Flux reconciling from clusters/dev, and the SOPS age key bootstrapped. Exit gates:

1. 3-node k3s HA (embedded etcd) Ready; Cilium healthy (Hubble up); no flannel/kube-proxy/Traefik.
2. Flux reconciling clusters/dev from git; drift corrected.
3. A node VM destroyed and rebuilt from tofu apply re-joins; cluster reconverges, no data loss.
4. Flip CLUSTER_AVAILABLE=true → the conformance L2/L3 behavior tests begin running (first real default-deny / data-survival green — the gate stops skipping).

Environment (verified 2026-08-16)

Resource	Finding	Implication
CPU	24 cores, AMD-V (svm), kvm_amd loaded	ample; nested-virt for KubeVirt (Phase 5) to be confirmed later
RAM	31 GiB total, ~22 GiB available	binding constraint — see sizing
Disk	137 GiB free on /	tight; use thin qcow2, keep dev volumes small
libvirt	12.0.0, functional (existing VMs, default NAT net, pools)	reuse or add a dedicated net
Present	ansible 2.20, virt-install, qemu-img, kubectl, helm, genisoimage	ok
Missing (prereqs)	OpenTofu(+libvirt provider), age, sops, flux CLI, cloud-localds	must install before execution

Prerequisites to install (before execution)

• OpenTofu + the dmacvicar/libvirt provider — node-VM IaC.
• age + sops — the SOPS secret bootstrap (ADR-0007).
• flux CLI — Flux bootstrap.
• (cloud-localds optional — the tofu libvirt provider builds cloud-init disks natively.)

Resource plan (constrained by ~22 GiB RAM / 137 GiB disk)

Per node: 4 vCPU · 6 GiB RAM · 20 GiB OS disk + 40 GiB thin data disk (Longhorn). Totals: 12 vCPU · 18 GiB RAM (~4 GiB host headroom) · ~180 GiB thin-allocated (sparse).

• ⚠️ RAM is tight. The full platform (Longhorn + Cilium + kube-prometheus-stack + Loki + Keycloak/Postgres + cert-manager + …) across 18 GiB is a squeeze. In dev we trim non-essential components (e.g. shorter Prometheus/Loki retention, single-replica where safe) or the operator frees more RAM. This workstation cannot carry prod-scale — which is exactly why prod is 3 dedicated mini-PCs.
• ⚠️ Disk is tight for media; keep Longhorn volumes minimal in dev — bulk media is external anyway (OQ-2). Thin qcow2 means allocation ≫ real usage initially.

Network plan

• A dedicated libvirt NAT network for the cluster (proposed k3s-dev, 10.73.0.0/24) — isolates it from the operator's other VMs, host-reachable. (Alternative: reuse default 192.168.122/24.)
• Static IPs: nodes .11 / .12 / .13; reserve .240–.250 for Cilium L2-announcement VIPs (ingress, per docs/09).
• Services are reachable from the host over the NAT subnet; split-horizon DNS in dev via the network's dnsmasq or /etc/hosts (PiHole arrives in Phase 6). Public/edge port-forwarding is a prod concern — N/A on the dev NAT.

Work breakdown (task-board T-1.1 … T-1.5)

Task	Deliverable	Location (proposed)
T-1.1 OpenTofu libvirt module	3 VMs + OS/data disks + NAT net + cloud-init seed	infra/tofu/
T-1.2 cloud-init baseline	users, SSH key, sysctls + kernel modules for Cilium/Longhorn	infra/cloud-init/
T-1.3 Ansible k3s + Cilium	HA k3s (3 servers, embedded etcd); Cilium (kube-proxy replacement, Gateway API); disable flannel/kube-proxy/Traefik/servicelb	infra/ansible/
T-1.4 Flux + SOPS bootstrap	flux bootstrap → clusters/dev; generate age key, install as cluster Secret (private key never committed)	bootstrap/
T-1.5 Destroy/rebuild proof	kill a node VM, tofu apply rebuild, re-join, no data loss; flip CLUSTER_AVAILABLE=true; first L2 conformance	conformance/ (QA)

Proposed Phase-1 impl tree: infra/{tofu,ansible,cloud-init}/ and bootstrap/{sops,flux}/.

Sequencing

tofu apply (VMs up) → cloud-init baseline → Ansible (k3s + Cilium) → Flux + SOPS bootstrap → verify reconcile → destroy/rebuild proof → CLUSTER_AVAILABLE=true → conformance L2/L3 begins.

Risks & mitigations

• libvirt perms — use qemu:///system; ensure the socket (virtqemud/libvirtd socket-activated) is up and the user is in the libvirt group. virsh already works here, so this is largely satisfied.
• k3s+Cilium install order — install k3s with --flannel-backend=none --disable-kube-proxy --disable=traefik,servicelb; the cluster is NotReady until Cilium is up (expected). Honor the spec's Cilium-ownership contract (docs/02): Ansible installs, Flux HelmRelease adopts config, with value-parity on kubeProxyReplacement / k8sServiceHost/Port.
• Flux ↔ SOPS chicken-egg — bootstrap Flux, create the age-key Secret out-of-band, then Flux decrypts. Private age key handled outside git (.gitignore already guards age.key).
• RAM ceiling — trim the dev component set; monitor; don't co-schedule heavy workloads during bring-up.
• Nested virt (Phase 5, not now) — verify /sys/module/kvm_amd/parameters/nested=1 before KubeVirt.

Operator decisions / hand-offs needed

1. Approve VM sizing (6 GiB / 4 vCPU × 3) given the RAM ceiling — or free more RAM for larger.
2. Approve installing the prereqs (OpenTofu + libvirt provider, age, sops, flux).
3. Network: dedicated NAT k3s-dev (recommended) vs reuse default.
4. Cloudflare token / PIA creds are Phase 2/3 — not needed for Phase 1.

Worker dispatch (on execution go)

• Infra/Cluster Engineer — T-1.1/1.2/1.3 (tofu + cloud-init + Ansible k3s/Cilium).
• Security Engineer — T-1.4 SOPS/age bootstrap.
• GitOps Engineer — T-1.4 Flux bootstrap → clusters/dev.
• QA/Conformance — T-1.5 destroy/rebuild test, flip CLUSTER_AVAILABLE, first L2 runs.