home-network-k3s-impl/infra/PHASE1-SCOPE.md

# Phase 1 Scope — dev/test Cluster Bring-up (local KVM)

**Owner:** Program Manager. **Spec refs (read-only):** `../home-network-k3s/roadmap/phases.md`
(Phase 1), `docs/02-environments-and-provisioning.md`, ADRs 0001 (k3s), 0002 (Cilium), 0004 (Flux),
0007 (SOPS). **Status:** scoped, awaiting operator go / prerequisite installs.

## Objective & exit criteria (roadmap Phase 1)
A **3-node k3s HA cluster** on local libvirt/KVM VMs, provisioned entirely by IaC
(**OpenTofu + cloud-init + Ansible**), with **Cilium** CNI, **Flux** reconciling from `clusters/dev`,
and the **SOPS age key** bootstrapped. Exit gates:
1. 3-node k3s HA (embedded etcd) **Ready**; Cilium healthy (Hubble up); no flannel/kube-proxy/Traefik.
2. Flux reconciling `clusters/dev` from git; drift corrected.
3. A node VM **destroyed and rebuilt** from `tofu apply` re-joins; cluster reconverges, **no data loss**.
4. Flip `CLUSTER_AVAILABLE=true` → the conformance **L2/L3 behavior tests begin running** (first real
   default-deny / data-survival green — the gate stops skipping).

## Environment (verified 2026-08-16)
| Resource | Finding | Implication |
|---|---|---|
| CPU | 24 cores, AMD-V (`svm`), `kvm_amd` loaded | ample; nested-virt for KubeVirt (Phase 5) to be confirmed later |
| RAM | 31 GiB total, **~22 GiB available** | **binding constraint** — see sizing |
| Disk | 137 GiB free on `/` | tight; use thin qcow2, keep dev volumes small |
| libvirt | 12.0.0, functional (existing VMs, `default` NAT net, pools) | reuse or add a dedicated net |
| Present | ansible 2.20, virt-install, qemu-img, kubectl, helm, genisoimage | ok |
| **Missing (prereqs)** | **OpenTofu(+libvirt provider), age, sops, flux CLI**, cloud-localds | must install before execution |

## Toolchain — pinned dev container (decided)
The prereqs are **not all Python** (OpenTofu/age/sops/flux are Go binaries), so per the operator's
rule they are delivered as a **pinned dev container**, not host installs (and not a venv). See
[`devcontainer/`](devcontainer/): OpenTofu + `dmacvicar/libvirt` deps, age, sops, flux, ansible,
kubectl, helm, libvirt-client — baked into a Fedora-44 image (matches host libvirt 12). Run with the
host libvirt socket + repo mounted:
```
cd infra/devcontainer && ./dev.sh build && ./dev.sh
```
This isolates the toolchain, keeps the host clean, and gives every worker the same baseline.

## Resource plan (constrained by ~22 GiB RAM / 137 GiB disk)
Per node: **4 vCPU · 6 GiB RAM · 20 GiB OS disk + 40 GiB thin data disk** (Longhorn).
Totals: 12 vCPU · **18 GiB RAM** (~4 GiB host headroom) · ~180 GiB thin-allocated (sparse).
- ⚠️ **RAM is tight.** The *full* platform (Longhorn + Cilium + kube-prometheus-stack + Loki +
  Keycloak/Postgres + cert-manager + …) across 18 GiB is a squeeze. In dev we **trim non-essential
  components** (e.g. shorter Prometheus/Loki retention, single-replica where safe) or the operator
  frees more RAM. This workstation cannot carry prod-scale — which is exactly why prod is 3 dedicated
  mini-PCs.
- ⚠️ **Disk is tight** for media; keep Longhorn volumes minimal in dev — bulk media is external anyway
  (OQ-2). Thin qcow2 means allocation ≫ real usage initially.

## Network plan
- A **dedicated libvirt NAT network** for the cluster (proposed `k3s-dev`, `10.73.0.0/24`) — isolates
  it from the operator's other VMs, host-reachable. (Alternative: reuse `default` 192.168.122/24.)
- **Static IPs**: nodes `.11 / .12 / .13`; reserve `.240–.250` for **Cilium L2-announcement VIPs**
  (ingress, per docs/09).
- Services are reachable **from the host** over the NAT subnet; split-horizon DNS in dev via the
  network's dnsmasq or `/etc/hosts` (PiHole arrives in Phase 6). Public/edge port-forwarding is a prod
  concern — N/A on the dev NAT.

## Work breakdown (task-board T-1.1 … T-1.5)
| Task | Deliverable | Location (proposed) |
|---|---|---|
| **T-1.1** OpenTofu libvirt module | 3 VMs + OS/data disks + NAT net + cloud-init seed | `infra/tofu/` |
| **T-1.2** cloud-init baseline | users, SSH key, sysctls + kernel modules for Cilium/Longhorn | `infra/cloud-init/` |
| **T-1.3** Ansible k3s + Cilium | HA k3s (3 servers, embedded etcd); Cilium (kube-proxy replacement, Gateway API); disable flannel/kube-proxy/Traefik/servicelb | `infra/ansible/` |
| **T-1.4** Flux + SOPS bootstrap | `flux bootstrap` → `clusters/dev`; generate age key, install as cluster Secret (private key **never** committed) | `bootstrap/` |
| **T-1.5** Destroy/rebuild proof | kill a node VM, `tofu apply` rebuild, re-join, no data loss; flip `CLUSTER_AVAILABLE=true`; first L2 conformance | `conformance/` (QA) |

Proposed Phase-1 impl tree: `infra/{tofu,ansible,cloud-init}/` and `bootstrap/{sops,flux}/`.

## Sequencing
**Start host Gitea** (`bootstrap/gitea/`) + push repo → `tofu apply` (VMs up) → cloud-init baseline →
Ansible (k3s + Cilium) → Flux + SOPS bootstrap (Flux pulls from the host Gitea) → verify reconcile →
destroy/rebuild proof → `CLUSTER_AVAILABLE=true` → conformance L2/L3 begins.

**Bootstrap source — RESOLVED:** a single **Gitea container on the host** (`bootstrap/gitea/`),
reachable from the cluster at `http://10.73.0.1:3000/…` (NAT gateway), public repo → Flux needs no
creds. Migrates to in-cluster Gitea in Phase 6. **Pins CONFIRMED:** k3s `v1.31.5+k3s1`, Cilium chart
`1.16.5`, single-node `k8sServiceHost` (dev-acceptable SPOF).

## Risks & mitigations
- **libvirt perms** — use `qemu:///system`; ensure the socket (virtqemud/libvirtd socket-activated) is
  up and the user is in the `libvirt` group. `virsh` already works here, so this is largely satisfied.
- **k3s+Cilium install order** — install k3s with `--flannel-backend=none --disable-kube-proxy
  --disable=traefik,servicelb`; the cluster is **NotReady until Cilium is up** (expected). Honor the
  spec's Cilium-ownership contract (docs/02): Ansible installs, Flux HelmRelease adopts config, with
  value-parity on `kubeProxyReplacement` / `k8sServiceHost/Port`.
- **Flux ↔ SOPS chicken-egg** — bootstrap Flux, create the age-key Secret out-of-band, then Flux
  decrypts. Private age key handled outside git (`.gitignore` already guards `age.key`).
- **RAM ceiling** — trim the dev component set; monitor; don't co-schedule heavy workloads during
  bring-up.
- **Nested virt (Phase 5, not now)** — verify `/sys/module/kvm_amd/parameters/nested=1` before
  KubeVirt.

## Operator decisions / hand-offs — RESOLVED (2026-08-16)
1. **VM sizing:** ✅ **6 GiB / 4 vCPU × 3** (18 GiB; ~4 GiB host headroom). Full stack in later phases
   will need trimming or more RAM.
2. **Toolchain:** ✅ **pinned dev container** (`infra/devcontainer/`) — prereqs baked in, not host-
   installed.
3. **Network:** ✅ **dedicated NAT `k3s-dev` (10.73.0.0/24)**; nodes .11/.12/.13; VIPs .240–.250.
4. Cloudflare token / PIA creds remain **Phase 2/3** — not needed for Phase 1.

## Worker dispatch (on execution go)
- **Infra/Cluster Engineer** — T-1.1/1.2/1.3 (tofu + cloud-init + Ansible k3s/Cilium).
- **Security Engineer** — T-1.4 SOPS/age bootstrap.
- **GitOps Engineer** — T-1.4 Flux bootstrap → `clusters/dev`.
- **QA/Conformance** — T-1.5 destroy/rebuild test, flip `CLUSTER_AVAILABLE`, first L2 runs.