jakman85andClaude Opus 5
12c40d8c9c
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.
Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.
README.md
- Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
- Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
- Real conformance number (36/51, 70.6%) and how to regenerate it.
- Devcontainer workflow; note that the gate was red and why.
infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
the toolchain, off-repo state that must not be lost, and the fact that Flux's
git source is an app inside the cluster (`git push origin main` IS the deploy).
infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
rationale are still worth keeping; the libvirt execution steps are not.
infra/README.md — Proxmox, plus a map of the subtree and where the cluster
actually lives (it is not on this workstation).
infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
talks to an HTTP API. Everything runs in the container now. Documents the full
CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
and the read-only spec repo especially — without the latter, spec-immutability
silently skips instead of verifying).
infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
against the running nodes), pfSense NAT marked done, and a warning that
terraform.tfstate is absent on a fresh workstation — an apply with empty state
would try to recreate the three live nodes.
infra/tofu/variables.tf — section comment disagreed with its own defaults.
infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
domains that do not exist; behaviour unchanged, but it no longer lies.
apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
since been dropped. Now: live status, 40Gi, resolved open questions marked
resolved, and the two constraints the next person will otherwise rediscover —
KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
postBuild.substituteFrom (envsubst corrupts the cloud-init).
DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
the Flux source cutover, CI going live in-cluster, ten app onboardings and the
renderer bugs they exposed, the wave split (verified: DBs adopted, not
recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
VM, and kubectl SSO. Plus verified live state and the carried-forward open list.
Two corrections to the record while verifying:
- kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
[oidc:platform-admins]), so the rolling restart 3472f1d deferred has
happened.
- UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
and the CNP explicitly permits cluster -> :11443, so it is genuinely not
listening rather than a policy artifact.
Docs and comments only; no manifest semantics change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
120 lines
8.0 KiB
Markdown
120 lines
8.0 KiB
Markdown
# Phase 1 Scope — dev/test Cluster Bring-up (local KVM)
|
||
|
||
> ## ⚠️ HISTORICAL — DO NOT EXECUTE
|
||
>
|
||
> **Phase 1 is complete.** This document is the *original* scoping for the first dev cluster, which
|
||
> ran on **local libvirt/KVM VMs on the workstation** (`10.73.0.0/24`). That cluster was retired: it
|
||
> was constrained by the workstation's ~22 GiB RAM and 137 GiB disk, and the dev cluster was migrated
|
||
> to a **Proxmox VE** host (nodes `192.168.1.50/.51/.52`, 6 vCPU / 16 GiB each).
|
||
>
|
||
> Kept for the record — the exit criteria, sizing rationale, and risk analysis below are still the
|
||
> reasoning behind the current design. **Everything about libvirt, `virsh`, the `dmacvicar/libvirt`
|
||
> provider, `10.73.0.x`, and the "missing prereqs" table is superseded.**
|
||
>
|
||
> For the cluster as it exists today, read **[`PHASE1-STATUS.md`](PHASE1-STATUS.md)**.
|
||
|
||
**Owner:** Program Manager. **Spec refs (read-only):** `../home-network-k3s/roadmap/phases.md`
|
||
(Phase 1), `docs/02-environments-and-provisioning.md`, ADRs 0001 (k3s), 0002 (Cilium), 0004 (Flux),
|
||
0007 (SOPS). **Status (as written, 2026-08-16):** scoped, awaiting operator go / prerequisite
|
||
installs. **Status now:** delivered, then superseded by the Proxmox migration.
|
||
|
||
## Objective & exit criteria (roadmap Phase 1)
|
||
A **3-node k3s HA cluster** on local libvirt/KVM VMs, provisioned entirely by IaC
|
||
(**OpenTofu + cloud-init + Ansible**), with **Cilium** CNI, **Flux** reconciling from `clusters/dev`,
|
||
and the **SOPS age key** bootstrapped. Exit gates:
|
||
1. 3-node k3s HA (embedded etcd) **Ready**; Cilium healthy (Hubble up); no flannel/kube-proxy/Traefik.
|
||
2. Flux reconciling `clusters/dev` from git; drift corrected.
|
||
3. A node VM **destroyed and rebuilt** from `tofu apply` re-joins; cluster reconverges, **no data loss**.
|
||
4. Flip `CLUSTER_AVAILABLE=true` → the conformance **L2/L3 behavior tests begin running** (first real
|
||
default-deny / data-survival green — the gate stops skipping).
|
||
|
||
## Environment (verified 2026-08-16)
|
||
| Resource | Finding | Implication |
|
||
|---|---|---|
|
||
| CPU | 24 cores, AMD-V (`svm`), `kvm_amd` loaded | ample; nested-virt for KubeVirt (Phase 5) to be confirmed later |
|
||
| RAM | 31 GiB total, **~22 GiB available** | **binding constraint** — see sizing |
|
||
| Disk | 137 GiB free on `/` | tight; use thin qcow2, keep dev volumes small |
|
||
| libvirt | 12.0.0, functional (existing VMs, `default` NAT net, pools) | reuse or add a dedicated net |
|
||
| Present | ansible 2.20, virt-install, qemu-img, kubectl, helm, genisoimage | ok |
|
||
| **Missing (prereqs)** | **OpenTofu(+libvirt provider), age, sops, flux CLI**, cloud-localds | must install before execution |
|
||
|
||
## Toolchain — pinned dev container (decided)
|
||
The prereqs are **not all Python** (OpenTofu/age/sops/flux are Go binaries), so per the operator's
|
||
rule they are delivered as a **pinned dev container**, not host installs (and not a venv). See
|
||
[`devcontainer/`](devcontainer/): OpenTofu + `dmacvicar/libvirt` deps, age, sops, flux, ansible,
|
||
kubectl, helm, libvirt-client — baked into a Fedora-44 image (matches host libvirt 12). Run with the
|
||
host libvirt socket + repo mounted:
|
||
```
|
||
cd infra/devcontainer && ./dev.sh build && ./dev.sh
|
||
```
|
||
This isolates the toolchain, keeps the host clean, and gives every worker the same baseline.
|
||
|
||
## Resource plan (constrained by ~22 GiB RAM / 137 GiB disk)
|
||
Per node: **4 vCPU · 6 GiB RAM · 20 GiB OS disk + 40 GiB thin data disk** (Longhorn).
|
||
Totals: 12 vCPU · **18 GiB RAM** (~4 GiB host headroom) · ~180 GiB thin-allocated (sparse).
|
||
- ⚠️ **RAM is tight.** The *full* platform (Longhorn + Cilium + kube-prometheus-stack + Loki +
|
||
Keycloak/Postgres + cert-manager + …) across 18 GiB is a squeeze. In dev we **trim non-essential
|
||
components** (e.g. shorter Prometheus/Loki retention, single-replica where safe) or the operator
|
||
frees more RAM. This workstation cannot carry prod-scale — which is exactly why prod is 3 dedicated
|
||
mini-PCs.
|
||
- ⚠️ **Disk is tight** for media; keep Longhorn volumes minimal in dev — bulk media is external anyway
|
||
(OQ-2). Thin qcow2 means allocation ≫ real usage initially.
|
||
|
||
## Network plan
|
||
- A **dedicated libvirt NAT network** for the cluster (proposed `k3s-dev`, `10.73.0.0/24`) — isolates
|
||
it from the operator's other VMs, host-reachable. (Alternative: reuse `default` 192.168.122/24.)
|
||
- **Static IPs**: nodes `.11 / .12 / .13`; reserve `.240–.250` for **Cilium L2-announcement VIPs**
|
||
(ingress, per docs/09).
|
||
- Services are reachable **from the host** over the NAT subnet; split-horizon DNS in dev via the
|
||
network's dnsmasq or `/etc/hosts` (PiHole arrives in Phase 6). Public/edge port-forwarding is a prod
|
||
concern — N/A on the dev NAT.
|
||
|
||
## Work breakdown (task-board T-1.1 … T-1.5)
|
||
| Task | Deliverable | Location (proposed) |
|
||
|---|---|---|
|
||
| **T-1.1** OpenTofu libvirt module | 3 VMs + OS/data disks + NAT net + cloud-init seed | `infra/tofu/` |
|
||
| **T-1.2** cloud-init baseline | users, SSH key, sysctls + kernel modules for Cilium/Longhorn | `infra/cloud-init/` |
|
||
| **T-1.3** Ansible k3s + Cilium | HA k3s (3 servers, embedded etcd); Cilium (kube-proxy replacement, Gateway API); disable flannel/kube-proxy/Traefik/servicelb | `infra/ansible/` |
|
||
| **T-1.4** Flux + SOPS bootstrap | `flux bootstrap` → `clusters/dev`; generate age key, install as cluster Secret (private key **never** committed) | `bootstrap/` |
|
||
| **T-1.5** Destroy/rebuild proof | kill a node VM, `tofu apply` rebuild, re-join, no data loss; flip `CLUSTER_AVAILABLE=true`; first L2 conformance | `conformance/` (QA) |
|
||
|
||
Proposed Phase-1 impl tree: `infra/{tofu,ansible,cloud-init}/` and `bootstrap/{sops,flux}/`.
|
||
|
||
## Sequencing
|
||
**Start host Gitea** (`bootstrap/gitea/`) + push repo → `tofu apply` (VMs up) → cloud-init baseline →
|
||
Ansible (k3s + Cilium) → Flux + SOPS bootstrap (Flux pulls from the host Gitea) → verify reconcile →
|
||
destroy/rebuild proof → `CLUSTER_AVAILABLE=true` → conformance L2/L3 begins.
|
||
|
||
**Bootstrap source — RESOLVED:** a single **Gitea container on the host** (`bootstrap/gitea/`),
|
||
reachable from the cluster at `http://10.73.0.1:3000/…` (NAT gateway), public repo → Flux needs no
|
||
creds. Migrates to in-cluster Gitea in Phase 6. **Pins CONFIRMED:** k3s `v1.31.5+k3s1`, Cilium chart
|
||
`1.16.5`, single-node `k8sServiceHost` (dev-acceptable SPOF).
|
||
|
||
## Risks & mitigations
|
||
- **libvirt perms** — use `qemu:///system`; ensure the socket (virtqemud/libvirtd socket-activated) is
|
||
up and the user is in the `libvirt` group. `virsh` already works here, so this is largely satisfied.
|
||
- **k3s+Cilium install order** — install k3s with `--flannel-backend=none --disable-kube-proxy
|
||
--disable=traefik,servicelb`; the cluster is **NotReady until Cilium is up** (expected). Honor the
|
||
spec's Cilium-ownership contract (docs/02): Ansible installs, Flux HelmRelease adopts config, with
|
||
value-parity on `kubeProxyReplacement` / `k8sServiceHost/Port`.
|
||
- **Flux ↔ SOPS chicken-egg** — bootstrap Flux, create the age-key Secret out-of-band, then Flux
|
||
decrypts. Private age key handled outside git (`.gitignore` already guards `age.key`).
|
||
- **RAM ceiling** — trim the dev component set; monitor; don't co-schedule heavy workloads during
|
||
bring-up.
|
||
- **Nested virt (Phase 5, not now)** — verify `/sys/module/kvm_amd/parameters/nested=1` before
|
||
KubeVirt.
|
||
|
||
## Operator decisions / hand-offs — RESOLVED (2026-08-16)
|
||
1. **VM sizing:** ✅ **6 GiB / 4 vCPU × 3** (18 GiB; ~4 GiB host headroom). Full stack in later phases
|
||
will need trimming or more RAM.
|
||
2. **Toolchain:** ✅ **pinned dev container** (`infra/devcontainer/`) — prereqs baked in, not host-
|
||
installed.
|
||
3. **Network:** ✅ **dedicated NAT `k3s-dev` (10.73.0.0/24)**; nodes .11/.12/.13; VIPs .240–.250.
|
||
4. Cloudflare token / PIA creds remain **Phase 2/3** — not needed for Phase 1.
|
||
|
||
## Worker dispatch (on execution go)
|
||
- **Infra/Cluster Engineer** — T-1.1/1.2/1.3 (tofu + cloud-init + Ansible k3s/Cilium).
|
||
- **Security Engineer** — T-1.4 SOPS/age bootstrap.
|
||
- **GitOps Engineer** — T-1.4 Flux bootstrap → `clusters/dev`.
|
||
- **QA/Conformance** — T-1.5 destroy/rebuild test, flip `CLUSTER_AVAILABLE`, first L2 runs.
|