0 Fork 0

Files

T

jakman85andClaude Opus 5 12c40d8c9c docs: reconcile the documentation with the deployed reality

The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.

Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.

README.md
  - Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
  - Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
  - Real conformance number (36/51, 70.6%) and how to regenerate it.
  - Devcontainer workflow; note that the gate was red and why.

infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
  cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
  paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
  the toolchain, off-repo state that must not be lost, and the fact that Flux's
  git source is an app inside the cluster (`git push origin main` IS the deploy).

infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
  rationale are still worth keeping; the libvirt execution steps are not.

infra/README.md — Proxmox, plus a map of the subtree and where the cluster
  actually lives (it is not on this workstation).

infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
  is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
  talks to an HTTP API. Everything runs in the container now. Documents the full
  CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
  and the read-only spec repo especially — without the latter, spec-immutability
  silently skips instead of verifying).

infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
  is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
  against the running nodes), pfSense NAT marked done, and a warning that
  terraform.tfstate is absent on a fresh workstation — an apply with empty state
  would try to recreate the three live nodes.

infra/tofu/variables.tf — section comment disagreed with its own defaults.

infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
  domains that do not exist; behaviour unchanged, but it no longer lies.

apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
  since been dropped. Now: live status, 40Gi, resolved open questions marked
  resolved, and the two constraints the next person will otherwise rediscover —
  KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
  postBuild.substituteFrom (envsubst corrupts the cloud-init).

DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
  migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
  the Flux source cutover, CI going live in-cluster, ten app onboardings and the
  renderer bugs they exposed, the wave split (verified: DBs adopted, not
  recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
  VM, and kubectl SSO. Plus verified live state and the carried-forward open list.

Two corrections to the record while verifying:
  - kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
    [oidc:platform-admins]), so the rolling restart 3472f1d deferred has
    happened.
  - UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
    24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
    and the CNP explicitly permits cluster -> :11443, so it is genuinely not
    listening rather than a policy artifact.

Docs and comments only; no manifest semantics change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

2026-08-27 00:00:05 -05:00

8.0 KiB

Raw Blame History

Phase 1 Scope — dev/test Cluster Bring-up (local KVM)

⚠️ HISTORICAL — DO NOT EXECUTE

Phase 1 is complete. This document is the original scoping for the first dev cluster, which ran on local libvirt/KVM VMs on the workstation (10.73.0.0/24). That cluster was retired: it was constrained by the workstation's ~22 GiB RAM and 137 GiB disk, and the dev cluster was migrated to a Proxmox VE host (nodes 192.168.1.50/.51/.52, 6 vCPU / 16 GiB each).

Kept for the record — the exit criteria, sizing rationale, and risk analysis below are still the reasoning behind the current design. Everything about libvirt, virsh, the dmacvicar/libvirt provider, 10.73.0.x, and the "missing prereqs" table is superseded.

For the cluster as it exists today, read PHASE1-STATUS.md.

Owner: Program Manager. Spec refs (read-only): ../home-network-k3s/roadmap/phases.md (Phase 1), docs/02-environments-and-provisioning.md, ADRs 0001 (k3s), 0002 (Cilium), 0004 (Flux), 0007 (SOPS). Status (as written, 2026-08-16): scoped, awaiting operator go / prerequisite installs. Status now: delivered, then superseded by the Proxmox migration.

Objective & exit criteria (roadmap Phase 1)

A 3-node k3s HA cluster on local libvirt/KVM VMs, provisioned entirely by IaC (OpenTofu + cloud-init + Ansible), with Cilium CNI, Flux reconciling from clusters/dev, and the SOPS age key bootstrapped. Exit gates:

3-node k3s HA (embedded etcd) Ready; Cilium healthy (Hubble up); no flannel/kube-proxy/Traefik.
Flux reconciling clusters/dev from git; drift corrected.
A node VM destroyed and rebuilt from tofu apply re-joins; cluster reconverges, no data loss.
Flip CLUSTER_AVAILABLE=true → the conformance L2/L3 behavior tests begin running (first real default-deny / data-survival green — the gate stops skipping).

Environment (verified 2026-08-16)

Resource	Finding	Implication
CPU	24 cores, AMD-V (`svm`), `kvm_amd` loaded	ample; nested-virt for KubeVirt (Phase 5) to be confirmed later
RAM	31 GiB total, ~22 GiB available	binding constraint — see sizing
Disk	137 GiB free on `/`	tight; use thin qcow2, keep dev volumes small
libvirt	12.0.0, functional (existing VMs, `default` NAT net, pools)	reuse or add a dedicated net
Present	ansible 2.20, virt-install, qemu-img, kubectl, helm, genisoimage	ok
Missing (prereqs)	OpenTofu(+libvirt provider), age, sops, flux CLI, cloud-localds	must install before execution

Toolchain — pinned dev container (decided)

The prereqs are not all Python (OpenTofu/age/sops/flux are Go binaries), so per the operator's rule they are delivered as a pinned dev container, not host installs (and not a venv). See devcontainer/: OpenTofu + dmacvicar/libvirt deps, age, sops, flux, ansible, kubectl, helm, libvirt-client — baked into a Fedora-44 image (matches host libvirt 12). Run with the host libvirt socket + repo mounted:

cd infra/devcontainer && ./dev.sh build && ./dev.sh

This isolates the toolchain, keeps the host clean, and gives every worker the same baseline.

Resource plan (constrained by ~22 GiB RAM / 137 GiB disk)

Per node: 4 vCPU · 6 GiB RAM · 20 GiB OS disk + 40 GiB thin data disk (Longhorn). Totals: 12 vCPU · 18 GiB RAM (~4 GiB host headroom) · ~180 GiB thin-allocated (sparse).

⚠️ RAM is tight. The full platform (Longhorn + Cilium + kube-prometheus-stack + Loki + Keycloak/Postgres + cert-manager + …) across 18 GiB is a squeeze. In dev we trim non-essential components (e.g. shorter Prometheus/Loki retention, single-replica where safe) or the operator frees more RAM. This workstation cannot carry prod-scale — which is exactly why prod is 3 dedicated mini-PCs.
⚠️ Disk is tight for media; keep Longhorn volumes minimal in dev — bulk media is external anyway (OQ-2). Thin qcow2 means allocation ≫ real usage initially.

Network plan

A dedicated libvirt NAT network for the cluster (proposed k3s-dev, 10.73.0.0/24) — isolates it from the operator's other VMs, host-reachable. (Alternative: reuse default 192.168.122/24.)
Static IPs: nodes .11 / .12 / .13; reserve .240–.250 for Cilium L2-announcement VIPs (ingress, per docs/09).
Services are reachable from the host over the NAT subnet; split-horizon DNS in dev via the network's dnsmasq or /etc/hosts (PiHole arrives in Phase 6). Public/edge port-forwarding is a prod concern — N/A on the dev NAT.

Work breakdown (task-board T-1.1 … T-1.5)

Task	Deliverable	Location (proposed)
T-1.1 OpenTofu libvirt module	3 VMs + OS/data disks + NAT net + cloud-init seed	`infra/tofu/`
T-1.2 cloud-init baseline	users, SSH key, sysctls + kernel modules for Cilium/Longhorn	`infra/cloud-init/`
T-1.3 Ansible k3s + Cilium	HA k3s (3 servers, embedded etcd); Cilium (kube-proxy replacement, Gateway API); disable flannel/kube-proxy/Traefik/servicelb	`infra/ansible/`
T-1.4 Flux + SOPS bootstrap	`flux bootstrap` → `clusters/dev`; generate age key, install as cluster Secret (private key never committed)	`bootstrap/`
T-1.5 Destroy/rebuild proof	kill a node VM, `tofu apply` rebuild, re-join, no data loss; flip `CLUSTER_AVAILABLE=true`; first L2 conformance	`conformance/` (QA)

Proposed Phase-1 impl tree: infra/{tofu,ansible,cloud-init}/ and bootstrap/{sops,flux}/.

Sequencing

Start host Gitea (bootstrap/gitea/) + push repo → tofu apply (VMs up) → cloud-init baseline → Ansible (k3s + Cilium) → Flux + SOPS bootstrap (Flux pulls from the host Gitea) → verify reconcile → destroy/rebuild proof → CLUSTER_AVAILABLE=true → conformance L2/L3 begins.

Bootstrap source — RESOLVED: a single Gitea container on the host (bootstrap/gitea/), reachable from the cluster at http://10.73.0.1:3000/… (NAT gateway), public repo → Flux needs no creds. Migrates to in-cluster Gitea in Phase 6. Pins CONFIRMED: k3s v1.31.5+k3s1, Cilium chart 1.16.5, single-node k8sServiceHost (dev-acceptable SPOF).

Risks & mitigations

libvirt perms — use qemu:///system; ensure the socket (virtqemud/libvirtd socket-activated) is up and the user is in the libvirt group. virsh already works here, so this is largely satisfied.
k3s+Cilium install order — install k3s with --flannel-backend=none --disable-kube-proxy --disable=traefik,servicelb; the cluster is NotReady until Cilium is up (expected). Honor the spec's Cilium-ownership contract (docs/02): Ansible installs, Flux HelmRelease adopts config, with value-parity on kubeProxyReplacement / k8sServiceHost/Port.
Flux ↔ SOPS chicken-egg — bootstrap Flux, create the age-key Secret out-of-band, then Flux decrypts. Private age key handled outside git (.gitignore already guards age.key).
RAM ceiling — trim the dev component set; monitor; don't co-schedule heavy workloads during bring-up.
Nested virt (Phase 5, not now) — verify /sys/module/kvm_amd/parameters/nested=1 before KubeVirt.

Operator decisions / hand-offs — RESOLVED (2026-08-16)

VM sizing: ✅ 6 GiB / 4 vCPU × 3 (18 GiB; ~4 GiB host headroom). Full stack in later phases will need trimming or more RAM.
Toolchain: ✅ pinned dev container (infra/devcontainer/) — prereqs baked in, not host- installed.
Network: ✅ dedicated NAT k3s-dev (10.73.0.0/24); nodes .11/.12/.13; VIPs .240–.250.
Cloudflare token / PIA creds remain Phase 2/3 — not needed for Phase 1.

Worker dispatch (on execution go)

Infra/Cluster Engineer — T-1.1/1.2/1.3 (tofu + cloud-init + Ansible k3s/Cilium).
Security Engineer — T-1.4 SOPS/age bootstrap.
GitOps Engineer — T-1.4 Flux bootstrap → clusters/dev.
QA/Conformance — T-1.5 destroy/rebuild test, flip CLUSTER_AVAILABLE, first L2 runs.

8.0 KiB Raw Blame History Unescape Escape