From 017ed0fb9c66432f7bdc7a543817f691bd8f4c68 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 16 Aug 2026 02:18:39 -0500 Subject: [PATCH] Phase 1 checkpoint: status/resume doc + cluster helper + toolchain note Cluster is UP (3/3 Ready, Cilium OK); Phase 1 paused before Flux bootstrap (Option A: suspend platform/apps until Phase 2). Add infra/PHASE1-STATUS.md (what's done, state file locations, reconnect steps, remaining T-1.4/1.5, Phase-2 platform prereq), infra/cluster.sh (status/start/stop/kubeconfig), and document that tofu+ansible run on the host (libvirt provider limitation). Co-Authored-By: Claude Opus 4.8 (1M context) --- infra/PHASE1-STATUS.md | 57 ++++++++++++++++++++++++++++++++++++ infra/cluster.sh | 19 ++++++++++++ infra/devcontainer/README.md | 12 ++++++++ 3 files changed, 88 insertions(+) create mode 100644 infra/PHASE1-STATUS.md create mode 100755 infra/cluster.sh diff --git a/infra/PHASE1-STATUS.md b/infra/PHASE1-STATUS.md new file mode 100644 index 0000000..3711140 --- /dev/null +++ b/infra/PHASE1-STATUS.md @@ -0,0 +1,57 @@ +# Phase 1 — Execution Status & Resume Guide + +**Checkpoint: 2026-08-16.** The dev/test cluster is **built and running**. Phase 1 is paused before +the Flux/SOPS bootstrap (a deliberate decision — see "Remaining"). This doc is how a future session +picks up. + +## DONE — a live 3-node HA k3s + Cilium cluster (T-1.1/1.2/1.3) +- `tofu apply` created the `k3s-dev` NAT network (10.73.0.0/24) + 3 VMs (`k3s-dev-1/2/3`, + 6 GiB/4 vCPU, 20 G OS + 40 G data, host-passthrough) — all **running**. +- cloud-init baselined the nodes; k3s HA (embedded etcd, 3 servers) + Cilium installed by Ansible. +- **Live health:** `kubectl get nodes` → **3/3 Ready**; `cilium status` → **OK**; CoreDNS up. + +### Fixes applied during execution (committed to main) +- `ansible.cfg`: `stdout_callback yaml` removed in ansible-core 2.20 → builtin default. +- `cilium-values`: `egressGateway` requires `bpf.masquerade: true` (+ `enableIPv4Masquerade`) or the + agent fatals — added. + +## Toolchain reality (important for resume) +The pinned **dev container** (`infra/devcontainer/`) holds flux/sops/age/kubectl/helm. BUT: +- **`tofu` runs on the HOST** — the `dmacvicar/libvirt` provider's read-write connect fails inside a + container (`Cannot find start time for pid`). The pinned binary was extracted to + **`~/.local/bin/tofu`** (OpenTofu 1.9.0). Run tofu from the host with `LIBVIRT_DEFAULT_URI=qemu:///system`. +- **`ansible` runs on the HOST** (pre-installed ansible-core 2.20; needs native network to the nodes). + +## State locations (persisted on disk; gitignored) +| What | Path | +|---|---| +| tofu state | `infra/tofu/terraform.tfstate` | +| tofu vars (ssh pubkey) | `infra/tofu/terraform.tfvars` | +| cluster kubeconfig | `infra/ansible/.cache/dev.kubeconfig` (API at 10.73.0.11:6443) | +| SSH key | `~/.ssh/home_network` (user `k3s`, NOPASSWD sudo) | +| VMs / network | libvirt `qemu:///system` (`virsh list`, net `k3s-dev`) | + +## Reconnect / manage +```bash +infra/cluster.sh status # VMs + kubectl get nodes +infra/cluster.sh start|stop # boot/graceful-shutdown the 3 VMs (state persists on disk) +eval "$(infra/cluster.sh kubeconfig)" # export KUBECONFIG +kubectl get nodes +ssh -i ~/.ssh/home_network k3s@10.73.0.11 +``` +The VMs are **running** at checkpoint (~18 GiB RAM). `infra/cluster.sh stop` frees RAM; k3s/etcd/ +Cilium recover on `start` (data on the disks). + +## REMAINING Phase 1 +- **T-1.4 Flux + SOPS bootstrap — Option A (decided):** start the host Gitea (`bootstrap/gitea/`), + push this repo, `flux install`, apply the `flux-system` GitRepository + age Secret — but **keep the + `platform` and `apps` Kustomizations SUSPENDED**. This proves Flux reconciles from the host Gitea + (Phase-1 exit) WITHOUT unleashing the incomplete Phase-0 platform stubs on the working cluster. +- **T-1.5** destroy/rebuild proof (`tofu destroy` a node → `apply` → re-join), then flip + `CLUSTER_AVAILABLE=true` so conformance L2/L3 runs. + +## Phase 2 prerequisite (before un-suspending `platform/`) +The Phase-0 `platform/` HelmReleases are stubs. **Do NOT let Flux reconcile them onto this cluster +until:** `platform/cilium` `k8sServiceHost` is set to **10.73.0.11** (parity with the Ansible install, +docs/02) and the REPLACE-ME values / secrets (Cloudflare token T-2.4p, etc.) are filled — otherwise +Flux would `helm upgrade` Cilium with a broken value and could break the running CNI. diff --git a/infra/cluster.sh b/infra/cluster.sh new file mode 100755 index 0000000..ca45394 --- /dev/null +++ b/infra/cluster.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +# infra/cluster.sh — manage the local dev k3s cluster (VMs + kubeconfig). Convenience for +# resuming Phase 1 development. Uses host virsh + kubectl (see infra/PHASE1-STATUS.md). +set -euo pipefail +export LIBVIRT_DEFAULT_URI=qemu:///system +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +KCFG="${HERE}/ansible/.cache/dev.kubeconfig" +NODES=(k3s-dev-1 k3s-dev-2 k3s-dev-3) + +case "${1:-status}" in + status) + virsh list --all | grep -E 'k3s-dev|Name|--' || true + if [ -f "$KCFG" ]; then KUBECONFIG="$KCFG" kubectl get nodes 2>/dev/null || echo "(cluster not reachable — VMs stopped? run: $0 start)"; fi + ;; + start) for n in "${NODES[@]}"; do virsh start "$n" 2>/dev/null || echo "$n already running"; done ;; + stop) for n in "${NODES[@]}"; do virsh shutdown "$n" 2>/dev/null || true; done; echo "graceful shutdown sent (state persists on disk; '$0 start' to resume)" ;; + kubeconfig) echo "export KUBECONFIG=${KCFG}" ;; + *) echo "usage: $0 {status|start|stop|kubeconfig}"; exit 2 ;; +esac diff --git a/infra/devcontainer/README.md b/infra/devcontainer/README.md index 5b4c027..77b5935 100644 --- a/infra/devcontainer/README.md +++ b/infra/devcontainer/README.md @@ -29,6 +29,18 @@ cd infra/devcontainer relabeling it (SELinux hosts). - `-v $REPO:/work` and `-v ~/.ssh:/root/.ssh:ro` — the repo and your SSH keys. +## Known limitation — run `tofu` and `ansible` on the HOST, not in the container +Discovered during Phase 1 bring-up (see `../PHASE1-STATUS.md`): +- **OpenTofu + `dmacvicar/libvirt`** fails *inside* the container — its read-write `qemu:///system` + connect hits a polkit/PID-identity path (`Cannot find start time for pid`) that a mounted socket + can't satisfy across the PID namespace (`--pid=host` didn't help). **Run the pinned tofu on the + host** (extract once: `podman run --rm -v ~/.local/bin:/out home-k3s-dev:local cp /usr/local/bin/tofu /out/`; + then `LIBVIRT_DEFAULT_URI=qemu:///system ~/.local/bin/tofu ...`). +- **Ansible** talks to the cluster VMs over SSH; simplest to run the pre-installed host `ansible-core` + with native networking to the `k3s-dev` NAT subnet. +- The container remains the baseline for **flux/sops/age/kubectl/helm** (used from inside with + `--network host` once the cluster exists). + ## Notes - Versions are **best-effort pins**; if a download 404s at build, bump the value in `versions.env` (one line) to the current stable and rebuild. The image records the resolved set at