Phase 1 checkpoint: status/resume doc + cluster helper + toolchain note
Cluster is UP (3/3 Ready, Cilium OK); Phase 1 paused before Flux bootstrap (Option A: suspend platform/apps until Phase 2). Add infra/PHASE1-STATUS.md (what's done, state file locations, reconnect steps, remaining T-1.4/1.5, Phase-2 platform prereq), infra/cluster.sh (status/start/stop/kubeconfig), and document that tofu+ansible run on the host (libvirt provider limitation). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed
1 parent
26e4b019fa
commit
017ed0fb9c
3 files changed
+88
No files matched your search
@@ -0,0 +1,57 @@
|
||||
# Phase 1 — Execution Status & Resume Guide
|
||||
|
||||
**Checkpoint: 2026-08-16.** The dev/test cluster is **built and running**. Phase 1 is paused before
|
||||
the Flux/SOPS bootstrap (a deliberate decision — see "Remaining"). This doc is how a future session
|
||||
picks up.
|
||||
|
||||
## DONE — a live 3-node HA k3s + Cilium cluster (T-1.1/1.2/1.3)
|
||||
- `tofu apply` created the `k3s-dev` NAT network (10.73.0.0/24) + 3 VMs (`k3s-dev-1/2/3`,
|
||||
6 GiB/4 vCPU, 20 G OS + 40 G data, host-passthrough) — all **running**.
|
||||
- cloud-init baselined the nodes; k3s HA (embedded etcd, 3 servers) + Cilium installed by Ansible.
|
||||
- **Live health:** `kubectl get nodes` → **3/3 Ready**; `cilium status` → **OK**; CoreDNS up.
|
||||
|
||||
### Fixes applied during execution (committed to main)
|
||||
- `ansible.cfg`: `stdout_callback yaml` removed in ansible-core 2.20 → builtin default.
|
||||
- `cilium-values`: `egressGateway` requires `bpf.masquerade: true` (+ `enableIPv4Masquerade`) or the
|
||||
agent fatals — added.
|
||||
|
||||
## Toolchain reality (important for resume)
|
||||
The pinned **dev container** (`infra/devcontainer/`) holds flux/sops/age/kubectl/helm. BUT:
|
||||
- **`tofu` runs on the HOST** — the `dmacvicar/libvirt` provider's read-write connect fails inside a
|
||||
container (`Cannot find start time for pid`). The pinned binary was extracted to
|
||||
**`~/.local/bin/tofu`** (OpenTofu 1.9.0). Run tofu from the host with `LIBVIRT_DEFAULT_URI=qemu:///system`.
|
||||
- **`ansible` runs on the HOST** (pre-installed ansible-core 2.20; needs native network to the nodes).
|
||||
|
||||
## State locations (persisted on disk; gitignored)
|
||||
| What | Path |
|
||||
|---|---|
|
||||
| tofu state | `infra/tofu/terraform.tfstate` |
|
||||
| tofu vars (ssh pubkey) | `infra/tofu/terraform.tfvars` |
|
||||
| cluster kubeconfig | `infra/ansible/.cache/dev.kubeconfig` (API at 10.73.0.11:6443) |
|
||||
| SSH key | `~/.ssh/home_network` (user `k3s`, NOPASSWD sudo) |
|
||||
| VMs / network | libvirt `qemu:///system` (`virsh list`, net `k3s-dev`) |
|
||||
|
||||
## Reconnect / manage
|
||||
```bash
|
||||
infra/cluster.sh status # VMs + kubectl get nodes
|
||||
infra/cluster.sh start|stop # boot/graceful-shutdown the 3 VMs (state persists on disk)
|
||||
eval "$(infra/cluster.sh kubeconfig)" # export KUBECONFIG
|
||||
kubectl get nodes
|
||||
ssh -i ~/.ssh/home_network [email protected]
|
||||
```
|
||||
The VMs are **running** at checkpoint (~18 GiB RAM). `infra/cluster.sh stop` frees RAM; k3s/etcd/
|
||||
Cilium recover on `start` (data on the disks).
|
||||
|
||||
## REMAINING Phase 1
|
||||
- **T-1.4 Flux + SOPS bootstrap — Option A (decided):** start the host Gitea (`bootstrap/gitea/`),
|
||||
push this repo, `flux install`, apply the `flux-system` GitRepository + age Secret — but **keep the
|
||||
`platform` and `apps` Kustomizations SUSPENDED**. This proves Flux reconciles from the host Gitea
|
||||
(Phase-1 exit) WITHOUT unleashing the incomplete Phase-0 platform stubs on the working cluster.
|
||||
- **T-1.5** destroy/rebuild proof (`tofu destroy` a node → `apply` → re-join), then flip
|
||||
`CLUSTER_AVAILABLE=true` so conformance L2/L3 runs.
|
||||
|
||||
## Phase 2 prerequisite (before un-suspending `platform/`)
|
||||
The Phase-0 `platform/` HelmReleases are stubs. **Do NOT let Flux reconcile them onto this cluster
|
||||
until:** `platform/cilium` `k8sServiceHost` is set to **10.73.0.11** (parity with the Ansible install,
|
||||
docs/02) and the REPLACE-ME values / secrets (Cloudflare token T-2.4p, etc.) are filled — otherwise
|
||||
Flux would `helm upgrade` Cilium with a broken value and could break the running CNI.
|
||||
Executable
+19
@@ -0,0 +1,19 @@
|
||||
#!/usr/bin/env bash
|
||||
# infra/cluster.sh — manage the local dev k3s cluster (VMs + kubeconfig). Convenience for
|
||||
# resuming Phase 1 development. Uses host virsh + kubectl (see infra/PHASE1-STATUS.md).
|
||||
set -euo pipefail
|
||||
export LIBVIRT_DEFAULT_URI=qemu:///system
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
KCFG="${HERE}/ansible/.cache/dev.kubeconfig"
|
||||
NODES=(k3s-dev-1 k3s-dev-2 k3s-dev-3)
|
||||
|
||||
case "${1:-status}" in
|
||||
status)
|
||||
virsh list --all | grep -E 'k3s-dev|Name|--' || true
|
||||
if [ -f "$KCFG" ]; then KUBECONFIG="$KCFG" kubectl get nodes 2>/dev/null || echo "(cluster not reachable — VMs stopped? run: $0 start)"; fi
|
||||
;;
|
||||
start) for n in "${NODES[@]}"; do virsh start "$n" 2>/dev/null || echo "$n already running"; done ;;
|
||||
stop) for n in "${NODES[@]}"; do virsh shutdown "$n" 2>/dev/null || true; done; echo "graceful shutdown sent (state persists on disk; '$0 start' to resume)" ;;
|
||||
kubeconfig) echo "export KUBECONFIG=${KCFG}" ;;
|
||||
*) echo "usage: $0 {status|start|stop|kubeconfig}"; exit 2 ;;
|
||||
esac
|
||||
@@ -29,6 +29,18 @@ cd infra/devcontainer
|
||||
relabeling it (SELinux hosts).
|
||||
- `-v $REPO:/work` and `-v ~/.ssh:/root/.ssh:ro` — the repo and your SSH keys.
|
||||
|
||||
## Known limitation — run `tofu` and `ansible` on the HOST, not in the container
|
||||
Discovered during Phase 1 bring-up (see `../PHASE1-STATUS.md`):
|
||||
- **OpenTofu + `dmacvicar/libvirt`** fails *inside* the container — its read-write `qemu:///system`
|
||||
connect hits a polkit/PID-identity path (`Cannot find start time for pid`) that a mounted socket
|
||||
can't satisfy across the PID namespace (`--pid=host` didn't help). **Run the pinned tofu on the
|
||||
host** (extract once: `podman run --rm -v ~/.local/bin:/out home-k3s-dev:local cp /usr/local/bin/tofu /out/`;
|
||||
then `LIBVIRT_DEFAULT_URI=qemu:///system ~/.local/bin/tofu ...`).
|
||||
- **Ansible** talks to the cluster VMs over SSH; simplest to run the pre-installed host `ansible-core`
|
||||
with native networking to the `k3s-dev` NAT subnet.
|
||||
- The container remains the baseline for **flux/sops/age/kubectl/helm** (used from inside with
|
||||
`--network host` once the cluster exists).
|
||||
|
||||
## Notes
|
||||
- Versions are **best-effort pins**; if a download 404s at build, bump the value in `versions.env`
|
||||
(one line) to the current stable and rebuild. The image records the resolved set at
|
||||
|
||||
Reference in new issue
Block a user