jakman85andClaude Opus 4.8
017ed0fb9c
Cluster is UP (3/3 Ready, Cilium OK); Phase 1 paused before Flux bootstrap (Option A: suspend platform/apps until Phase 2). Add infra/PHASE1-STATUS.md (what's done, state file locations, reconnect steps, remaining T-1.4/1.5, Phase-2 platform prereq), infra/cluster.sh (status/start/stop/kubeconfig), and document that tofu+ansible run on the host (libvirt provider limitation). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
3.0 KiB
3.0 KiB
Dev toolbox (containerized baseline)
The Phase 1 prerequisites are not all Python (OpenTofu, age, sops, flux are Go binaries;
only Ansible is Python). Per the operator's rule, the toolchain is therefore a pinned dev
container baked with everything needed to drive the local KVM cluster — no host pollution, one
reproducible baseline.
What's inside (pinned in versions.env)
OpenTofu, age + age-keygen, sops, flux, kubectl, helm, ansible-core, libvirt-client
(so the tofu dmacvicar/libvirt provider can talk to the host libvirt), qemu-img, cloud-init seed
tooling (cloud-utils/genisoimage), openssh-clients, and common utilities.
Use
cd infra/devcontainer
./dev.sh build # build the image (needs network: base image + pinned binaries)
./dev.sh # build if needed + open a shell in /work (the repo)
./dev.sh run tofu -version
ENGINE=docker ./dev.sh to use docker instead of rootless podman.
How it reaches the host KVM
dev.sh runs the container with:
--network host— so Ansible can SSH to the cluster VMs on thek3s-devNAT subnet (10.73.0.0/24) andkubectl/fluxcan reach the API server.-v /run/libvirt:/run/libvirt+LIBVIRT_DEFAULT_URI=qemu:///system— so OpenTofu/virsh drive the host libvirt daemon (the socket is world-writable; no group change needed).--security-opt label=disable— lets the container use the host libvirt socket without relabeling it (SELinux hosts).-v $REPO:/workand-v ~/.ssh:/root/.ssh:ro— the repo and your SSH keys.
Known limitation — run tofu and ansible on the HOST, not in the container
Discovered during Phase 1 bring-up (see ../PHASE1-STATUS.md):
- OpenTofu +
dmacvicar/libvirtfails inside the container — its read-writeqemu:///systemconnect hits a polkit/PID-identity path (Cannot find start time for pid) that a mounted socket can't satisfy across the PID namespace (--pid=hostdidn't help). Run the pinned tofu on the host (extract once:podman run --rm -v ~/.local/bin:/out home-k3s-dev:local cp /usr/local/bin/tofu /out/; thenLIBVIRT_DEFAULT_URI=qemu:///system ~/.local/bin/tofu ...). - Ansible talks to the cluster VMs over SSH; simplest to run the pre-installed host
ansible-corewith native networking to thek3s-devNAT subnet. - The container remains the baseline for flux/sops/age/kubectl/helm (used from inside with
--network hostonce the cluster exists).
Notes
- Versions are best-effort pins; if a download 404s at build, bump the value in
versions.env(one line) to the current stable and rebuild. The image records the resolved set at/etc/dev-toolchain-versions. - Secrets never enter the image. The SOPS age private key is generated at bootstrap and kept
out of git (
.gitignoreguardsage.key). - This container is the toolchain, not the cluster. It provisions the 3 KVM nodes (T-1.1–T-1.4); the cluster itself runs on libvirt on the host.