jakman85andClaude Opus 4.8
017ed0fb9c
Cluster is UP (3/3 Ready, Cilium OK); Phase 1 paused before Flux bootstrap (Option A: suspend platform/apps until Phase 2). Add infra/PHASE1-STATUS.md (what's done, state file locations, reconnect steps, remaining T-1.4/1.5, Phase-2 platform prereq), infra/cluster.sh (status/start/stop/kubeconfig), and document that tofu+ansible run on the host (libvirt provider limitation). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
52 lines
3.0 KiB
Markdown
52 lines
3.0 KiB
Markdown
# Dev toolbox (containerized baseline)
|
||
|
||
The Phase 1 prerequisites are **not all Python** (OpenTofu, `age`, `sops`, `flux` are Go binaries;
|
||
only Ansible is Python). Per the operator's rule, the toolchain is therefore a **pinned dev
|
||
container** baked with everything needed to drive the local KVM cluster — no host pollution, one
|
||
reproducible baseline.
|
||
|
||
## What's inside (pinned in `versions.env`)
|
||
OpenTofu, `age` + `age-keygen`, `sops`, `flux`, `kubectl`, `helm`, `ansible-core`, `libvirt-client`
|
||
(so the tofu `dmacvicar/libvirt` provider can talk to the host libvirt), `qemu-img`, cloud-init seed
|
||
tooling (`cloud-utils`/`genisoimage`), `openssh-clients`, and common utilities.
|
||
|
||
## Use
|
||
```bash
|
||
cd infra/devcontainer
|
||
./dev.sh build # build the image (needs network: base image + pinned binaries)
|
||
./dev.sh # build if needed + open a shell in /work (the repo)
|
||
./dev.sh run tofu -version
|
||
```
|
||
`ENGINE=docker ./dev.sh` to use docker instead of rootless podman.
|
||
|
||
## How it reaches the host KVM
|
||
`dev.sh` runs the container with:
|
||
- `--network host` — so Ansible can SSH to the cluster VMs on the `k3s-dev` NAT subnet
|
||
(10.73.0.0/24) and `kubectl`/`flux` can reach the API server.
|
||
- `-v /run/libvirt:/run/libvirt` + `LIBVIRT_DEFAULT_URI=qemu:///system` — so OpenTofu/virsh drive
|
||
the host libvirt daemon (the socket is world-writable; no group change needed).
|
||
- `--security-opt label=disable` — lets the container use the host libvirt socket without
|
||
relabeling it (SELinux hosts).
|
||
- `-v $REPO:/work` and `-v ~/.ssh:/root/.ssh:ro` — the repo and your SSH keys.
|
||
|
||
## Known limitation — run `tofu` and `ansible` on the HOST, not in the container
|
||
Discovered during Phase 1 bring-up (see `../PHASE1-STATUS.md`):
|
||
- **OpenTofu + `dmacvicar/libvirt`** fails *inside* the container — its read-write `qemu:///system`
|
||
connect hits a polkit/PID-identity path (`Cannot find start time for pid`) that a mounted socket
|
||
can't satisfy across the PID namespace (`--pid=host` didn't help). **Run the pinned tofu on the
|
||
host** (extract once: `podman run --rm -v ~/.local/bin:/out home-k3s-dev:local cp /usr/local/bin/tofu /out/`;
|
||
then `LIBVIRT_DEFAULT_URI=qemu:///system ~/.local/bin/tofu ...`).
|
||
- **Ansible** talks to the cluster VMs over SSH; simplest to run the pre-installed host `ansible-core`
|
||
with native networking to the `k3s-dev` NAT subnet.
|
||
- The container remains the baseline for **flux/sops/age/kubectl/helm** (used from inside with
|
||
`--network host` once the cluster exists).
|
||
|
||
## Notes
|
||
- Versions are **best-effort pins**; if a download 404s at build, bump the value in `versions.env`
|
||
(one line) to the current stable and rebuild. The image records the resolved set at
|
||
`/etc/dev-toolchain-versions`.
|
||
- Secrets never enter the image. The SOPS **age private key** is generated at bootstrap and kept
|
||
out of git (`.gitignore` guards `age.key`).
|
||
- This container is the toolchain, not the cluster. It provisions the 3 KVM nodes (T-1.1–T-1.4);
|
||
the cluster itself runs on libvirt on the host.
|