jakman85andClaude Opus 5
12c40d8c9c
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.
Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.
README.md
- Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
- Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
- Real conformance number (36/51, 70.6%) and how to regenerate it.
- Devcontainer workflow; note that the gate was red and why.
infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
the toolchain, off-repo state that must not be lost, and the fact that Flux's
git source is an app inside the cluster (`git push origin main` IS the deploy).
infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
rationale are still worth keeping; the libvirt execution steps are not.
infra/README.md — Proxmox, plus a map of the subtree and where the cluster
actually lives (it is not on this workstation).
infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
talks to an HTTP API. Everything runs in the container now. Documents the full
CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
and the read-only spec repo especially — without the latter, spec-immutability
silently skips instead of verifying).
infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
against the running nodes), pfSense NAT marked done, and a warning that
terraform.tfstate is absent on a fresh workstation — an apply with empty state
would try to recreate the three live nodes.
infra/tofu/variables.tf — section comment disagreed with its own defaults.
infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
domains that do not exist; behaviour unchanged, but it no longer lies.
apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
since been dropped. Now: live status, 40Gi, resolved open questions marked
resolved, and the two constraints the next person will otherwise rediscover —
KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
postBuild.substituteFrom (envsubst corrupts the cloud-init).
DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
the Flux source cutover, CI going live in-cluster, ten app onboardings and the
renderer bugs they exposed, the wave split (verified: DBs adopted, not
recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
VM, and kubectl SSO. Plus verified live state and the carried-forward open list.
Two corrections to the record while verifying:
- kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
[oidc:platform-admins]), so the rolling restart 3472f1d deferred has
happened.
- UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
and the CNP explicitly permits cluster -> :11443, so it is genuinely not
listening rather than a policy artifact.
Docs and comments only; no manifest semantics change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Dev toolbox (containerized baseline)
The toolchain for this repo is not all Python (OpenTofu, age, sops, flux, chainsaw,
kyverno are Go binaries; only Ansible and a couple of linters are Python). Per the operator's
rule it is therefore delivered as a pinned dev container — no host pollution, one reproducible
baseline, and the same binaries CI uses.
This is the supported way to run anything in this repo. Don't install the toolchain on the host.
What's inside (pinned in versions.env)
| Group | Tools |
|---|---|
| Provisioning | OpenTofu (bpg/proxmox provider), ansible-core, openssh-clients |
| GitOps / cluster | flux, kubectl, helm |
| Secrets | sops, age + age-keygen |
| CI gate | yamllint, check-jsonschema, kyverno CLI, kyverno-json, chainsaw, yq, diffutils, make |
| Misc | qemu-img, cloud-utils/genisoimage, libvirt-client (disk-image utilities only — leftovers from the libvirt era, harmless) |
The conformance-tool versions are kept in lockstep with ci/versions.env so
make -C ci gate in here is byte-identical to the CI runner (R-TEST-02). The image records
the resolved set at /etc/dev-toolchain-versions.
Use
cd infra/devcontainer
./dev.sh build # build the image (needs network)
./dev.sh # build if needed + shell in /work (the repo)
./dev.sh run make -C ci gate # the merge gate, exactly as CI runs it
./dev.sh run sops -d path/to.sops.yaml
./dev.sh run tofu -chdir=/work/infra/tofu plan
ENGINE=docker ./dev.sh to use docker instead of rootless podman.
What dev.sh mounts, and why each one matters
--network host— Ansible SSHes to the nodes through the pfSense NAT (192.168.2.102:225x) andkubectl/flux/tofureach the kube-API (192.168.2.50:6443) and the Proxmox API (192.168.2.102:8006) on the host's network.-v $REPO:/work(workdir) — the repo.-v ~/.config/sops:/root/.config/sops— the age keyring, read-write. This is deliberate: the private key generated bybootstrap/sops/age-keygen.shmust land on the host, or it dies with the--rmcontainer and every SOPS-encrypted secret in the repo becomes undecryptable.-v ~/.ssh:/root/.ssh:ro— node SSH key (home_network).-v ~/.config/home-k3s/ansible-vault-pass:...:ro— the Ansible vault passphrase, so vaultedgroup_varsdecrypt unattended.-v ../home-network-k3s:/home-network-k3s:ro+SPEC_REPO— the spec repo, read-only, when it exists beside this one. Without itci/spec-immutability.sh(R-TEST-06) honestly skips rather than passing — mount it, or that gate silently verifies nothing.--security-opt label=disable— SELinux hosts, so the bind mounts work without relabeling.
tofu and ansible now run inside the container
This reverses the old Phase-1 limitation. When the dev cluster was on host libvirt, OpenTofu's
dmacvicar/libvirt provider could not do a read-write qemu:///system connect from inside a
container (polkit/PID-identity: Cannot find start time for pid), so tofu had to be extracted and
run on the host. The cluster moved to Proxmox VE and the provider is now bpg/proxmox, which
talks to the Proxmox HTTP API — nothing host-local is required. Ansible likewise only needs
SSH, which --network host provides.
So: run everything in here. The host needs only podman (or docker).
kubectl inside the container — known gap
dev.sh sets KUBECONFIG to /work/infra/ansible/.cache/dev.kubeconfig, a libvirt-era path
that no longer exists, and does not mount ~/.kube. Override it explicitly:
DEV_KUBECONFIG=/work/path/to/break-glass.kubeconfig ./dev.sh run kubectl get nodes
Use a break-glass cert kubeconfig, not the k3s-oidc context — OIDC login needs a browser,
which a container shell does not have. See ../PHASE1-STATUS.md for both
access paths. (Not yet fixed in dev.sh.)
Notes
- Versions are best-effort pins; if a download 404s at build, bump the value in
versions.envand rebuild. Keep the conformance-tool block aligned withci/versions.env. - Secrets never enter the image — they arrive only as the bind mounts listed above.
- This container is the toolchain, not the cluster. The cluster runs on Proxmox VE; see
../PHASE1-STATUS.md.