jakman85andClaude Opus 5
12c40d8c9c
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.
Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.
README.md
- Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
- Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
- Real conformance number (36/51, 70.6%) and how to regenerate it.
- Devcontainer workflow; note that the gate was red and why.
infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
the toolchain, off-repo state that must not be lost, and the fact that Flux's
git source is an app inside the cluster (`git push origin main` IS the deploy).
infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
rationale are still worth keeping; the libvirt execution steps are not.
infra/README.md — Proxmox, plus a map of the subtree and where the cluster
actually lives (it is not on this workstation).
infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
talks to an HTTP API. Everything runs in the container now. Documents the full
CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
and the read-only spec repo especially — without the latter, spec-immutability
silently skips instead of verifying).
infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
against the running nodes), pfSense NAT marked done, and a warning that
terraform.tfstate is absent on a fresh workstation — an apply with empty state
would try to recreate the three live nodes.
infra/tofu/variables.tf — section comment disagreed with its own defaults.
infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
domains that do not exist; behaviour unchanged, but it no longer lies.
apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
since been dropped. Now: live status, 40Gi, resolved open questions marked
resolved, and the two constraints the next person will otherwise rediscover —
KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
postBuild.substituteFrom (envsubst corrupts the cloud-init).
DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
the Flux source cutover, CI going live in-cluster, ten app onboardings and the
renderer bugs they exposed, the wave split (verified: DBs adopted, not
recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
VM, and kubectl SSO. Plus verified live state and the carried-forward open list.
Two corrections to the record while verifying:
- kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
[oidc:platform-admins]), so the rolling restart 3472f1d deferred has
happened.
- UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
and the CNP explicitly permits cluster -> :11443, so it is genuinely not
listening rather than a policy artifact.
Docs and comments only; no manifest semantics change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
84 lines
4.3 KiB
Markdown
84 lines
4.3 KiB
Markdown
# Dev toolbox (containerized baseline)
|
|
|
|
The toolchain for this repo is **not all Python** (OpenTofu, `age`, `sops`, `flux`, `chainsaw`,
|
|
`kyverno` are Go binaries; only Ansible and a couple of linters are Python). Per the operator's
|
|
rule it is therefore delivered as a **pinned dev container** — no host pollution, one reproducible
|
|
baseline, and the *same* binaries CI uses.
|
|
|
|
**This is the supported way to run anything in this repo.** Don't install the toolchain on the host.
|
|
|
|
## What's inside (pinned in `versions.env`)
|
|
|
|
| Group | Tools |
|
|
|---|---|
|
|
| Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
|
|
| GitOps / cluster | `flux`, `kubectl`, `helm` |
|
|
| Secrets | `sops`, `age` + `age-keygen` |
|
|
| **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
|
|
| Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |
|
|
|
|
The conformance-tool versions are kept in lockstep with `ci/versions.env` so
|
|
`make -C ci gate` in here is byte-identical to the CI runner (**R-TEST-02**). The image records
|
|
the resolved set at `/etc/dev-toolchain-versions`.
|
|
|
|
## Use
|
|
|
|
```bash
|
|
cd infra/devcontainer
|
|
./dev.sh build # build the image (needs network)
|
|
./dev.sh # build if needed + shell in /work (the repo)
|
|
./dev.sh run make -C ci gate # the merge gate, exactly as CI runs it
|
|
./dev.sh run sops -d path/to.sops.yaml
|
|
./dev.sh run tofu -chdir=/work/infra/tofu plan
|
|
```
|
|
`ENGINE=docker ./dev.sh` to use docker instead of rootless podman.
|
|
|
|
## What `dev.sh` mounts, and why each one matters
|
|
|
|
- `--network host` — Ansible SSHes to the nodes through the pfSense NAT (`192.168.2.102:225x`) and
|
|
`kubectl`/`flux`/`tofu` reach the kube-API (`192.168.2.50:6443`) and the Proxmox API
|
|
(`192.168.2.102:8006`) on the host's network.
|
|
- `-v $REPO:/work` (workdir) — the repo.
|
|
- `-v ~/.config/sops:/root/.config/sops` — **the age keyring, read-write**. This is deliberate: the
|
|
private key generated by `bootstrap/sops/age-keygen.sh` must land on the *host*, or it dies with
|
|
the `--rm` container and every SOPS-encrypted secret in the repo becomes undecryptable.
|
|
- `-v ~/.ssh:/root/.ssh:ro` — node SSH key (`home_network`).
|
|
- `-v ~/.config/home-k3s/ansible-vault-pass:...:ro` — the Ansible vault passphrase, so vaulted
|
|
`group_vars` decrypt unattended.
|
|
- `-v ../home-network-k3s:/home-network-k3s:ro` + `SPEC_REPO` — the **spec repo**, read-only, when
|
|
it exists beside this one. Without it `ci/spec-immutability.sh` (R-TEST-06) honestly *skips*
|
|
rather than passing — mount it, or that gate silently verifies nothing.
|
|
- `--security-opt label=disable` — SELinux hosts, so the bind mounts work without relabeling.
|
|
|
|
## tofu and ansible now run **inside** the container
|
|
|
|
This reverses the old Phase-1 limitation. When the dev cluster was on host libvirt, OpenTofu's
|
|
`dmacvicar/libvirt` provider could not do a read-write `qemu:///system` connect from inside a
|
|
container (polkit/PID-identity: `Cannot find start time for pid`), so tofu had to be extracted and
|
|
run on the host. The cluster moved to **Proxmox VE** and the provider is now `bpg/proxmox`, which
|
|
talks to the Proxmox **HTTP API** — nothing host-local is required. Ansible likewise only needs
|
|
SSH, which `--network host` provides.
|
|
|
|
So: run everything in here. The host needs only podman (or docker).
|
|
|
|
## kubectl inside the container — known gap
|
|
|
|
`dev.sh` sets `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`, a **libvirt-era path
|
|
that no longer exists**, and does not mount `~/.kube`. Override it explicitly:
|
|
|
|
```bash
|
|
DEV_KUBECONFIG=/work/path/to/break-glass.kubeconfig ./dev.sh run kubectl get nodes
|
|
```
|
|
|
|
Use a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login needs a browser,
|
|
which a container shell does not have. See [`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both
|
|
access paths. (Not yet fixed in `dev.sh`.)
|
|
|
|
## Notes
|
|
|
|
- Versions are **best-effort pins**; if a download 404s at build, bump the value in `versions.env`
|
|
and rebuild. Keep the conformance-tool block aligned with `ci/versions.env`.
|
|
- **Secrets never enter the image** — they arrive only as the bind mounts listed above.
|
|
- This container is the toolchain, not the cluster. The cluster runs on Proxmox VE; see
|
|
[`../PHASE1-STATUS.md`](../PHASE1-STATUS.md).
|