jakman85andClaude Opus 5
8059297036
Operator correction: "sop, wg, and any other tool requirement is supposed to be
baked into the devcontainer."
My previous commit discovered that `wg` existed only on the host, `sops` only in
the container, and `qrencode` on neither -- and fixed it the wrong way, by
splitting every step of the key procedure across the boundary and telling the
operator to `dnf install qrencode` on their workstation. That breaks CLAUDE.md §5
and the devcontainer README's "Don't install the toolchain on the host", produces
a procedure that cannot be reproduced in CI, and is miserable to follow.
wireguard-tools + qrencode are now distro packages in the Containerfile. Both are
in Fedora 44's repos (wireguard-tools 1.0.20260223 -- the same upstream version as
the image this component runs -- and qrencode 4.1.1), unpinned like every other
dnf package there: the pinned thing is the Fedora base, and neither is a gate tool
whose version changes what green means.
Three things got SIMPLER rather than merely relocated, which is the tell that the
rule was right:
* Step 2 is one pipeline in one shell -- `sops set ... "$(wg genkey)"` writes the
key straight into the already-encrypted document. No staged copy in .cache, no
`*.key`-suffix warning, no sha256 comparison to prove the right bytes landed.
* new-peer.sh derives the server public key from wg-server-key.sops.yaml itself
(`sops -d --extract | wg pubkey`), so there is no local key copy to keep in
step with the encrypted one, and no magic host path it must live at.
* `.cache/` is gitignored at the repo root (mirroring infra/ansible/.gitignore),
giving the container a persistent non-committable scratch dir. new-peer.sh
writes profiles to .cache/wg-peers/ -- necessary because the repo bind-mount is
the container's only persistent read-write mount, so a profile in /tmp there is
gone before it can reach a desktop. Its blanket "refuse to write inside the
checkout" guard is now a `git check-ignore` test of the real property.
Verified by execution, not reasoning: the image rebuilds; wg/qrencode/sops all
resolve in it; `sops -d | wg pubkey` runs in one shell; new-peer.sh refuses the
REPLACE-ME placeholder with an actionable message; and a full run with a throwaway
server key renders a correct profile at 0600 under a 0700 gitignored dir, prints
the QR, and refuses a non-ignored destination inside the repo.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
161 lines
9.3 KiB
Markdown
161 lines
9.3 KiB
Markdown
# Dev toolbox (containerized baseline)
|
|
|
|
The toolchain for this repo is **not all Python** (OpenTofu, `age`, `sops`, `flux`, `chainsaw`,
|
|
`kyverno` are Go binaries; only Ansible and a couple of linters are Python). Per the operator's
|
|
rule it is therefore delivered as a **pinned dev container** — no host pollution, one reproducible
|
|
baseline, and the *same* binaries CI uses.
|
|
|
|
**This is the supported way to run anything in this repo.** Don't install the toolchain on the host.
|
|
|
|
## What's inside (pinned in `versions.env`)
|
|
|
|
| Group | Tools |
|
|
|---|---|
|
|
| Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
|
|
| GitOps / cluster | `flux`, `kubectl`, `helm` |
|
|
| Secrets | `sops`, `age` + `age-keygen` |
|
|
| VPN (`platform/vpn-ingress`) | `wg` (wireguard-tools) — mint the server key, derive client public keys; `qrencode` — render a client profile as a phone-scannable QR |
|
|
| **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
|
|
| Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |
|
|
|
|
The conformance-tool versions are kept in lockstep with `ci/versions.env` so
|
|
`make -C ci gate` in here is byte-identical to the CI runner (**R-TEST-02**). The image records
|
|
the resolved set at `/etc/dev-toolchain-versions`.
|
|
|
|
## Use
|
|
|
|
```bash
|
|
cd infra/devcontainer
|
|
./dev.sh build # build the image (needs network)
|
|
./dev.sh # build if needed + shell in /work (the repo)
|
|
./dev.sh run make -C ci gate # the merge gate, exactly as CI runs it
|
|
./dev.sh run sops -d path/to.sops.yaml
|
|
./dev.sh run tofu -chdir=/work/infra/tofu plan
|
|
```
|
|
`ENGINE=docker ./dev.sh` to use docker instead of rootless podman.
|
|
|
|
## Choosing a cluster — `CLUSTER=dev|prod`
|
|
|
|
```bash
|
|
./dev.sh run kubectl get nodes # DEV — the default, unchanged
|
|
CLUSTER=prod ./dev.sh run kubectl get nodes # PROD
|
|
CLUSTER=prod ./dev.sh # interactive shell pointed at prod
|
|
```
|
|
|
|
An env-var **prefix**, so the target sits on the same line as the command. Every invocation prints
|
|
a banner naming the cluster before the container starts, because `CLUSTER` can also be inherited
|
|
from an exported shell variable.
|
|
|
|
Inside the container `KUBECONFIG` is a two-file **chain**,
|
|
`<per-invocation selector>:<shared merged config>`:
|
|
|
|
| file | shared? | holds |
|
|
|---|---|---|
|
|
| selector, in a private `mktemp` dir | no — one per `dev.sh` invocation | only `current-context: k3s-$CLUSTER`; no credentials |
|
|
| `infra/ansible/.cache/clusters.kubeconfig` | yes — bind-mounted, read by every container | both clusters under unambiguous names; **no `current-context`** |
|
|
|
|
kubectl merges the chain first-wins, so the selector supplies this invocation's default:
|
|
|
|
```
|
|
CURRENT NAME CLUSTER AUTHINFO
|
|
* k3s-dev k3s-dev k3s-dev-admin
|
|
k3s-prod k3s-prod k3s-prod-admin
|
|
```
|
|
|
|
So `kubectl config current-context` answers "which cluster am I on?" honestly, and
|
|
`kubectl --context k3s-prod …` works from a dev-defaulted shell.
|
|
|
|
**Why the default context is not stored in the merged file.** It used to be, and that was a
|
|
cross-cluster data-corruption bug. The merged file lives in the repo's `.cache`, which every
|
|
container bind-mounts, so `current-context` was **shared mutable state**: two concurrent `dev.sh`
|
|
runs with different `CLUSTER` values overwrote each other's default *mid-command*. A
|
|
`CLUSTER=prod` script read prod for its first half and **dev** for its second; it was caught only
|
|
because a dev-only PVC name showed up in a prod query. Moving selection into a per-invocation file
|
|
removes the shared field rather than locking it — there is no window left to lose. Read on its own
|
|
the merged file now resolves to nothing (`error: current-context is not set`), which is loud and
|
|
safe; `--context k3s-dev` / `--context k3s-prod` against it are unaffected and remain the pattern
|
|
to prefer in scripts. Concurrent `CLUSTER=dev` and `CLUSTER=prod` invocations are now safe.
|
|
|
|
Consequences worth knowing:
|
|
|
|
- Export what `kubeconfig.sh` prints **verbatim** — it is a colon-joined chain, not one path.
|
|
- `kubectl config use-context …` in an interactive toolbox shell writes to the **first** file in
|
|
the chain (the selector), so a manual switch stays local to that container.
|
|
|
|
**Why a merge script and not `KUBECONFIG=dev.kubeconfig:prod.kubeconfig`.** k3s names the context,
|
|
the cluster *and* the user in **both** files `default`. kubectl's merge keeps the **first**
|
|
occurrence of a duplicated name, so a colon-joined `KUBECONFIG` silently resolves `default` to
|
|
**dev** and `--context default` can never reach prod — you would believe you were on prod while
|
|
issuing commands against the live dev cluster. `kubeconfig.sh` therefore **renames on merge**
|
|
(`k3s-dev` / `k3s-prod`, `*-admin` users) with `yq`, structurally — not with `sed`, which corrupts
|
|
these files because the cluster and the user entry are both `- name: default` at identical indent.
|
|
|
|
- Generator: [`kubeconfig.sh`](kubeconfig.sh). Output: `infra/ansible/.cache/clusters.kubeconfig`,
|
|
mode 0600, **regenerated on every `dev.sh run`** — a build artifact, never hand-edited, never
|
|
committed (`.cache/` and `*.kubeconfig` are both gitignored; it embeds cluster-admin certs).
|
|
Its content no longer depends on which cluster was asked for, so concurrent regenerations write
|
|
identical bytes; the write is published by atomic rename so a concurrent reader can never load a
|
|
half-written kubeconfig.
|
|
- A cluster whose source kubeconfig is missing is simply omitted from the merge, so a machine with
|
|
only `dev.kubeconfig` still works.
|
|
- **Asking for a cluster whose kubeconfig is absent is a hard error.** `dev.sh` refuses on the host
|
|
before starting a container and never falls back to the other cluster — that fallback is exactly
|
|
how a prod command lands on dev.
|
|
- `bootstrap/standup.sh` is unaffected: it exports its own `KUBECONFIG` from the mandatory
|
|
`--kubeconfig` flag, which overrides what `dev.sh` sets. Its `--expect-server` guard remains the
|
|
thing that makes it safe, not `CLUSTER`.
|
|
|
|
### `DEV_KUBECONFIG` — superseded, still honoured
|
|
|
|
`DEV_KUBECONFIG=<path> ./dev.sh run …` still pins `KUBECONFIG` verbatim and takes precedence over
|
|
`CLUSTER` (nothing depending on it breaks), but it is **deprecated**: the name says "dev" while the
|
|
established workaround set it to a *prod* path, and it produces a single-cluster config whose
|
|
context is the ambiguous `default`. Use `CLUSTER=dev|prod`. Invocations that use it print a warning.
|
|
|
|
## What `dev.sh` mounts, and why each one matters
|
|
|
|
- `--network host` — Ansible SSHes to the nodes through the pfSense NAT (`192.168.2.102:225x`) and
|
|
`kubectl`/`flux`/`tofu` reach the kube-API (`192.168.2.50:6443`) and the Proxmox API
|
|
(`192.168.2.102:8006`) on the host's network.
|
|
- `-v $REPO:/work` (workdir) — the repo.
|
|
- `-v ~/.config/sops:/root/.config/sops` — **the age keyring, read-write**. This is deliberate: the
|
|
private key generated by `bootstrap/sops/age-keygen.sh` must land on the *host*, or it dies with
|
|
the `--rm` container and every SOPS-encrypted secret in the repo becomes undecryptable.
|
|
- `-v ~/.ssh:/root/.ssh:ro` — node SSH key (`home_network`).
|
|
- `-v ~/.config/home-k3s/ansible-vault-pass:...:ro` — the Ansible vault passphrase, so vaulted
|
|
`group_vars` decrypt unattended.
|
|
- `-v ../home-network-k3s:/home-network-k3s:ro` + `SPEC_REPO` — the **spec repo**, read-only, when
|
|
it exists beside this one. Without it `ci/spec-immutability.sh` honestly *skips*
|
|
rather than passing — mount it, or that gate silently verifies nothing.
|
|
- `--security-opt label=disable` — SELinux hosts, so the bind mounts work without relabeling.
|
|
|
|
## tofu and ansible now run **inside** the container
|
|
|
|
This reverses the old Phase-1 limitation. When the dev cluster was on host libvirt, OpenTofu's
|
|
`dmacvicar/libvirt` provider could not do a read-write `qemu:///system` connect from inside a
|
|
container (polkit/PID-identity: `Cannot find start time for pid`), so tofu had to be extracted and
|
|
run on the host. The cluster moved to **Proxmox VE** and the provider is now `bpg/proxmox`, which
|
|
talks to the Proxmox **HTTP API** — nothing host-local is required. Ansible likewise only needs
|
|
SSH, which `--network host` provides.
|
|
|
|
So: run everything in here. The host needs only podman (or docker).
|
|
|
|
## kubectl inside the container — was a known gap, now fixed
|
|
|
|
`dev.sh` used to hard-wire `KUBECONFIG` to `dev.kubeconfig` with no supported way to reach prod, and
|
|
it does not mount `~/.kube`. It now selects between `infra/ansible/.cache/dev.kubeconfig` and
|
|
`prod.kubeconfig` via `CLUSTER=dev|prod` (see above) and merges both under distinct context names.
|
|
|
|
The container still needs a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login
|
|
needs a browser, which a container shell does not have. See
|
|
[`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both access paths and for how to regenerate these
|
|
two files from `/etc/rancher/k3s/k3s.yaml`.
|
|
|
|
## Notes
|
|
|
|
- Versions are **best-effort pins**; if a download 404s at build, bump the value in `versions.env`
|
|
and rebuild. Keep the conformance-tool block aligned with `ci/versions.env`.
|
|
- **Secrets never enter the image** — they arrive only as the bind mounts listed above.
|
|
- This container is the toolchain, not the cluster. The cluster runs on Proxmox VE; see
|
|
[`../PHASE1-STATUS.md`](../PHASE1-STATUS.md).
|