home-network-k3s-impl/infra/devcontainer

Dev toolbox (containerized baseline)

The toolchain for this repo is not all Python (OpenTofu, age, sops, flux, chainsaw, kyverno are Go binaries; only Ansible and a couple of linters are Python). Per the operator's rule it is therefore delivered as a pinned dev container — no host pollution, one reproducible baseline, and the same binaries CI uses.

This is the supported way to run anything in this repo. Don't install the toolchain on the host.

What's inside (pinned in versions.env)

Group	Tools
Provisioning	OpenTofu (bpg/proxmox provider), ansible-core, openssh-clients
GitOps / cluster	flux, kubectl, helm
Secrets	sops, age + age-keygen
VPN (platform/vpn-ingress)	wg (wireguard-tools) — mint the server key, derive client public keys; qrencode — render a client profile as a phone-scannable QR
CI gate	yamllint, check-jsonschema, kyverno CLI, kyverno-json, chainsaw, yq, diffutils, make
Misc	qemu-img, cloud-utils/genisoimage, libvirt-client (disk-image utilities only — leftovers from the libvirt era, harmless)

The conformance-tool versions are kept in lockstep with ci/versions.env so make -C ci gate in here is byte-identical to the CI runner (R-TEST-02). The image records the resolved set at /etc/dev-toolchain-versions.

Use

cd infra/devcontainer
./dev.sh build                       # build the image (needs network)
./dev.sh                             # build if needed + shell in /work (the repo)
./dev.sh run make -C ci gate         # the merge gate, exactly as CI runs it
./dev.sh run sops -d path/to.sops.yaml
./dev.sh run tofu -chdir=/work/infra/tofu plan

ENGINE=docker ./dev.sh to use docker instead of rootless podman.

The published toolbox

./dev.sh build delegates to build-image.sh. The recipe is the Containerfile plus that script's build-arg list, and the proof is the Containerfile's verify stage (every tool on PATH, the version-sensitive ones report their versions.env pins, no network). CI builds the same Containerfile on rootless BuildKit with the args from build-image.sh --print-build-args, and, on a push to main that changes Containerfile, versions.env or build-image.sh, publishes the result as gitea.home.keithjackson.dev/home-network/toolbox (tags sha-<commit> and main) — see .gitea/workflows/images.yaml. agent-runner is built FROM that published digest (infra/agent/versions.env, kept current by Renovate). Nobody pushes this image from a workstation. To use the published one locally instead of building, after podman login gitea.home.keithjackson.dev (a token with read:package):

IMAGE=gitea.home.keithjackson.dev/home-network/toolbox:main ./dev.sh run make -C ci gate

Choosing a cluster — CLUSTER=dev|prod

./dev.sh run kubectl get nodes                 # DEV — the default, unchanged
CLUSTER=prod ./dev.sh run kubectl get nodes    # PROD
CLUSTER=prod ./dev.sh                          # interactive shell pointed at prod

An env-var prefix, so the target sits on the same line as the command. Every invocation prints a banner naming the cluster before the container starts, because CLUSTER can also be inherited from an exported shell variable.

Inside the container KUBECONFIG is a two-file chain, <per-invocation selector>:<shared merged config>:

file	shared?	holds
selector, in a private mktemp dir	no — one per dev.sh invocation	only current-context: k3s-$CLUSTER; no credentials
infra/ansible/.cache/clusters.kubeconfig	yes — bind-mounted, read by every container	both clusters under unambiguous names; no current-context

kubectl merges the chain first-wins, so the selector supplies this invocation's default:

CURRENT   NAME       CLUSTER    AUTHINFO
*         k3s-dev    k3s-dev    k3s-dev-admin
          k3s-prod   k3s-prod   k3s-prod-admin

So kubectl config current-context answers "which cluster am I on?" honestly, and kubectl --context k3s-prod … works from a dev-defaulted shell.

Why the default context is not stored in the merged file. It used to be, and that was a cross-cluster data-corruption bug. The merged file lives in the repo's .cache, which every container bind-mounts, so current-context was shared mutable state: two concurrent dev.sh runs with different CLUSTER values overwrote each other's default mid-command. A CLUSTER=prod script read prod for its first half and dev for its second; it was caught only because a dev-only PVC name showed up in a prod query. Moving selection into a per-invocation file removes the shared field rather than locking it — there is no window left to lose. Read on its own the merged file now resolves to nothing (error: current-context is not set), which is loud and safe; --context k3s-dev / --context k3s-prod against it are unaffected and remain the pattern to prefer in scripts. Concurrent CLUSTER=dev and CLUSTER=prod invocations are now safe.

Consequences worth knowing:

• Export what kubeconfig.sh prints verbatim — it is a colon-joined chain, not one path.
• kubectl config use-context … in an interactive toolbox shell writes to the first file in the chain (the selector), so a manual switch stays local to that container.

Why a merge script and not KUBECONFIG=dev.kubeconfig:prod.kubeconfig. k3s names the context, the cluster and the user in both files default. kubectl's merge keeps the first occurrence of a duplicated name, so a colon-joined KUBECONFIG silently resolves default to dev and --context default can never reach prod — you would believe you were on prod while issuing commands against the live dev cluster. kubeconfig.sh therefore renames on merge (k3s-dev / k3s-prod, *-admin users) with yq, structurally — not with sed, which corrupts these files because the cluster and the user entry are both - name: default at identical indent.

• Generator: kubeconfig.sh. Output: infra/ansible/.cache/clusters.kubeconfig, mode 0600, regenerated on every dev.sh run — a build artifact, never hand-edited, never committed (.cache/ and *.kubeconfig are both gitignored; it embeds cluster-admin certs). Its content no longer depends on which cluster was asked for, so concurrent regenerations write identical bytes; the write is published by atomic rename so a concurrent reader can never load a half-written kubeconfig.
• A cluster whose source kubeconfig is missing is simply omitted from the merge, so a machine with only dev.kubeconfig still works.
• Asking for a cluster whose kubeconfig is absent is a hard error. dev.sh refuses on the host before starting a container and never falls back to the other cluster — that fallback is exactly how a prod command lands on dev.
• bootstrap/standup.sh is unaffected: it exports its own KUBECONFIG from the mandatory --kubeconfig flag, which overrides what dev.sh sets. Its --expect-server guard remains the thing that makes it safe, not CLUSTER.

DEV_KUBECONFIG — superseded, still honoured

DEV_KUBECONFIG=<path> ./dev.sh run … still pins KUBECONFIG verbatim and takes precedence over CLUSTER (nothing depending on it breaks), but it is deprecated: the name says "dev" while the established workaround set it to a prod path, and it produces a single-cluster config whose context is the ambiguous default. Use CLUSTER=dev|prod. Invocations that use it print a warning.

What dev.sh mounts, and why each one matters

• --network host — Ansible SSHes to the nodes through the pfSense NAT (192.168.2.102:225x) and kubectl/flux/tofu reach the kube-API (192.168.2.50:6443) and the Proxmox API (192.168.2.102:8006) on the host's network.
• -v $REPO:/work (workdir) — the repo.
• -v ~/.config/sops:/root/.config/sops — the age keyring, read-write. This is deliberate: the private key generated by bootstrap/sops/age-keygen.sh must land on the host, or it dies with the --rm container and every SOPS-encrypted secret in the repo becomes undecryptable.
• -v ~/.ssh:/root/.ssh:ro — node SSH key (home_network).
• -v ~/.config/home-k3s/ansible-vault-pass:...:ro — the Ansible vault passphrase, so vaulted group_vars decrypt unattended.
• -v ../home-network-k3s:/home-network-k3s:ro + SPEC_REPO — the spec repo, read-only, when it exists beside this one. Without it ci/spec-immutability.sh honestly skips rather than passing — mount it, or that gate silently verifies nothing.
• --security-opt label=disable — SELinux hosts, so the bind mounts work without relabeling.

tofu and ansible now run inside the container

This reverses the old Phase-1 limitation. When the dev cluster was on host libvirt, OpenTofu's dmacvicar/libvirt provider could not do a read-write qemu:///system connect from inside a container (polkit/PID-identity: Cannot find start time for pid), so tofu had to be extracted and run on the host. The cluster moved to Proxmox VE and the provider is now bpg/proxmox, which talks to the Proxmox HTTP API — nothing host-local is required. Ansible likewise only needs SSH, which --network host provides.

So: run everything in here. The host needs only podman (or docker).

kubectl inside the container — was a known gap, now fixed

dev.sh used to hard-wire KUBECONFIG to dev.kubeconfig with no supported way to reach prod, and it does not mount ~/.kube. It now selects between infra/ansible/.cache/dev.kubeconfig and prod.kubeconfig via CLUSTER=dev|prod (see above) and merges both under distinct context names.

The container still needs a break-glass cert kubeconfig, not the k3s-oidc context — OIDC login needs a browser, which a container shell does not have. See ../PHASE1-STATUS.md for both access paths and for how to regenerate these two files from /etc/rancher/k3s/k3s.yaml.

Notes

• Versions are best-effort pins; if a download 404s at build, bump the value in versions.env and rebuild. Keep the conformance-tool block aligned with ci/versions.env.
• Secrets never enter the image — they arrive only as the bind mounts listed above.
• This container is the toolchain, not the cluster. The cluster runs on Proxmox VE; see ../PHASE1-STATUS.md.