home-network-k3s-impl/infra/devcontainer/README.md

# Dev toolbox (containerized baseline)

The toolchain for this repo is **not all Python** (OpenTofu, `age`, `sops`, `flux`, `chainsaw`,
`kyverno` are Go binaries; only Ansible and a couple of linters are Python). Per the operator's
rule it is therefore delivered as a **pinned dev container** — no host pollution, one reproducible
baseline, and the *same* binaries CI uses.

**This is the supported way to run anything in this repo.** Don't install the toolchain on the host.

## What's inside (pinned in `versions.env`)

| Group | Tools |
|---|---|
| Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
| GitOps / cluster | `flux`, `kubectl`, `helm` |
| Secrets | `sops`, `age` + `age-keygen` |
| VPN (`platform/vpn-ingress`) | `wg` (wireguard-tools) — mint the server key, derive client public keys; `qrencode` — render a client profile as a phone-scannable QR |
| **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
| Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |

The conformance-tool versions are kept in lockstep with `ci/versions.env` so
`make -C ci gate` in here is byte-identical to the CI runner (**R-TEST-02**). The image records
the resolved set at `/etc/dev-toolchain-versions`.

## Use

```bash
cd infra/devcontainer
./dev.sh build                       # build the image (needs network)
./dev.sh                             # build if needed + shell in /work (the repo)
./dev.sh run make -C ci gate         # the merge gate, exactly as CI runs it
./dev.sh run sops -d path/to.sops.yaml
./dev.sh run tofu -chdir=/work/infra/tofu plan
```
`ENGINE=docker ./dev.sh` to use docker instead of rootless podman.

### The published toolbox

`./dev.sh build` delegates to [`build-image.sh`](build-image.sh), which is the **one** build
recipe: CI runs the same script (with its in-image verification — every tool executes, and the
version-sensitive ones report their `versions.env` pins) and, on a push to `main` that changes
`Containerfile`, `versions.env` or `build-image.sh`, **publishes** the result as
`gitea.home.keithjackson.dev/home-network/toolbox` (tags `sha-<commit>` and `main`) —
see [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). `agent-runner` is
built `FROM` that published digest (`infra/agent/versions.env`, kept current by Renovate).
Nobody pushes this image from a workstation. To use the published one locally instead of
building, after `podman login gitea.home.keithjackson.dev` (a token with `read:package`):

```bash
IMAGE=gitea.home.keithjackson.dev/home-network/toolbox:main ./dev.sh run make -C ci gate
```

## Choosing a cluster — `CLUSTER=dev|prod`

```bash
./dev.sh run kubectl get nodes                 # DEV — the default, unchanged
CLUSTER=prod ./dev.sh run kubectl get nodes    # PROD
CLUSTER=prod ./dev.sh                          # interactive shell pointed at prod
```

An env-var **prefix**, so the target sits on the same line as the command. Every invocation prints
a banner naming the cluster before the container starts, because `CLUSTER` can also be inherited
from an exported shell variable.

Inside the container `KUBECONFIG` is a two-file **chain**,
`<per-invocation selector>:<shared merged config>`:

| file | shared? | holds |
|---|---|---|
| selector, in a private `mktemp` dir | no — one per `dev.sh` invocation | only `current-context: k3s-$CLUSTER`; no credentials |
| `infra/ansible/.cache/clusters.kubeconfig` | yes — bind-mounted, read by every container | both clusters under unambiguous names; **no `current-context`** |

kubectl merges the chain first-wins, so the selector supplies this invocation's default:

```
CURRENT   NAME       CLUSTER    AUTHINFO
*         k3s-dev    k3s-dev    k3s-dev-admin
          k3s-prod   k3s-prod   k3s-prod-admin
```

So `kubectl config current-context` answers "which cluster am I on?" honestly, and
`kubectl --context k3s-prod …` works from a dev-defaulted shell.

**Why the default context is not stored in the merged file.** It used to be, and that was a
cross-cluster data-corruption bug. The merged file lives in the repo's `.cache`, which every
container bind-mounts, so `current-context` was **shared mutable state**: two concurrent `dev.sh`
runs with different `CLUSTER` values overwrote each other's default *mid-command*. A
`CLUSTER=prod` script read prod for its first half and **dev** for its second; it was caught only
because a dev-only PVC name showed up in a prod query. Moving selection into a per-invocation file
removes the shared field rather than locking it — there is no window left to lose. Read on its own
the merged file now resolves to nothing (`error: current-context is not set`), which is loud and
safe; `--context k3s-dev` / `--context k3s-prod` against it are unaffected and remain the pattern
to prefer in scripts. Concurrent `CLUSTER=dev` and `CLUSTER=prod` invocations are now safe.

Consequences worth knowing:

- Export what `kubeconfig.sh` prints **verbatim** — it is a colon-joined chain, not one path.
- `kubectl config use-context …` in an interactive toolbox shell writes to the **first** file in
  the chain (the selector), so a manual switch stays local to that container.

**Why a merge script and not `KUBECONFIG=dev.kubeconfig:prod.kubeconfig`.** k3s names the context,
the cluster *and* the user in **both** files `default`. kubectl's merge keeps the **first**
occurrence of a duplicated name, so a colon-joined `KUBECONFIG` silently resolves `default` to
**dev** and `--context default` can never reach prod — you would believe you were on prod while
issuing commands against the live dev cluster. `kubeconfig.sh` therefore **renames on merge**
(`k3s-dev` / `k3s-prod`, `*-admin` users) with `yq`, structurally — not with `sed`, which corrupts
these files because the cluster and the user entry are both `- name: default` at identical indent.

- Generator: [`kubeconfig.sh`](kubeconfig.sh). Output: `infra/ansible/.cache/clusters.kubeconfig`,
  mode 0600, **regenerated on every `dev.sh run`** — a build artifact, never hand-edited, never
  committed (`.cache/` and `*.kubeconfig` are both gitignored; it embeds cluster-admin certs).
  Its content no longer depends on which cluster was asked for, so concurrent regenerations write
  identical bytes; the write is published by atomic rename so a concurrent reader can never load a
  half-written kubeconfig.
- A cluster whose source kubeconfig is missing is simply omitted from the merge, so a machine with
  only `dev.kubeconfig` still works.
- **Asking for a cluster whose kubeconfig is absent is a hard error.** `dev.sh` refuses on the host
  before starting a container and never falls back to the other cluster — that fallback is exactly
  how a prod command lands on dev.
- `bootstrap/standup.sh` is unaffected: it exports its own `KUBECONFIG` from the mandatory
  `--kubeconfig` flag, which overrides what `dev.sh` sets. Its `--expect-server` guard remains the
  thing that makes it safe, not `CLUSTER`.

### `DEV_KUBECONFIG` — superseded, still honoured

`DEV_KUBECONFIG=<path> ./dev.sh run …` still pins `KUBECONFIG` verbatim and takes precedence over
`CLUSTER` (nothing depending on it breaks), but it is **deprecated**: the name says "dev" while the
established workaround set it to a *prod* path, and it produces a single-cluster config whose
context is the ambiguous `default`. Use `CLUSTER=dev|prod`. Invocations that use it print a warning.

## What `dev.sh` mounts, and why each one matters

- `--network host` — Ansible SSHes to the nodes through the pfSense NAT (`192.168.2.102:225x`) and
  `kubectl`/`flux`/`tofu` reach the kube-API (`192.168.2.50:6443`) and the Proxmox API
  (`192.168.2.102:8006`) on the host's network.
- `-v $REPO:/work` (workdir) — the repo.
- `-v ~/.config/sops:/root/.config/sops` — **the age keyring, read-write**. This is deliberate: the
  private key generated by `bootstrap/sops/age-keygen.sh` must land on the *host*, or it dies with
  the `--rm` container and every SOPS-encrypted secret in the repo becomes undecryptable.
- `-v ~/.ssh:/root/.ssh:ro` — node SSH key (`home_network`).
- `-v ~/.config/home-k3s/ansible-vault-pass:...:ro` — the Ansible vault passphrase, so vaulted
  `group_vars` decrypt unattended.
- `-v ../home-network-k3s:/home-network-k3s:ro` + `SPEC_REPO` — the **spec repo**, read-only, when
  it exists beside this one. Without it `ci/spec-immutability.sh` honestly *skips*
  rather than passing — mount it, or that gate silently verifies nothing.
- `--security-opt label=disable` — SELinux hosts, so the bind mounts work without relabeling.

## tofu and ansible now run **inside** the container

This reverses the old Phase-1 limitation. When the dev cluster was on host libvirt, OpenTofu's
`dmacvicar/libvirt` provider could not do a read-write `qemu:///system` connect from inside a
container (polkit/PID-identity: `Cannot find start time for pid`), so tofu had to be extracted and
run on the host. The cluster moved to **Proxmox VE** and the provider is now `bpg/proxmox`, which
talks to the Proxmox **HTTP API** — nothing host-local is required. Ansible likewise only needs
SSH, which `--network host` provides.

So: run everything in here. The host needs only podman (or docker).

## kubectl inside the container — was a known gap, now fixed

`dev.sh` used to hard-wire `KUBECONFIG` to `dev.kubeconfig` with no supported way to reach prod, and
it does not mount `~/.kube`. It now selects between `infra/ansible/.cache/dev.kubeconfig` and
`prod.kubeconfig` via `CLUSTER=dev|prod` (see above) and merges both under distinct context names.

The container still needs a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login
needs a browser, which a container shell does not have. See
[`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both access paths and for how to regenerate these
two files from `/etc/rancher/k3s/k3s.yaml`.

## Notes

- Versions are **best-effort pins**; if a download 404s at build, bump the value in `versions.env`
  and rebuild. Keep the conformance-tool block aligned with `ci/versions.env`.
- **Secrets never enter the image** — they arrive only as the bind mounts listed above.
- This container is the toolchain, not the cluster. The cluster runs on Proxmox VE; see
  [`../PHASE1-STATUS.md`](../PHASE1-STATUS.md).