feat(devcontainer): dev.sh can target prod — and cannot silently target the wrong cluster

Operator report: dev.sh only reached the dev cluster. Line 72 hardcoded
`KUBECONFIG=${DEV_KUBECONFIG:-…/dev.kubeconfig}`, and the only escape was an
undocumented override whose NAME misleads — you set `DEV_KUBECONFIG` to a *prod*
path. infra/PHASE1-STATUS.md had recorded this as a "Known gap" for months.

WHY THIS WAS NOT A ONE-LINER. Both kubeconfigs name their context, cluster AND
user `default`. kubectl's merge keeps the FIRST occurrence of a duplicated name,
so the obvious `KUBECONFIG=dev.kubeconfig:prod.kubeconfig` resolves `default` to
DEV and `--context default` can never select prod — an operator or agent would
believe they were on prod and be issuing commands against the LIVE dev cluster
that runs the operator's platform. Same collision class as ~/.kube/config an hour
earlier, and worse here because agents run in this container.

Interface: `CLUSTER=dev|prod` as an env prefix, default **dev** (back-compat: every
existing invocation, CI job and habit keeps meaning dev).

    infra/devcontainer/dev.sh run kubectl get nodes                # dev
    CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes   # prod

Its one weakness — `CLUSTER` can be inherited from an exported shell var — is
covered by a banner printed on EVERY invocation before the container starts:
`==> dev.sh: cluster=prod (kube context k3s-prod)`.

New infra/devcontainer/kubeconfig.sh generates a merged config with DISTINCT names
(contexts k3s-dev/k3s-prod, users k3s-*-admin), renaming structurally with `yq`
on .clusters[].name / .users[].name / .contexts[].{name,context.cluster,
context.user} before letting `kubectl config view --flatten` merge them. NOT sed —
sed renaming corrupted a config earlier today because cluster and user entries are
both `- name: default` at identical indent. Output is 0600 in the gitignored
.cache/, regenerated every run, and post-conditions assert the current context is
`k3s-<target>` and that NO context named `default` survived.

FAILS LOUDLY AT BOTH LAYERS, and never falls back to the other cluster — that
fallback is exactly how a prod command ends up hitting dev. Host-side refusal if
the requested cluster's kubeconfig is missing; in-container refusal likewise;
unknown CLUSTER exits 2. A missing source for the cluster you did NOT ask for is
tolerated (merges the other one only).

VERIFIED BY ME, independently of the agent — the two BASE_DOMAIN reads are the
unambiguous proof of which cluster was reached, since a node list is not:
  default (no selector)  -> home.keithjackson.dev        (DEV, unchanged)
  CLUSTER=prod           -> prod.home.keithjackson.dev   (PROD)
  contexts               -> k3s-dev, k3s-prod; NO `default` context survives
  exit codes             -> invalid CLUSTER=2, valid dev/prod=0
  standup.sh             -> `--kubeconfig` still wins over the new default
                            (preflight against prod: target confirmed 192.168.2.9)
  credentials            -> dev.kubeconfig and prod.kubeconfig byte-intact and each
                            still reaching 3 nodes after the agent's move-aside test
  git                    -> no kubeconfig committable; clusters.kubeconfig ignored
Full CI gate PASS.

NOT VERIFIED: `dev.sh build` and the interactive shell path (existing image reused,
no TTY available). The shell path shares run()'s code; only the TTY differs.

DEV_KUBECONFIG still wins verbatim, now with a deprecation warning. All existing
mounts are untouched: SOPS age keyring rw, ~/.ssh ro, ansible-vault-pass ro, spec
repo ro + SPEC_REPO, --network host, label=disable.

Closes the PHASE1-STATUS "Known gap".

Refs R-ENV-02, R-TEST-02.

Co-Authored-By: Claude Opus 5 <[email protected]>

infra/PHASE1-STATUS.md

@@ -84,6 +84,9 @@ toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image
 infra/devcontainer/dev.sh build                    # once
 infra/devcontainer/dev.sh run make -C ci gate      # exactly what CI runs
 infra/devcontainer/dev.sh                          # interactive shell at /work
+
+infra/devcontainer/dev.sh run kubectl get nodes                # DEV (the default)
+CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes   # PROD
 ```
 
 Unlike the libvirt era, **tofu and ansible now run fine inside the container** — the
@@ -95,10 +98,21 @@ which is what broke the old `dmacvicar/libvirt` provider in a container.
 file, and the sibling spec repo read-only at `/home-network-k3s` so the spec-immutability
 check runs instead of silently skipping.
 
-> **Known gap.** `dev.sh` defaults `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`,
-> a libvirt-era path that no longer exists, and does not mount `~/.kube`. In-container `kubectl`
-> therefore needs an explicit `DEV_KUBECONFIG=<path to a break-glass cert kubeconfig>` — the OIDC
-> context cannot work there anyway (no browser). Not yet fixed.
+> **Closed (was a known gap): `dev.sh` can now target either cluster.** It used to hard-wire
+> `KUBECONFIG` to `dev.kubeconfig`, leaving an undocumented — and misleadingly named —
+> `DEV_KUBECONFIG=<prod path>` as the only route to prod. Select with **`CLUSTER=dev|prod`**;
+> **dev remains the default**, so every existing invocation means what it always meant. Inside the
+> container `KUBECONFIG` is a merged config carrying both clusters as contexts **`k3s-dev`** and
+> **`k3s-prod`** (never `default`), regenerated each run by
+> [`infra/devcontainer/kubeconfig.sh`](devcontainer/kubeconfig.sh) into the gitignored
+> `infra/ansible/.cache/clusters.kubeconfig`. Naming matters: k3s writes *both* source kubeconfigs
+> with context, cluster and user all called `default`, and kubectl's merge keeps the first
+> duplicate — so a plain `KUBECONFIG=dev:prod` would silently resolve `default` to **dev** while you
+> believed you were on prod. Requesting a cluster whose kubeconfig is absent is a hard error, never
+> a fallback. `DEV_KUBECONFIG` still works and still wins, but is **superseded** — it is opaque
+> about its target and yields an ambiguous `default` context. `~/.kube` is still not mounted and the
+> OIDC context still cannot work in a container (no browser); use the break-glass cert kubeconfigs
+> above. Details: [`devcontainer/README.md`](devcontainer/README.md).
 
 ## Off-repo state you must not lose