feat(devcontainer): dev.sh can target prod — and cannot silently target the wrong cluster

Operator report: dev.sh only reached the dev cluster. Line 72 hardcoded
`KUBECONFIG=${DEV_KUBECONFIG:-…/dev.kubeconfig}`, and the only escape was an
undocumented override whose NAME misleads — you set `DEV_KUBECONFIG` to a *prod*
path. infra/PHASE1-STATUS.md had recorded this as a "Known gap" for months.

WHY THIS WAS NOT A ONE-LINER. Both kubeconfigs name their context, cluster AND
user `default`. kubectl's merge keeps the FIRST occurrence of a duplicated name,
so the obvious `KUBECONFIG=dev.kubeconfig:prod.kubeconfig` resolves `default` to
DEV and `--context default` can never select prod — an operator or agent would
believe they were on prod and be issuing commands against the LIVE dev cluster
that runs the operator's platform. Same collision class as ~/.kube/config an hour
earlier, and worse here because agents run in this container.

Interface: `CLUSTER=dev|prod` as an env prefix, default **dev** (back-compat: every
existing invocation, CI job and habit keeps meaning dev).

    infra/devcontainer/dev.sh run kubectl get nodes                # dev
    CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes   # prod

Its one weakness — `CLUSTER` can be inherited from an exported shell var — is
covered by a banner printed on EVERY invocation before the container starts:
`==> dev.sh: cluster=prod (kube context k3s-prod)`.

New infra/devcontainer/kubeconfig.sh generates a merged config with DISTINCT names
(contexts k3s-dev/k3s-prod, users k3s-*-admin), renaming structurally with `yq`
on .clusters[].name / .users[].name / .contexts[].{name,context.cluster,
context.user} before letting `kubectl config view --flatten` merge them. NOT sed —
sed renaming corrupted a config earlier today because cluster and user entries are
both `- name: default` at identical indent. Output is 0600 in the gitignored
.cache/, regenerated every run, and post-conditions assert the current context is
`k3s-<target>` and that NO context named `default` survived.

FAILS LOUDLY AT BOTH LAYERS, and never falls back to the other cluster — that
fallback is exactly how a prod command ends up hitting dev. Host-side refusal if
the requested cluster's kubeconfig is missing; in-container refusal likewise;
unknown CLUSTER exits 2. A missing source for the cluster you did NOT ask for is
tolerated (merges the other one only).

VERIFIED BY ME, independently of the agent — the two BASE_DOMAIN reads are the
unambiguous proof of which cluster was reached, since a node list is not:
  default (no selector)  -> home.keithjackson.dev        (DEV, unchanged)
  CLUSTER=prod           -> prod.home.keithjackson.dev   (PROD)
  contexts               -> k3s-dev, k3s-prod; NO `default` context survives
  exit codes             -> invalid CLUSTER=2, valid dev/prod=0
  standup.sh             -> `--kubeconfig` still wins over the new default
                            (preflight against prod: target confirmed 192.168.2.9)
  credentials            -> dev.kubeconfig and prod.kubeconfig byte-intact and each
                            still reaching 3 nodes after the agent's move-aside test
  git                    -> no kubeconfig committable; clusters.kubeconfig ignored
Full CI gate PASS.

NOT VERIFIED: `dev.sh build` and the interactive shell path (existing image reused,
no TTY available). The shell path shares run()'s code; only the TTY differs.

DEV_KUBECONFIG still wins verbatim, now with a deprecation warning. All existing
mounts are untouched: SOPS age keyring rw, ~/.ssh ro, ansible-vault-pass ro, spec
repo ro + SPEC_REPO, --network host, label=disable.

Closes the PHASE1-STATUS "Known gap".

Refs R-ENV-02, R-TEST-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-05 16:38:32 -05:00
1 parent 577c62136b
commit 21e111b08c
5 files changed
+268 -17

No files matched your search

+11
View File
@@ -69,6 +69,17 @@ infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
```
`kubectl`/`flux`/`helm` in there talk to the **dev** cluster by default. Prefix `CLUSTER=prod` to
target prod instead:
```bash
infra/devcontainer/dev.sh run kubectl get nodes # dev (default)
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # prod
```
Both clusters are always present in the container's `KUBECONFIG` as the contexts `k3s-dev` and
`k3s-prod`; `CLUSTER` picks which one is current, and each run prints the cluster it is targeting.
The container mounts the repo at `/work`, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
[`infra/devcontainer/README.md`](infra/devcontainer/README.md).
+18 -4
View File
@@ -84,6 +84,9 @@ toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # exactly what CI runs
infra/devcontainer/dev.sh # interactive shell at /work
infra/devcontainer/dev.sh run kubectl get nodes # DEV (the default)
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # PROD
```
Unlike the libvirt era, **tofu and ansible now run fine inside the container** — the
@@ -95,10 +98,21 @@ which is what broke the old `dmacvicar/libvirt` provider in a container.
file, and the sibling spec repo read-only at `/home-network-k3s` so the spec-immutability
check runs instead of silently skipping.
> **Known gap.** `dev.sh` defaults `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`,
> a libvirt-era path that no longer exists, and does not mount `~/.kube`. In-container `kubectl`
> therefore needs an explicit `DEV_KUBECONFIG=<path to a break-glass cert kubeconfig>` — the OIDC
> context cannot work there anyway (no browser). Not yet fixed.
> **Closed (was a known gap): `dev.sh` can now target either cluster.** It used to hard-wire
> `KUBECONFIG` to `dev.kubeconfig`, leaving an undocumented — and misleadingly named —
> `DEV_KUBECONFIG=<prod path>` as the only route to prod. Select with **`CLUSTER=dev|prod`**;
> **dev remains the default**, so every existing invocation means what it always meant. Inside the
> container `KUBECONFIG` is a merged config carrying both clusters as contexts **`k3s-dev`** and
> **`k3s-prod`** (never `default`), regenerated each run by
> [`infra/devcontainer/kubeconfig.sh`](devcontainer/kubeconfig.sh) into the gitignored
> `infra/ansible/.cache/clusters.kubeconfig`. Naming matters: k3s writes *both* source kubeconfigs
> with context, cluster and user all called `default`, and kubectl's merge keeps the first
> duplicate — so a plain `KUBECONFIG=dev:prod` would silently resolve `default` to **dev** while you
> believed you were on prod. Requesting a cluster whose kubeconfig is absent is a hard error, never
> a fallback. `DEV_KUBECONFIG` still works and still wins, but is **superseded** — it is opaque
> about its target and yields an ambiguous `default` context. `~/.kube` is still not mounted and the
> OIDC context still cannot work in a container (no browser); use the break-glass cert kubeconfigs
> above. Details: [`devcontainer/README.md`](devcontainer/README.md).
## Off-repo state you must not lose
+59 -10
View File
@@ -33,6 +33,57 @@ cd infra/devcontainer
```
`ENGINE=docker ./dev.sh` to use docker instead of rootless podman.
## Choosing a cluster — `CLUSTER=dev|prod`
```bash
./dev.sh run kubectl get nodes # DEV — the default, unchanged
CLUSTER=prod ./dev.sh run kubectl get nodes # PROD
CLUSTER=prod ./dev.sh # interactive shell pointed at prod
```
An env-var **prefix**, so the target sits on the same line as the command. Every invocation prints
a banner naming the cluster before the container starts, because `CLUSTER` can also be inherited
from an exported shell variable.
Inside the container `KUBECONFIG` points at a **merged** config holding both clusters under
unambiguous names, with `CLUSTER`'s context current:
```
CURRENT NAME CLUSTER AUTHINFO
* k3s-dev k3s-dev k3s-dev-admin
k3s-prod k3s-prod k3s-prod-admin
```
So `kubectl config current-context` answers "which cluster am I on?" honestly, and
`kubectl --context k3s-prod …` works from a dev-defaulted shell.
**Why a merge script and not `KUBECONFIG=dev.kubeconfig:prod.kubeconfig`.** k3s names the context,
the cluster *and* the user in **both** files `default`. kubectl's merge keeps the **first**
occurrence of a duplicated name, so a colon-joined `KUBECONFIG` silently resolves `default` to
**dev** and `--context default` can never reach prod — you would believe you were on prod while
issuing commands against the live dev cluster. `kubeconfig.sh` therefore **renames on merge**
(`k3s-dev` / `k3s-prod`, `*-admin` users) with `yq`, structurally — not with `sed`, which corrupts
these files because the cluster and the user entry are both `- name: default` at identical indent.
- Generator: [`kubeconfig.sh`](kubeconfig.sh). Output: `infra/ansible/.cache/clusters.kubeconfig`,
mode 0600, **regenerated on every `dev.sh run`** — a build artifact, never hand-edited, never
committed (`.cache/` and `*.kubeconfig` are both gitignored; it embeds cluster-admin certs).
- A cluster whose source kubeconfig is missing is simply omitted from the merge, so a machine with
only `dev.kubeconfig` still works.
- **Asking for a cluster whose kubeconfig is absent is a hard error.** `dev.sh` refuses on the host
before starting a container and never falls back to the other cluster — that fallback is exactly
how a prod command lands on dev.
- `bootstrap/standup.sh` is unaffected: it exports its own `KUBECONFIG` from the mandatory
`--kubeconfig` flag, which overrides what `dev.sh` sets. Its `--expect-server` guard remains the
thing that makes it safe, not `CLUSTER`.
### `DEV_KUBECONFIG` — superseded, still honoured
`DEV_KUBECONFIG=<path> ./dev.sh run …` still pins `KUBECONFIG` verbatim and takes precedence over
`CLUSTER` (nothing depending on it breaks), but it is **deprecated**: the name says "dev" while the
established workaround set it to a *prod* path, and it produces a single-cluster config whose
context is the ambiguous `default`. Use `CLUSTER=dev|prod`. Invocations that use it print a warning.
## What `dev.sh` mounts, and why each one matters
- `--network host` — Ansible SSHes to the nodes through the pfSense NAT (`192.168.2.102:225x`) and
@@ -61,18 +112,16 @@ SSH, which `--network host` provides.
So: run everything in here. The host needs only podman (or docker).
## kubectl inside the container — known gap
## kubectl inside the container — was a known gap, now fixed
`dev.sh` sets `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`, a **libvirt-era path
that no longer exists**, and does not mount `~/.kube`. Override it explicitly:
`dev.sh` used to hard-wire `KUBECONFIG` to `dev.kubeconfig` with no supported way to reach prod, and
it does not mount `~/.kube`. It now selects between `infra/ansible/.cache/dev.kubeconfig` and
`prod.kubeconfig` via `CLUSTER=dev|prod` (see above) and merges both under distinct context names.
```bash
DEV_KUBECONFIG=/work/path/to/break-glass.kubeconfig ./dev.sh run kubectl get nodes
```
Use a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login needs a browser,
which a container shell does not have. See [`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both
access paths. (Not yet fixed in `dev.sh`.)
The container still needs a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login
needs a browser, which a container shell does not have. See
[`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both access paths and for how to regenerate these
two files from `/etc/rancher/k3s/k3s.yaml`.
## Notes
+62 -3
View File
@@ -11,7 +11,24 @@
# ./dev.sh build # build the image only
# ./dev.sh # build (if needed) + open a shell
# ./dev.sh run tofu -version # run one command in the toolbox
# ./dev.sh run kubectl get nodes # talks to the Proxmox k3s cluster directly (no tunnel)
# ./dev.sh run kubectl get nodes # talks to the DEV k3s cluster directly (no tunnel)
#
# CHOOSING A CLUSTER — `CLUSTER=<dev|prod>`, defaulting to dev:
#
# ./dev.sh run kubectl get nodes # dev (the default; unchanged behaviour)
# CLUSTER=prod ./dev.sh run kubectl get nodes # prod
# CLUSTER=prod ./dev.sh # interactive shell pointed at prod
#
# The selector is an env-var PREFIX so the target is visible on the same line as the command, and
# every invocation prints a banner naming the cluster it is about to talk to. Inside the container
# KUBECONFIG is a MERGED config holding both clusters under unambiguous names — contexts `k3s-dev`
# and `k3s-prod`, never `default` — with `CLUSTER`'s context selected as current. So `kubectl
# config current-context` always answers the "which cluster am I on?" question honestly, and
# `--context k3s-prod` actually works. See infra/devcontainer/kubeconfig.sh for why the naive
# `KUBECONFIG=dev:prod` merge is dangerous (both source files name everything `default`).
#
# Requesting a cluster whose kubeconfig is missing is a hard ERROR — it never falls back to the
# other cluster.
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
@@ -24,6 +41,15 @@ source "${HERE}/versions.env"
# the pfSense NAT (192.168.2.50:6443 -> kube-vip VIP 192.168.1.49:6443). No SSH tunnel is
# opened. infra/ansible/.cache/dev.kubeconfig targets https://192.168.2.50:6443 with
# tls-server-name=192.168.1.49 so TLS validates against the VIP's SAN through the NAT.
# Prod is reached the same way via infra/ansible/.cache/prod.kubeconfig (https://192.168.2.9:6443).
# ------------------------------------------------------------------ cluster selection
CLUSTER="${CLUSTER:-dev}"
case "${CLUSTER}" in
dev|prod) ;;
*) printf '\033[31mERROR: CLUSTER=%s is not a known cluster (dev|prod).\033[0m\n' "${CLUSTER}" >&2; exit 2 ;;
esac
CLUSTER_KUBECONFIG="${REPO_ROOT}/infra/ansible/.cache/${CLUSTER}.kubeconfig"
build() {
"$ENGINE" build \
@@ -61,6 +87,26 @@ run() {
if [ -n "${spec_host}" ] && [ -d "${spec_host}/.git" ]; then
spec_mount=(-v "${spec_host}:/home-network-k3s:ro" -e SPEC_REPO=/home-network-k3s)
fi
# Which cluster, said out loud, before anything runs. CLUSTER can be inherited from an exported
# shell variable, so the banner is what stops "I thought I was on dev".
if [ -n "${DEV_KUBECONFIG:-}" ]; then
# SUPERSEDED but still honoured: DEV_KUBECONFIG used to be the only way to reach a cluster other
# than the (then non-existent) default dev path. It wins over CLUSTER so nothing that depends on
# it breaks, but it is opaque about its target — prefer CLUSTER=dev|prod.
printf '\033[33m==> dev.sh: KUBECONFIG pinned by DEV_KUBECONFIG=%s (CLUSTER=%s ignored; DEV_KUBECONFIG is superseded — use CLUSTER=dev|prod)\033[0m\n' \
"${DEV_KUBECONFIG}" "${CLUSTER}" >&2
else
# Fail here, on the host, before a container is even started: a missing kubeconfig must never
# degrade into "whatever config we could find", because that is how a prod command lands on dev.
if [ ! -f "${CLUSTER_KUBECONFIG}" ]; then
printf '\033[31mERROR: dev.sh: CLUSTER=%s but its kubeconfig is MISSING: %s\n Refusing to run. dev.sh does NOT fall back to the other cluster.\033[0m\n' \
"${CLUSTER}" "${CLUSTER_KUBECONFIG}" >&2
exit 1
fi
printf '\033[1m==> dev.sh: cluster=%s (kube context k3s-%s)\033[0m\n' "${CLUSTER}" "${CLUSTER}" >&2
fi
"$ENGINE" run --rm -it \
--network host \
--security-opt label=disable \
@@ -69,8 +115,21 @@ run() {
-v "${HOME}/.ssh:/root/.ssh:ro" \
-v "${HOME}/.config/sops:/root/.config/sops" \
-v "${HOME}/.config/home-k3s/ansible-vault-pass:/root/.config/home-k3s/ansible-vault-pass:ro" \
-e KUBECONFIG="${DEV_KUBECONFIG:-/work/infra/ansible/.cache/dev.kubeconfig}" \
"$IMAGE" bash -lc 'if [ "$#" -gt 0 ]; then exec "$@"; else exec bash -l; fi' dev.sh "$@"
-e CLUSTER="${CLUSTER}" \
-e DEV_KUBECONFIG="${DEV_KUBECONFIG:-}" \
"$IMAGE" bash -lc '
set -e
if [ -n "${DEV_KUBECONFIG:-}" ]; then
export KUBECONFIG="${DEV_KUBECONFIG}"
else
# Regenerated every run: cheap, and it means an updated dev/prod.kubeconfig is picked up
# without anyone remembering to refresh a hand-made merged file. Exits non-zero (set -e)
# if the requested cluster is missing — no silent fallback.
KUBECONFIG="$(/work/infra/devcontainer/kubeconfig.sh "${CLUSTER}")"
export KUBECONFIG
fi
if [ "$#" -gt 0 ]; then exec "$@"; else exec bash -l; fi
' dev.sh "$@"
}
case "${1:-shell}" in
+118
View File
@@ -0,0 +1,118 @@
#!/usr/bin/env bash
#
# infra/devcontainer/kubeconfig.sh — build ONE merged kubeconfig that can address BOTH clusters
# under unambiguous names, and select which of them is current.
#
# kubeconfig.sh <dev|prod> # regenerate the merged file, set its current-context, print its path
#
# WHY THIS EXISTS — the name collision that makes the obvious fix dangerous
# infra/ansible/.cache/dev.kubeconfig and .cache/prod.kubeconfig each name their context, their
# cluster AND their user `default`. k3s writes them that way. So the "obvious" fix —
# KUBECONFIG=dev.kubeconfig:prod.kubeconfig — is actively unsafe: kubectl's merge keeps the FIRST
# occurrence of a duplicated name, so `default` silently resolves to DEV and `--context default`
# can NEVER select prod. An operator (or an agent) would believe they were on prod while issuing
# commands against the live dev cluster.
#
# Therefore we RENAME on merge rather than relying on file order:
# dev.kubeconfig -> cluster/context `k3s-dev`, user `k3s-dev-admin`
# prod.kubeconfig -> cluster/context `k3s-prod`, user `k3s-prod-admin`
# The rename is done with `yq` (structural, YAML-aware). It is NOT done with sed: the cluster and
# the user entry are both `- name: default` at identical indent, so a textual rename corrupts the
# file. The two renamed copies are then merged by kubectl itself, which is safe once the names are
# distinct.
#
# OUTPUT
# ${KUBECONFIG_CACHE_DIR:-<repo>/infra/ansible/.cache}/clusters.kubeconfig, mode 0600.
# That directory is gitignored (infra/ansible/.gitignore: `.cache/`) and `*.kubeconfig` is ignored
# repo-wide. The file embeds cluster-admin client certs — it must NEVER be committed. It is a
# BUILD ARTIFACT: delete it any time, this script rebuilds it.
#
# TOLERANCE / FAILURE
# A cluster whose source kubeconfig is absent is simply left out of the merged file (so a machine
# that only has dev still works). But asking for a cluster whose kubeconfig is absent is a hard
# ERROR — never a silent fallback to the other cluster. Silently landing on the wrong cluster is
# the exact failure this script exists to prevent.
#
# Requires: kubectl, yq (both baked into the dev container image). Normally invoked by dev.sh from
# inside the container; it runs fine anywhere those two binaries exist.
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
CACHE_DIR="${KUBECONFIG_CACHE_DIR:-${REPO_ROOT}/infra/ansible/.cache}"
OUT="${CACHE_DIR}/clusters.kubeconfig"
# cluster name -> source kubeconfig. Add a cluster here and it flows through dev.sh unchanged.
src_for() {
case "$1" in
dev) echo "${DEV_SRC_KUBECONFIG:-${CACHE_DIR}/dev.kubeconfig}" ;;
prod) echo "${PROD_SRC_KUBECONFIG:-${CACHE_DIR}/prod.kubeconfig}" ;;
*) return 1 ;;
esac
}
CLUSTERS=(dev prod)
die() { printf '\033[31mERROR: kubeconfig.sh: %s\033[0m\n' "$*" >&2; exit 1; }
TARGET="${1:-}"
[ -n "${TARGET}" ] || die "usage: kubeconfig.sh <$(IFS='|'; echo "${CLUSTERS[*]}")>"
src_for "${TARGET}" >/dev/null 2>&1 || die "unknown cluster '${TARGET}' (known: ${CLUSTERS[*]})"
for bin in kubectl yq; do
command -v "$bin" >/dev/null 2>&1 || die "'$bin' not found — run this inside infra/devcontainer."
done
# --- FAIL LOUDLY, do not fall back -------------------------------------------------------------
target_src="$(src_for "${TARGET}")"
if [ ! -f "${target_src}" ]; then
die "cluster '${TARGET}' was requested but its kubeconfig is MISSING: ${target_src}
Refusing to continue. Falling back to the other cluster would run your '${TARGET}'
commands against a DIFFERENT LIVE CLUSTER. Fetch the kubeconfig first (see
infra/PHASE1-STATUS.md, 'break-glass cert kubeconfig')."
fi
mkdir -p "${CACHE_DIR}"
tmp="$(mktemp -d)"
chmod 700 "${tmp}"
trap 'rm -rf "${tmp}"' EXIT
parts=()
present=()
for c in "${CLUSTERS[@]}"; do
s="$(src_for "$c")"
[ -f "$s" ] || { printf ' kubeconfig.sh: %-4s kubeconfig absent (%s) — omitted from the merged config\n' "$c" "$s" >&2; continue; }
ctx="k3s-${c}"; usr="k3s-${c}-admin"
# Structural rename. Every kubeconfig k3s emits here has exactly one cluster/user/context, but
# [] indexing renames all of them consistently rather than assuming index 0 exists.
ctx="$ctx" usr="$usr" yq '
.clusters[].name = env(ctx)
| .users[].name = env(usr)
| .contexts[].name = env(ctx)
| .contexts[].context.cluster = env(ctx)
| .contexts[].context.user = env(usr)
| ."current-context" = env(ctx)
' "$s" > "${tmp}/${c}.yaml"
chmod 600 "${tmp}/${c}.yaml"
parts+=("${tmp}/${c}.yaml")
present+=("$c")
done
[ "${#parts[@]}" -gt 0 ] || die "no kubeconfigs found under ${CACHE_DIR}"
# Names are distinct now, so kubectl's merge cannot silently drop one. --raw keeps the embedded
# credential data (a merged config without it authenticates as nobody).
KUBECONFIG="$(IFS=:; echo "${parts[*]}")" kubectl config view --flatten --raw > "${tmp}/merged.yaml"
chmod 600 "${tmp}/merged.yaml"
KUBECONFIG="${tmp}/merged.yaml" kubectl config use-context "k3s-${TARGET}" >/dev/null
install -m 600 "${tmp}/merged.yaml" "${OUT}"
# Post-condition: the requested context must exist and be current, and must NOT be `default`.
cur="$(KUBECONFIG="${OUT}" kubectl config current-context)"
[ "${cur}" = "k3s-${TARGET}" ] || die "internal: current-context is '${cur}', expected 'k3s-${TARGET}'"
if KUBECONFIG="${OUT}" kubectl config get-contexts -o name | grep -qx default; then
die "internal: a context named 'default' survived the rename — refusing to hand back an ambiguous config"
fi
printf ' kubeconfig.sh: merged [%s] -> %s (current-context: %s)\n' "${present[*]}" "${OUT}" "${cur}" >&2
echo "${OUT}"