feat(devcontainer): dev.sh can target prod — and cannot silently target the wrong cluster
Operator report: dev.sh only reached the dev cluster. Line 72 hardcoded
`KUBECONFIG=${DEV_KUBECONFIG:-…/dev.kubeconfig}`, and the only escape was an
undocumented override whose NAME misleads — you set `DEV_KUBECONFIG` to a *prod*
path. infra/PHASE1-STATUS.md had recorded this as a "Known gap" for months.
WHY THIS WAS NOT A ONE-LINER. Both kubeconfigs name their context, cluster AND
user `default`. kubectl's merge keeps the FIRST occurrence of a duplicated name,
so the obvious `KUBECONFIG=dev.kubeconfig:prod.kubeconfig` resolves `default` to
DEV and `--context default` can never select prod — an operator or agent would
believe they were on prod and be issuing commands against the LIVE dev cluster
that runs the operator's platform. Same collision class as ~/.kube/config an hour
earlier, and worse here because agents run in this container.
Interface: `CLUSTER=dev|prod` as an env prefix, default **dev** (back-compat: every
existing invocation, CI job and habit keeps meaning dev).
infra/devcontainer/dev.sh run kubectl get nodes # dev
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # prod
Its one weakness — `CLUSTER` can be inherited from an exported shell var — is
covered by a banner printed on EVERY invocation before the container starts:
`==> dev.sh: cluster=prod (kube context k3s-prod)`.
New infra/devcontainer/kubeconfig.sh generates a merged config with DISTINCT names
(contexts k3s-dev/k3s-prod, users k3s-*-admin), renaming structurally with `yq`
on .clusters[].name / .users[].name / .contexts[].{name,context.cluster,
context.user} before letting `kubectl config view --flatten` merge them. NOT sed —
sed renaming corrupted a config earlier today because cluster and user entries are
both `- name: default` at identical indent. Output is 0600 in the gitignored
.cache/, regenerated every run, and post-conditions assert the current context is
`k3s-<target>` and that NO context named `default` survived.
FAILS LOUDLY AT BOTH LAYERS, and never falls back to the other cluster — that
fallback is exactly how a prod command ends up hitting dev. Host-side refusal if
the requested cluster's kubeconfig is missing; in-container refusal likewise;
unknown CLUSTER exits 2. A missing source for the cluster you did NOT ask for is
tolerated (merges the other one only).
VERIFIED BY ME, independently of the agent — the two BASE_DOMAIN reads are the
unambiguous proof of which cluster was reached, since a node list is not:
default (no selector) -> home.keithjackson.dev (DEV, unchanged)
CLUSTER=prod -> prod.home.keithjackson.dev (PROD)
contexts -> k3s-dev, k3s-prod; NO `default` context survives
exit codes -> invalid CLUSTER=2, valid dev/prod=0
standup.sh -> `--kubeconfig` still wins over the new default
(preflight against prod: target confirmed 192.168.2.9)
credentials -> dev.kubeconfig and prod.kubeconfig byte-intact and each
still reaching 3 nodes after the agent's move-aside test
git -> no kubeconfig committable; clusters.kubeconfig ignored
Full CI gate PASS.
NOT VERIFIED: `dev.sh build` and the interactive shell path (existing image reused,
no TTY available). The shell path shares run()'s code; only the TTY differs.
DEV_KUBECONFIG still wins verbatim, now with a deprecation warning. All existing
mounts are untouched: SOPS age keyring rw, ~/.ssh ro, ansible-vault-pass ro, spec
repo ro + SPEC_REPO, --network host, label=disable.
Closes the PHASE1-STATUS "Known gap".
Refs R-ENV-02, R-TEST-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
577c62136b
commit
21e111b08c
5 files changed
+268
-17
No files matched your search
@@ -11,7 +11,24 @@
|
||||
# ./dev.sh build # build the image only
|
||||
# ./dev.sh # build (if needed) + open a shell
|
||||
# ./dev.sh run tofu -version # run one command in the toolbox
|
||||
# ./dev.sh run kubectl get nodes # talks to the Proxmox k3s cluster directly (no tunnel)
|
||||
# ./dev.sh run kubectl get nodes # talks to the DEV k3s cluster directly (no tunnel)
|
||||
#
|
||||
# CHOOSING A CLUSTER — `CLUSTER=<dev|prod>`, defaulting to dev:
|
||||
#
|
||||
# ./dev.sh run kubectl get nodes # dev (the default; unchanged behaviour)
|
||||
# CLUSTER=prod ./dev.sh run kubectl get nodes # prod
|
||||
# CLUSTER=prod ./dev.sh # interactive shell pointed at prod
|
||||
#
|
||||
# The selector is an env-var PREFIX so the target is visible on the same line as the command, and
|
||||
# every invocation prints a banner naming the cluster it is about to talk to. Inside the container
|
||||
# KUBECONFIG is a MERGED config holding both clusters under unambiguous names — contexts `k3s-dev`
|
||||
# and `k3s-prod`, never `default` — with `CLUSTER`'s context selected as current. So `kubectl
|
||||
# config current-context` always answers the "which cluster am I on?" question honestly, and
|
||||
# `--context k3s-prod` actually works. See infra/devcontainer/kubeconfig.sh for why the naive
|
||||
# `KUBECONFIG=dev:prod` merge is dangerous (both source files name everything `default`).
|
||||
#
|
||||
# Requesting a cluster whose kubeconfig is missing is a hard ERROR — it never falls back to the
|
||||
# other cluster.
|
||||
set -euo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
|
||||
@@ -24,6 +41,15 @@ source "${HERE}/versions.env"
|
||||
# the pfSense NAT (192.168.2.50:6443 -> kube-vip VIP 192.168.1.49:6443). No SSH tunnel is
|
||||
# opened. infra/ansible/.cache/dev.kubeconfig targets https://192.168.2.50:6443 with
|
||||
# tls-server-name=192.168.1.49 so TLS validates against the VIP's SAN through the NAT.
|
||||
# Prod is reached the same way via infra/ansible/.cache/prod.kubeconfig (https://192.168.2.9:6443).
|
||||
|
||||
# ------------------------------------------------------------------ cluster selection
|
||||
CLUSTER="${CLUSTER:-dev}"
|
||||
case "${CLUSTER}" in
|
||||
dev|prod) ;;
|
||||
*) printf '\033[31mERROR: CLUSTER=%s is not a known cluster (dev|prod).\033[0m\n' "${CLUSTER}" >&2; exit 2 ;;
|
||||
esac
|
||||
CLUSTER_KUBECONFIG="${REPO_ROOT}/infra/ansible/.cache/${CLUSTER}.kubeconfig"
|
||||
|
||||
build() {
|
||||
"$ENGINE" build \
|
||||
@@ -61,6 +87,26 @@ run() {
|
||||
if [ -n "${spec_host}" ] && [ -d "${spec_host}/.git" ]; then
|
||||
spec_mount=(-v "${spec_host}:/home-network-k3s:ro" -e SPEC_REPO=/home-network-k3s)
|
||||
fi
|
||||
|
||||
# Which cluster, said out loud, before anything runs. CLUSTER can be inherited from an exported
|
||||
# shell variable, so the banner is what stops "I thought I was on dev".
|
||||
if [ -n "${DEV_KUBECONFIG:-}" ]; then
|
||||
# SUPERSEDED but still honoured: DEV_KUBECONFIG used to be the only way to reach a cluster other
|
||||
# than the (then non-existent) default dev path. It wins over CLUSTER so nothing that depends on
|
||||
# it breaks, but it is opaque about its target — prefer CLUSTER=dev|prod.
|
||||
printf '\033[33m==> dev.sh: KUBECONFIG pinned by DEV_KUBECONFIG=%s (CLUSTER=%s ignored; DEV_KUBECONFIG is superseded — use CLUSTER=dev|prod)\033[0m\n' \
|
||||
"${DEV_KUBECONFIG}" "${CLUSTER}" >&2
|
||||
else
|
||||
# Fail here, on the host, before a container is even started: a missing kubeconfig must never
|
||||
# degrade into "whatever config we could find", because that is how a prod command lands on dev.
|
||||
if [ ! -f "${CLUSTER_KUBECONFIG}" ]; then
|
||||
printf '\033[31mERROR: dev.sh: CLUSTER=%s but its kubeconfig is MISSING: %s\n Refusing to run. dev.sh does NOT fall back to the other cluster.\033[0m\n' \
|
||||
"${CLUSTER}" "${CLUSTER_KUBECONFIG}" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf '\033[1m==> dev.sh: cluster=%s (kube context k3s-%s)\033[0m\n' "${CLUSTER}" "${CLUSTER}" >&2
|
||||
fi
|
||||
|
||||
"$ENGINE" run --rm -it \
|
||||
--network host \
|
||||
--security-opt label=disable \
|
||||
@@ -69,8 +115,21 @@ run() {
|
||||
-v "${HOME}/.ssh:/root/.ssh:ro" \
|
||||
-v "${HOME}/.config/sops:/root/.config/sops" \
|
||||
-v "${HOME}/.config/home-k3s/ansible-vault-pass:/root/.config/home-k3s/ansible-vault-pass:ro" \
|
||||
-e KUBECONFIG="${DEV_KUBECONFIG:-/work/infra/ansible/.cache/dev.kubeconfig}" \
|
||||
"$IMAGE" bash -lc 'if [ "$#" -gt 0 ]; then exec "$@"; else exec bash -l; fi' dev.sh "$@"
|
||||
-e CLUSTER="${CLUSTER}" \
|
||||
-e DEV_KUBECONFIG="${DEV_KUBECONFIG:-}" \
|
||||
"$IMAGE" bash -lc '
|
||||
set -e
|
||||
if [ -n "${DEV_KUBECONFIG:-}" ]; then
|
||||
export KUBECONFIG="${DEV_KUBECONFIG}"
|
||||
else
|
||||
# Regenerated every run: cheap, and it means an updated dev/prod.kubeconfig is picked up
|
||||
# without anyone remembering to refresh a hand-made merged file. Exits non-zero (set -e)
|
||||
# if the requested cluster is missing — no silent fallback.
|
||||
KUBECONFIG="$(/work/infra/devcontainer/kubeconfig.sh "${CLUSTER}")"
|
||||
export KUBECONFIG
|
||||
fi
|
||||
if [ "$#" -gt 0 ]; then exec "$@"; else exec bash -l; fi
|
||||
' dev.sh "$@"
|
||||
}
|
||||
|
||||
case "${1:-shell}" in
|
||||
|
||||
Reference in new issue
Block a user