jakman85andClaude Opus 5
21e111b08c
Operator report: dev.sh only reached the dev cluster. Line 72 hardcoded
`KUBECONFIG=${DEV_KUBECONFIG:-…/dev.kubeconfig}`, and the only escape was an
undocumented override whose NAME misleads — you set `DEV_KUBECONFIG` to a *prod*
path. infra/PHASE1-STATUS.md had recorded this as a "Known gap" for months.
WHY THIS WAS NOT A ONE-LINER. Both kubeconfigs name their context, cluster AND
user `default`. kubectl's merge keeps the FIRST occurrence of a duplicated name,
so the obvious `KUBECONFIG=dev.kubeconfig:prod.kubeconfig` resolves `default` to
DEV and `--context default` can never select prod — an operator or agent would
believe they were on prod and be issuing commands against the LIVE dev cluster
that runs the operator's platform. Same collision class as ~/.kube/config an hour
earlier, and worse here because agents run in this container.
Interface: `CLUSTER=dev|prod` as an env prefix, default **dev** (back-compat: every
existing invocation, CI job and habit keeps meaning dev).
infra/devcontainer/dev.sh run kubectl get nodes # dev
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # prod
Its one weakness — `CLUSTER` can be inherited from an exported shell var — is
covered by a banner printed on EVERY invocation before the container starts:
`==> dev.sh: cluster=prod (kube context k3s-prod)`.
New infra/devcontainer/kubeconfig.sh generates a merged config with DISTINCT names
(contexts k3s-dev/k3s-prod, users k3s-*-admin), renaming structurally with `yq`
on .clusters[].name / .users[].name / .contexts[].{name,context.cluster,
context.user} before letting `kubectl config view --flatten` merge them. NOT sed —
sed renaming corrupted a config earlier today because cluster and user entries are
both `- name: default` at identical indent. Output is 0600 in the gitignored
.cache/, regenerated every run, and post-conditions assert the current context is
`k3s-<target>` and that NO context named `default` survived.
FAILS LOUDLY AT BOTH LAYERS, and never falls back to the other cluster — that
fallback is exactly how a prod command ends up hitting dev. Host-side refusal if
the requested cluster's kubeconfig is missing; in-container refusal likewise;
unknown CLUSTER exits 2. A missing source for the cluster you did NOT ask for is
tolerated (merges the other one only).
VERIFIED BY ME, independently of the agent — the two BASE_DOMAIN reads are the
unambiguous proof of which cluster was reached, since a node list is not:
default (no selector) -> home.keithjackson.dev (DEV, unchanged)
CLUSTER=prod -> prod.home.keithjackson.dev (PROD)
contexts -> k3s-dev, k3s-prod; NO `default` context survives
exit codes -> invalid CLUSTER=2, valid dev/prod=0
standup.sh -> `--kubeconfig` still wins over the new default
(preflight against prod: target confirmed 192.168.2.9)
credentials -> dev.kubeconfig and prod.kubeconfig byte-intact and each
still reaching 3 nodes after the agent's move-aside test
git -> no kubeconfig committable; clusters.kubeconfig ignored
Full CI gate PASS.
NOT VERIFIED: `dev.sh build` and the interactive shell path (existing image reused,
no TTY available). The shell path shares run()'s code; only the TTY differs.
DEV_KUBECONFIG still wins verbatim, now with a deprecation warning. All existing
mounts are untouched: SOPS age keyring rw, ~/.ssh ro, ansible-vault-pass ro, spec
repo ro + SPEC_REPO, --network host, label=disable.
Closes the PHASE1-STATUS "Known gap".
Refs R-ENV-02, R-TEST-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
119 lines
5.6 KiB
Bash
Executable File
119 lines
5.6 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# infra/devcontainer/kubeconfig.sh — build ONE merged kubeconfig that can address BOTH clusters
|
|
# under unambiguous names, and select which of them is current.
|
|
#
|
|
# kubeconfig.sh <dev|prod> # regenerate the merged file, set its current-context, print its path
|
|
#
|
|
# WHY THIS EXISTS — the name collision that makes the obvious fix dangerous
|
|
# infra/ansible/.cache/dev.kubeconfig and .cache/prod.kubeconfig each name their context, their
|
|
# cluster AND their user `default`. k3s writes them that way. So the "obvious" fix —
|
|
# KUBECONFIG=dev.kubeconfig:prod.kubeconfig — is actively unsafe: kubectl's merge keeps the FIRST
|
|
# occurrence of a duplicated name, so `default` silently resolves to DEV and `--context default`
|
|
# can NEVER select prod. An operator (or an agent) would believe they were on prod while issuing
|
|
# commands against the live dev cluster.
|
|
#
|
|
# Therefore we RENAME on merge rather than relying on file order:
|
|
# dev.kubeconfig -> cluster/context `k3s-dev`, user `k3s-dev-admin`
|
|
# prod.kubeconfig -> cluster/context `k3s-prod`, user `k3s-prod-admin`
|
|
# The rename is done with `yq` (structural, YAML-aware). It is NOT done with sed: the cluster and
|
|
# the user entry are both `- name: default` at identical indent, so a textual rename corrupts the
|
|
# file. The two renamed copies are then merged by kubectl itself, which is safe once the names are
|
|
# distinct.
|
|
#
|
|
# OUTPUT
|
|
# ${KUBECONFIG_CACHE_DIR:-<repo>/infra/ansible/.cache}/clusters.kubeconfig, mode 0600.
|
|
# That directory is gitignored (infra/ansible/.gitignore: `.cache/`) and `*.kubeconfig` is ignored
|
|
# repo-wide. The file embeds cluster-admin client certs — it must NEVER be committed. It is a
|
|
# BUILD ARTIFACT: delete it any time, this script rebuilds it.
|
|
#
|
|
# TOLERANCE / FAILURE
|
|
# A cluster whose source kubeconfig is absent is simply left out of the merged file (so a machine
|
|
# that only has dev still works). But asking for a cluster whose kubeconfig is absent is a hard
|
|
# ERROR — never a silent fallback to the other cluster. Silently landing on the wrong cluster is
|
|
# the exact failure this script exists to prevent.
|
|
#
|
|
# Requires: kubectl, yq (both baked into the dev container image). Normally invoked by dev.sh from
|
|
# inside the container; it runs fine anywhere those two binaries exist.
|
|
set -euo pipefail
|
|
|
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
|
|
CACHE_DIR="${KUBECONFIG_CACHE_DIR:-${REPO_ROOT}/infra/ansible/.cache}"
|
|
OUT="${CACHE_DIR}/clusters.kubeconfig"
|
|
|
|
# cluster name -> source kubeconfig. Add a cluster here and it flows through dev.sh unchanged.
|
|
src_for() {
|
|
case "$1" in
|
|
dev) echo "${DEV_SRC_KUBECONFIG:-${CACHE_DIR}/dev.kubeconfig}" ;;
|
|
prod) echo "${PROD_SRC_KUBECONFIG:-${CACHE_DIR}/prod.kubeconfig}" ;;
|
|
*) return 1 ;;
|
|
esac
|
|
}
|
|
CLUSTERS=(dev prod)
|
|
|
|
die() { printf '\033[31mERROR: kubeconfig.sh: %s\033[0m\n' "$*" >&2; exit 1; }
|
|
|
|
TARGET="${1:-}"
|
|
[ -n "${TARGET}" ] || die "usage: kubeconfig.sh <$(IFS='|'; echo "${CLUSTERS[*]}")>"
|
|
src_for "${TARGET}" >/dev/null 2>&1 || die "unknown cluster '${TARGET}' (known: ${CLUSTERS[*]})"
|
|
|
|
for bin in kubectl yq; do
|
|
command -v "$bin" >/dev/null 2>&1 || die "'$bin' not found — run this inside infra/devcontainer."
|
|
done
|
|
|
|
# --- FAIL LOUDLY, do not fall back -------------------------------------------------------------
|
|
target_src="$(src_for "${TARGET}")"
|
|
if [ ! -f "${target_src}" ]; then
|
|
die "cluster '${TARGET}' was requested but its kubeconfig is MISSING: ${target_src}
|
|
Refusing to continue. Falling back to the other cluster would run your '${TARGET}'
|
|
commands against a DIFFERENT LIVE CLUSTER. Fetch the kubeconfig first (see
|
|
infra/PHASE1-STATUS.md, 'break-glass cert kubeconfig')."
|
|
fi
|
|
|
|
mkdir -p "${CACHE_DIR}"
|
|
tmp="$(mktemp -d)"
|
|
chmod 700 "${tmp}"
|
|
trap 'rm -rf "${tmp}"' EXIT
|
|
|
|
parts=()
|
|
present=()
|
|
for c in "${CLUSTERS[@]}"; do
|
|
s="$(src_for "$c")"
|
|
[ -f "$s" ] || { printf ' kubeconfig.sh: %-4s kubeconfig absent (%s) — omitted from the merged config\n' "$c" "$s" >&2; continue; }
|
|
ctx="k3s-${c}"; usr="k3s-${c}-admin"
|
|
# Structural rename. Every kubeconfig k3s emits here has exactly one cluster/user/context, but
|
|
# [] indexing renames all of them consistently rather than assuming index 0 exists.
|
|
ctx="$ctx" usr="$usr" yq '
|
|
.clusters[].name = env(ctx)
|
|
| .users[].name = env(usr)
|
|
| .contexts[].name = env(ctx)
|
|
| .contexts[].context.cluster = env(ctx)
|
|
| .contexts[].context.user = env(usr)
|
|
| ."current-context" = env(ctx)
|
|
' "$s" > "${tmp}/${c}.yaml"
|
|
chmod 600 "${tmp}/${c}.yaml"
|
|
parts+=("${tmp}/${c}.yaml")
|
|
present+=("$c")
|
|
done
|
|
|
|
[ "${#parts[@]}" -gt 0 ] || die "no kubeconfigs found under ${CACHE_DIR}"
|
|
|
|
# Names are distinct now, so kubectl's merge cannot silently drop one. --raw keeps the embedded
|
|
# credential data (a merged config without it authenticates as nobody).
|
|
KUBECONFIG="$(IFS=:; echo "${parts[*]}")" kubectl config view --flatten --raw > "${tmp}/merged.yaml"
|
|
chmod 600 "${tmp}/merged.yaml"
|
|
KUBECONFIG="${tmp}/merged.yaml" kubectl config use-context "k3s-${TARGET}" >/dev/null
|
|
|
|
install -m 600 "${tmp}/merged.yaml" "${OUT}"
|
|
|
|
# Post-condition: the requested context must exist and be current, and must NOT be `default`.
|
|
cur="$(KUBECONFIG="${OUT}" kubectl config current-context)"
|
|
[ "${cur}" = "k3s-${TARGET}" ] || die "internal: current-context is '${cur}', expected 'k3s-${TARGET}'"
|
|
if KUBECONFIG="${OUT}" kubectl config get-contexts -o name | grep -qx default; then
|
|
die "internal: a context named 'default' survived the rename — refusing to hand back an ambiguous config"
|
|
fi
|
|
|
|
printf ' kubeconfig.sh: merged [%s] -> %s (current-context: %s)\n' "${present[*]}" "${OUT}" "${cur}" >&2
|
|
echo "${OUT}"
|