jakman85andClaude Opus 5
708a05dfc1
infra/devcontainer/kubeconfig.sh was never staged into the delivery-A commit, so the branch built on main carried main's single-cluster tooling and `--context k3s-prod` failed with 'context does not exist' mid-incident. Not Flux-reconciled, so it never affected the cluster — but it breaks prod diagnosis from a checkout of this branch, which is when you need it most. Carries the concurrency fix: the merged kubeconfig holds both contexts and NO current-context, with each invocation getting a per-invocation selector file prepended to KUBECONFIG. Two concurrent dev.sh runs with different CLUSTER values previously overwrote each other's current-context mid-command — reproduced live, a CLUSTER=dev reader returned k3s-prod nodes on 3 of 7 iterations. Co-Authored-By: Claude Opus 5 <[email protected]>
184 lines
10 KiB
Bash
Executable File
184 lines
10 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# infra/devcontainer/kubeconfig.sh — build ONE merged kubeconfig that can address BOTH clusters
|
|
# under unambiguous names, and select which of them is current WITHOUT mutating shared state.
|
|
#
|
|
# kubeconfig.sh <dev|prod> # regenerate the merged file, mint a per-invocation selector,
|
|
# # print the KUBECONFIG *chain* to use (selector:merged)
|
|
#
|
|
# WHY THIS EXISTS — the name collision that makes the obvious fix dangerous
|
|
# infra/ansible/.cache/dev.kubeconfig and .cache/prod.kubeconfig each name their context, their
|
|
# cluster AND their user `default`. k3s writes them that way. So the "obvious" fix —
|
|
# KUBECONFIG=dev.kubeconfig:prod.kubeconfig — is actively unsafe: kubectl's merge keeps the FIRST
|
|
# occurrence of a duplicated name, so `default` silently resolves to DEV and `--context default`
|
|
# can NEVER select prod. An operator (or an agent) would believe they were on prod while issuing
|
|
# commands against the live dev cluster.
|
|
#
|
|
# Therefore we RENAME on merge rather than relying on file order:
|
|
# dev.kubeconfig -> cluster/context `k3s-dev`, user `k3s-dev-admin`
|
|
# prod.kubeconfig -> cluster/context `k3s-prod`, user `k3s-prod-admin`
|
|
# The rename is done with `yq` (structural, YAML-aware). It is NOT done with sed: the cluster and
|
|
# the user entry are both `- name: default` at identical indent, so a textual rename corrupts the
|
|
# file. The two renamed copies are then merged by kubectl itself, which is safe once the names are
|
|
# distinct.
|
|
#
|
|
# WHY THE MERGED FILE HAS NO `current-context` — the concurrency bug this shape fixes
|
|
# This script used to stamp `current-context: k3s-<target>` into the ONE shared merged file. That
|
|
# file lives in the repo's .cache, which every dev.sh container bind-mounts, so it is shared
|
|
# mutable state across concurrent invocations. Two `dev.sh` runs with different CLUSTER values
|
|
# therefore raced: `CLUSTER=prod dev.sh run <script>` would start on prod, a `CLUSTER=dev` run
|
|
# would rewrite `current-context` to k3s-dev mid-script, and the second half of the prod script
|
|
# would silently read the DEV cluster. This was hit live; it was caught only by luck (a PVC name
|
|
# that could not exist on prod). Silent cross-cluster reads — and, worse, cross-cluster WRITES —
|
|
# are the exact failure mode this file exists to prevent.
|
|
#
|
|
# The fix removes the shared mutable field entirely instead of trying to lock it:
|
|
# * the merged file carries BOTH contexts and NO current-context. Read on its own it resolves
|
|
# to nothing and kubectl errors with "current-context is not set" — loud, never a wrong
|
|
# cluster. `--context k3s-dev` / `--context k3s-prod` against it keep working unchanged, and
|
|
# that remains the pattern to prefer.
|
|
# * the default context is supplied by a tiny PER-INVOCATION selector file — literally just
|
|
# `current-context: k3s-<target>`, no credentials — created under a private mktemp dir, and
|
|
# handed back FIRST in a KUBECONFIG chain: "<selector>:<merged>". kubectl merges the chain
|
|
# first-wins, so the selector decides this invocation's default and nothing else's.
|
|
# Nothing in the chain is shared and writable at once, so there is no lock to take and no window
|
|
# to lose. A concurrent regeneration can only rewrite the merged file with byte-identical
|
|
# content, and that write is done by atomic rename so a reader never sees a half-written file.
|
|
# As a bonus, `kubectl config use-context` inside an interactive shell writes to the FIRST file
|
|
# in the chain — the per-invocation selector — so even a manual context switch stays local to
|
|
# that container instead of leaking into everyone else's session.
|
|
#
|
|
# OUTPUT
|
|
# stdout: the KUBECONFIG chain "<selector>:<merged>" — export it verbatim, do not split it.
|
|
# merged: ${KUBECONFIG_CACHE_DIR:-<repo>/infra/ansible/.cache}/clusters.kubeconfig, mode 0600.
|
|
# That directory is gitignored (infra/ansible/.gitignore: `.cache/`) and `*.kubeconfig` is
|
|
# ignored repo-wide. The file embeds cluster-admin client certs — it must NEVER be committed.
|
|
# It is a BUILD ARTIFACT: delete it any time, this script rebuilds it.
|
|
# selector: a ~90-byte file in a 0700 mktemp dir (override the parent with
|
|
# KUBECONFIG_SELECTOR_DIR). It holds no credentials. It is deliberately NOT cleaned up on exit,
|
|
# because the caller needs it for the whole life of its shell; inside dev.sh's `--rm` container
|
|
# it dies with the container. Run standalone on a host it leaves one tiny dir behind.
|
|
#
|
|
# TOLERANCE / FAILURE
|
|
# A cluster whose source kubeconfig is absent is simply left out of the merged file (so a machine
|
|
# that only has dev still works). But asking for a cluster whose kubeconfig is absent is a hard
|
|
# ERROR — never a silent fallback to the other cluster.
|
|
#
|
|
# Requires: kubectl, yq (both baked into the dev container image). Normally invoked by dev.sh from
|
|
# inside the container; it runs fine anywhere those two binaries exist.
|
|
set -euo pipefail
|
|
|
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
|
|
CACHE_DIR="${KUBECONFIG_CACHE_DIR:-${REPO_ROOT}/infra/ansible/.cache}"
|
|
OUT="${CACHE_DIR}/clusters.kubeconfig"
|
|
|
|
# cluster name -> source kubeconfig. Add a cluster here and it flows through dev.sh unchanged.
|
|
src_for() {
|
|
case "$1" in
|
|
dev) echo "${DEV_SRC_KUBECONFIG:-${CACHE_DIR}/dev.kubeconfig}" ;;
|
|
prod) echo "${PROD_SRC_KUBECONFIG:-${CACHE_DIR}/prod.kubeconfig}" ;;
|
|
*) return 1 ;;
|
|
esac
|
|
}
|
|
CLUSTERS=(dev prod)
|
|
|
|
die() { printf '\033[31mERROR: kubeconfig.sh: %s\033[0m\n' "$*" >&2; exit 1; }
|
|
|
|
TARGET="${1:-}"
|
|
[ -n "${TARGET}" ] || die "usage: kubeconfig.sh <$(IFS='|'; echo "${CLUSTERS[*]}")>"
|
|
src_for "${TARGET}" >/dev/null 2>&1 || die "unknown cluster '${TARGET}' (known: ${CLUSTERS[*]})"
|
|
|
|
for bin in kubectl yq; do
|
|
command -v "$bin" >/dev/null 2>&1 || die "'$bin' not found — run this inside infra/devcontainer."
|
|
done
|
|
|
|
# --- FAIL LOUDLY, do not fall back -------------------------------------------------------------
|
|
target_src="$(src_for "${TARGET}")"
|
|
if [ ! -f "${target_src}" ]; then
|
|
die "cluster '${TARGET}' was requested but its kubeconfig is MISSING: ${target_src}
|
|
Refusing to continue. Falling back to the other cluster would run your '${TARGET}'
|
|
commands against a DIFFERENT LIVE CLUSTER. Fetch the kubeconfig first (see
|
|
infra/PHASE1-STATUS.md, 'break-glass cert kubeconfig')."
|
|
fi
|
|
|
|
mkdir -p "${CACHE_DIR}"
|
|
tmp="$(mktemp -d)"
|
|
chmod 700 "${tmp}"
|
|
staged=""
|
|
trap 'rm -rf "${tmp}"; [ -n "${staged}" ] && rm -f "${staged}"; true' EXIT
|
|
|
|
parts=()
|
|
present=()
|
|
for c in "${CLUSTERS[@]}"; do
|
|
s="$(src_for "$c")"
|
|
[ -f "$s" ] || { printf ' kubeconfig.sh: %-4s kubeconfig absent (%s) — omitted from the merged config\n' "$c" "$s" >&2; continue; }
|
|
ctx="k3s-${c}"; usr="k3s-${c}-admin"
|
|
# Structural rename. Every kubeconfig k3s emits here has exactly one cluster/user/context, but
|
|
# [] indexing renames all of them consistently rather than assuming index 0 exists.
|
|
# No `current-context` is set on the parts, and none survives into the merged file: the merged
|
|
# file must stay identical no matter which cluster was asked for, because it is SHARED across
|
|
# concurrent invocations. Selection happens in the per-invocation selector below.
|
|
ctx="$ctx" usr="$usr" yq '
|
|
.clusters[].name = env(ctx)
|
|
| .users[].name = env(usr)
|
|
| .contexts[].name = env(ctx)
|
|
| .contexts[].context.cluster = env(ctx)
|
|
| .contexts[].context.user = env(usr)
|
|
| del(."current-context")
|
|
' "$s" > "${tmp}/${c}.yaml"
|
|
chmod 600 "${tmp}/${c}.yaml"
|
|
parts+=("${tmp}/${c}.yaml")
|
|
present+=("$c")
|
|
done
|
|
|
|
[ "${#parts[@]}" -gt 0 ] || die "no kubeconfigs found under ${CACHE_DIR}"
|
|
|
|
# Names are distinct now, so kubectl's merge cannot silently drop one. --raw keeps the embedded
|
|
# credential data (a merged config without it authenticates as nobody). `view` re-emits an empty
|
|
# `current-context: ""` even when no part had one; drop the key so the file is unambiguous.
|
|
KUBECONFIG="$(IFS=:; echo "${parts[*]}")" kubectl config view --flatten --raw > "${tmp}/merged.yaml"
|
|
yq -i 'del(."current-context")' "${tmp}/merged.yaml"
|
|
chmod 600 "${tmp}/merged.yaml"
|
|
|
|
# Publish the merged file by ATOMIC RENAME, not by writing in place. Concurrent invocations produce
|
|
# byte-identical content, but `install`/`>` truncate first, so an unlucky concurrent reader could
|
|
# otherwise load a half-written kubeconfig. rename(2) within one filesystem swaps it whole.
|
|
staged="$(mktemp "${CACHE_DIR}/.clusters.kubeconfig.XXXXXX")"
|
|
cat "${tmp}/merged.yaml" > "${staged}"
|
|
chmod 600 "${staged}"
|
|
mv -f "${staged}" "${OUT}"
|
|
|
|
# Post-condition on the SHARED file: both renamed contexts present, nothing called `default`, and
|
|
# no current-context to race over.
|
|
if KUBECONFIG="${OUT}" kubectl config get-contexts -o name | grep -qx default; then
|
|
die "internal: a context named 'default' survived the rename — refusing to hand back an ambiguous config"
|
|
fi
|
|
KUBECONFIG="${OUT}" kubectl config get-contexts -o name | grep -qx "k3s-${TARGET}" \
|
|
|| die "internal: context 'k3s-${TARGET}' is missing from ${OUT}"
|
|
if KUBECONFIG="${OUT}" kubectl config current-context >/dev/null 2>&1; then
|
|
die "internal: ${OUT} carries a current-context — that field is shared mutable state and must stay unset"
|
|
fi
|
|
|
|
# --- per-invocation selector: this invocation's default context, private to this invocation ------
|
|
# `kubectl config set` (a config subcommand, never sed) writes the single scalar. It does not
|
|
# validate the name, which is why the context's existence was asserted against the merged file
|
|
# above.
|
|
sel_dir="$(mktemp -d "${KUBECONFIG_SELECTOR_DIR:-${TMPDIR:-/tmp}}/kubeconfig-sel.XXXXXX")"
|
|
chmod 700 "${sel_dir}"
|
|
SEL="${sel_dir}/selector.kubeconfig"
|
|
printf 'apiVersion: v1\nkind: Config\n' > "${SEL}"
|
|
chmod 600 "${SEL}"
|
|
KUBECONFIG="${SEL}" kubectl config set current-context "k3s-${TARGET}" >/dev/null
|
|
|
|
CHAIN="${SEL}:${OUT}"
|
|
|
|
# Post-condition on the CHAIN — the thing the caller will actually export. This is the end-to-end
|
|
# assertion that this invocation defaults to the cluster that was asked for.
|
|
cur="$(KUBECONFIG="${CHAIN}" kubectl config current-context)"
|
|
[ "${cur}" = "k3s-${TARGET}" ] || die "internal: current-context is '${cur}', expected 'k3s-${TARGET}'"
|
|
|
|
printf ' kubeconfig.sh: merged [%s] -> %s (no current-context; shared)\n' "${present[*]}" "${OUT}" >&2
|
|
printf ' kubeconfig.sh: this invocation defaults to %s via %s\n' "${cur}" "${SEL}" >&2
|
|
echo "${CHAIN}"
|