diff --git a/README.md b/README.md index 7fd74c1..0d4383a 100644 --- a/README.md +++ b/README.md @@ -69,6 +69,17 @@ infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible) ``` +`kubectl`/`flux`/`helm` in there talk to the **dev** cluster by default. Prefix `CLUSTER=prod` to +target prod instead: + +```bash +infra/devcontainer/dev.sh run kubectl get nodes # dev (default) +CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # prod +``` + +Both clusters are always present in the container's `KUBECONFIG` as the contexts `k3s-dev` and +`k3s-prod`; `CLUSTER` picks which one is current, and each run prints the cluster it is targeting. + The container mounts the repo at `/work`, your age keyring, and the sibling spec repo read-only so the spec-immutability check can actually run. See [`infra/devcontainer/README.md`](infra/devcontainer/README.md). diff --git a/infra/PHASE1-STATUS.md b/infra/PHASE1-STATUS.md index e3b7a79..9218309 100644 --- a/infra/PHASE1-STATUS.md +++ b/infra/PHASE1-STATUS.md @@ -84,6 +84,9 @@ toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image infra/devcontainer/dev.sh build # once infra/devcontainer/dev.sh run make -C ci gate # exactly what CI runs infra/devcontainer/dev.sh # interactive shell at /work + +infra/devcontainer/dev.sh run kubectl get nodes # DEV (the default) +CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # PROD ``` Unlike the libvirt era, **tofu and ansible now run fine inside the container** — the @@ -95,10 +98,21 @@ which is what broke the old `dmacvicar/libvirt` provider in a container. file, and the sibling spec repo read-only at `/home-network-k3s` so the spec-immutability check runs instead of silently skipping. -> **Known gap.** `dev.sh` defaults `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`, -> a libvirt-era path that no longer exists, and does not mount `~/.kube`. In-container `kubectl` -> therefore needs an explicit `DEV_KUBECONFIG=` — the OIDC -> context cannot work there anyway (no browser). Not yet fixed. +> **Closed (was a known gap): `dev.sh` can now target either cluster.** It used to hard-wire +> `KUBECONFIG` to `dev.kubeconfig`, leaving an undocumented — and misleadingly named — +> `DEV_KUBECONFIG=` as the only route to prod. Select with **`CLUSTER=dev|prod`**; +> **dev remains the default**, so every existing invocation means what it always meant. Inside the +> container `KUBECONFIG` is a merged config carrying both clusters as contexts **`k3s-dev`** and +> **`k3s-prod`** (never `default`), regenerated each run by +> [`infra/devcontainer/kubeconfig.sh`](devcontainer/kubeconfig.sh) into the gitignored +> `infra/ansible/.cache/clusters.kubeconfig`. Naming matters: k3s writes *both* source kubeconfigs +> with context, cluster and user all called `default`, and kubectl's merge keeps the first +> duplicate — so a plain `KUBECONFIG=dev:prod` would silently resolve `default` to **dev** while you +> believed you were on prod. Requesting a cluster whose kubeconfig is absent is a hard error, never +> a fallback. `DEV_KUBECONFIG` still works and still wins, but is **superseded** — it is opaque +> about its target and yields an ambiguous `default` context. `~/.kube` is still not mounted and the +> OIDC context still cannot work in a container (no browser); use the break-glass cert kubeconfigs +> above. Details: [`devcontainer/README.md`](devcontainer/README.md). ## Off-repo state you must not lose diff --git a/infra/devcontainer/README.md b/infra/devcontainer/README.md index 189f4df..db96968 100644 --- a/infra/devcontainer/README.md +++ b/infra/devcontainer/README.md @@ -33,6 +33,57 @@ cd infra/devcontainer ``` `ENGINE=docker ./dev.sh` to use docker instead of rootless podman. +## Choosing a cluster — `CLUSTER=dev|prod` + +```bash +./dev.sh run kubectl get nodes # DEV — the default, unchanged +CLUSTER=prod ./dev.sh run kubectl get nodes # PROD +CLUSTER=prod ./dev.sh # interactive shell pointed at prod +``` + +An env-var **prefix**, so the target sits on the same line as the command. Every invocation prints +a banner naming the cluster before the container starts, because `CLUSTER` can also be inherited +from an exported shell variable. + +Inside the container `KUBECONFIG` points at a **merged** config holding both clusters under +unambiguous names, with `CLUSTER`'s context current: + +``` +CURRENT NAME CLUSTER AUTHINFO +* k3s-dev k3s-dev k3s-dev-admin + k3s-prod k3s-prod k3s-prod-admin +``` + +So `kubectl config current-context` answers "which cluster am I on?" honestly, and +`kubectl --context k3s-prod …` works from a dev-defaulted shell. + +**Why a merge script and not `KUBECONFIG=dev.kubeconfig:prod.kubeconfig`.** k3s names the context, +the cluster *and* the user in **both** files `default`. kubectl's merge keeps the **first** +occurrence of a duplicated name, so a colon-joined `KUBECONFIG` silently resolves `default` to +**dev** and `--context default` can never reach prod — you would believe you were on prod while +issuing commands against the live dev cluster. `kubeconfig.sh` therefore **renames on merge** +(`k3s-dev` / `k3s-prod`, `*-admin` users) with `yq`, structurally — not with `sed`, which corrupts +these files because the cluster and the user entry are both `- name: default` at identical indent. + +- Generator: [`kubeconfig.sh`](kubeconfig.sh). Output: `infra/ansible/.cache/clusters.kubeconfig`, + mode 0600, **regenerated on every `dev.sh run`** — a build artifact, never hand-edited, never + committed (`.cache/` and `*.kubeconfig` are both gitignored; it embeds cluster-admin certs). +- A cluster whose source kubeconfig is missing is simply omitted from the merge, so a machine with + only `dev.kubeconfig` still works. +- **Asking for a cluster whose kubeconfig is absent is a hard error.** `dev.sh` refuses on the host + before starting a container and never falls back to the other cluster — that fallback is exactly + how a prod command lands on dev. +- `bootstrap/standup.sh` is unaffected: it exports its own `KUBECONFIG` from the mandatory + `--kubeconfig` flag, which overrides what `dev.sh` sets. Its `--expect-server` guard remains the + thing that makes it safe, not `CLUSTER`. + +### `DEV_KUBECONFIG` — superseded, still honoured + +`DEV_KUBECONFIG= ./dev.sh run …` still pins `KUBECONFIG` verbatim and takes precedence over +`CLUSTER` (nothing depending on it breaks), but it is **deprecated**: the name says "dev" while the +established workaround set it to a *prod* path, and it produces a single-cluster config whose +context is the ambiguous `default`. Use `CLUSTER=dev|prod`. Invocations that use it print a warning. + ## What `dev.sh` mounts, and why each one matters - `--network host` — Ansible SSHes to the nodes through the pfSense NAT (`192.168.2.102:225x`) and @@ -61,18 +112,16 @@ SSH, which `--network host` provides. So: run everything in here. The host needs only podman (or docker). -## kubectl inside the container — known gap +## kubectl inside the container — was a known gap, now fixed -`dev.sh` sets `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`, a **libvirt-era path -that no longer exists**, and does not mount `~/.kube`. Override it explicitly: +`dev.sh` used to hard-wire `KUBECONFIG` to `dev.kubeconfig` with no supported way to reach prod, and +it does not mount `~/.kube`. It now selects between `infra/ansible/.cache/dev.kubeconfig` and +`prod.kubeconfig` via `CLUSTER=dev|prod` (see above) and merges both under distinct context names. -```bash -DEV_KUBECONFIG=/work/path/to/break-glass.kubeconfig ./dev.sh run kubectl get nodes -``` - -Use a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login needs a browser, -which a container shell does not have. See [`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both -access paths. (Not yet fixed in `dev.sh`.) +The container still needs a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login +needs a browser, which a container shell does not have. See +[`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both access paths and for how to regenerate these +two files from `/etc/rancher/k3s/k3s.yaml`. ## Notes diff --git a/infra/devcontainer/dev.sh b/infra/devcontainer/dev.sh index 19652fb..11c1a1e 100755 --- a/infra/devcontainer/dev.sh +++ b/infra/devcontainer/dev.sh @@ -11,7 +11,24 @@ # ./dev.sh build # build the image only # ./dev.sh # build (if needed) + open a shell # ./dev.sh run tofu -version # run one command in the toolbox -# ./dev.sh run kubectl get nodes # talks to the Proxmox k3s cluster directly (no tunnel) +# ./dev.sh run kubectl get nodes # talks to the DEV k3s cluster directly (no tunnel) +# +# CHOOSING A CLUSTER — `CLUSTER=`, defaulting to dev: +# +# ./dev.sh run kubectl get nodes # dev (the default; unchanged behaviour) +# CLUSTER=prod ./dev.sh run kubectl get nodes # prod +# CLUSTER=prod ./dev.sh # interactive shell pointed at prod +# +# The selector is an env-var PREFIX so the target is visible on the same line as the command, and +# every invocation prints a banner naming the cluster it is about to talk to. Inside the container +# KUBECONFIG is a MERGED config holding both clusters under unambiguous names — contexts `k3s-dev` +# and `k3s-prod`, never `default` — with `CLUSTER`'s context selected as current. So `kubectl +# config current-context` always answers the "which cluster am I on?" question honestly, and +# `--context k3s-prod` actually works. See infra/devcontainer/kubeconfig.sh for why the naive +# `KUBECONFIG=dev:prod` merge is dangerous (both source files name everything `default`). +# +# Requesting a cluster whose kubeconfig is missing is a hard ERROR — it never falls back to the +# other cluster. set -euo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_ROOT="$(cd "${HERE}/../.." && pwd)" @@ -24,6 +41,15 @@ source "${HERE}/versions.env" # the pfSense NAT (192.168.2.50:6443 -> kube-vip VIP 192.168.1.49:6443). No SSH tunnel is # opened. infra/ansible/.cache/dev.kubeconfig targets https://192.168.2.50:6443 with # tls-server-name=192.168.1.49 so TLS validates against the VIP's SAN through the NAT. +# Prod is reached the same way via infra/ansible/.cache/prod.kubeconfig (https://192.168.2.9:6443). + +# ------------------------------------------------------------------ cluster selection +CLUSTER="${CLUSTER:-dev}" +case "${CLUSTER}" in + dev|prod) ;; + *) printf '\033[31mERROR: CLUSTER=%s is not a known cluster (dev|prod).\033[0m\n' "${CLUSTER}" >&2; exit 2 ;; +esac +CLUSTER_KUBECONFIG="${REPO_ROOT}/infra/ansible/.cache/${CLUSTER}.kubeconfig" build() { "$ENGINE" build \ @@ -61,6 +87,26 @@ run() { if [ -n "${spec_host}" ] && [ -d "${spec_host}/.git" ]; then spec_mount=(-v "${spec_host}:/home-network-k3s:ro" -e SPEC_REPO=/home-network-k3s) fi + + # Which cluster, said out loud, before anything runs. CLUSTER can be inherited from an exported + # shell variable, so the banner is what stops "I thought I was on dev". + if [ -n "${DEV_KUBECONFIG:-}" ]; then + # SUPERSEDED but still honoured: DEV_KUBECONFIG used to be the only way to reach a cluster other + # than the (then non-existent) default dev path. It wins over CLUSTER so nothing that depends on + # it breaks, but it is opaque about its target — prefer CLUSTER=dev|prod. + printf '\033[33m==> dev.sh: KUBECONFIG pinned by DEV_KUBECONFIG=%s (CLUSTER=%s ignored; DEV_KUBECONFIG is superseded — use CLUSTER=dev|prod)\033[0m\n' \ + "${DEV_KUBECONFIG}" "${CLUSTER}" >&2 + else + # Fail here, on the host, before a container is even started: a missing kubeconfig must never + # degrade into "whatever config we could find", because that is how a prod command lands on dev. + if [ ! -f "${CLUSTER_KUBECONFIG}" ]; then + printf '\033[31mERROR: dev.sh: CLUSTER=%s but its kubeconfig is MISSING: %s\n Refusing to run. dev.sh does NOT fall back to the other cluster.\033[0m\n' \ + "${CLUSTER}" "${CLUSTER_KUBECONFIG}" >&2 + exit 1 + fi + printf '\033[1m==> dev.sh: cluster=%s (kube context k3s-%s)\033[0m\n' "${CLUSTER}" "${CLUSTER}" >&2 + fi + "$ENGINE" run --rm -it \ --network host \ --security-opt label=disable \ @@ -69,8 +115,21 @@ run() { -v "${HOME}/.ssh:/root/.ssh:ro" \ -v "${HOME}/.config/sops:/root/.config/sops" \ -v "${HOME}/.config/home-k3s/ansible-vault-pass:/root/.config/home-k3s/ansible-vault-pass:ro" \ - -e KUBECONFIG="${DEV_KUBECONFIG:-/work/infra/ansible/.cache/dev.kubeconfig}" \ - "$IMAGE" bash -lc 'if [ "$#" -gt 0 ]; then exec "$@"; else exec bash -l; fi' dev.sh "$@" + -e CLUSTER="${CLUSTER}" \ + -e DEV_KUBECONFIG="${DEV_KUBECONFIG:-}" \ + "$IMAGE" bash -lc ' + set -e + if [ -n "${DEV_KUBECONFIG:-}" ]; then + export KUBECONFIG="${DEV_KUBECONFIG}" + else + # Regenerated every run: cheap, and it means an updated dev/prod.kubeconfig is picked up + # without anyone remembering to refresh a hand-made merged file. Exits non-zero (set -e) + # if the requested cluster is missing — no silent fallback. + KUBECONFIG="$(/work/infra/devcontainer/kubeconfig.sh "${CLUSTER}")" + export KUBECONFIG + fi + if [ "$#" -gt 0 ]; then exec "$@"; else exec bash -l; fi + ' dev.sh "$@" } case "${1:-shell}" in diff --git a/infra/devcontainer/kubeconfig.sh b/infra/devcontainer/kubeconfig.sh new file mode 100755 index 0000000..26f398e --- /dev/null +++ b/infra/devcontainer/kubeconfig.sh @@ -0,0 +1,118 @@ +#!/usr/bin/env bash +# +# infra/devcontainer/kubeconfig.sh — build ONE merged kubeconfig that can address BOTH clusters +# under unambiguous names, and select which of them is current. +# +# kubeconfig.sh # regenerate the merged file, set its current-context, print its path +# +# WHY THIS EXISTS — the name collision that makes the obvious fix dangerous +# infra/ansible/.cache/dev.kubeconfig and .cache/prod.kubeconfig each name their context, their +# cluster AND their user `default`. k3s writes them that way. So the "obvious" fix — +# KUBECONFIG=dev.kubeconfig:prod.kubeconfig — is actively unsafe: kubectl's merge keeps the FIRST +# occurrence of a duplicated name, so `default` silently resolves to DEV and `--context default` +# can NEVER select prod. An operator (or an agent) would believe they were on prod while issuing +# commands against the live dev cluster. +# +# Therefore we RENAME on merge rather than relying on file order: +# dev.kubeconfig -> cluster/context `k3s-dev`, user `k3s-dev-admin` +# prod.kubeconfig -> cluster/context `k3s-prod`, user `k3s-prod-admin` +# The rename is done with `yq` (structural, YAML-aware). It is NOT done with sed: the cluster and +# the user entry are both `- name: default` at identical indent, so a textual rename corrupts the +# file. The two renamed copies are then merged by kubectl itself, which is safe once the names are +# distinct. +# +# OUTPUT +# ${KUBECONFIG_CACHE_DIR:-/infra/ansible/.cache}/clusters.kubeconfig, mode 0600. +# That directory is gitignored (infra/ansible/.gitignore: `.cache/`) and `*.kubeconfig` is ignored +# repo-wide. The file embeds cluster-admin client certs — it must NEVER be committed. It is a +# BUILD ARTIFACT: delete it any time, this script rebuilds it. +# +# TOLERANCE / FAILURE +# A cluster whose source kubeconfig is absent is simply left out of the merged file (so a machine +# that only has dev still works). But asking for a cluster whose kubeconfig is absent is a hard +# ERROR — never a silent fallback to the other cluster. Silently landing on the wrong cluster is +# the exact failure this script exists to prevent. +# +# Requires: kubectl, yq (both baked into the dev container image). Normally invoked by dev.sh from +# inside the container; it runs fine anywhere those two binaries exist. +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "${HERE}/../.." && pwd)" +CACHE_DIR="${KUBECONFIG_CACHE_DIR:-${REPO_ROOT}/infra/ansible/.cache}" +OUT="${CACHE_DIR}/clusters.kubeconfig" + +# cluster name -> source kubeconfig. Add a cluster here and it flows through dev.sh unchanged. +src_for() { + case "$1" in + dev) echo "${DEV_SRC_KUBECONFIG:-${CACHE_DIR}/dev.kubeconfig}" ;; + prod) echo "${PROD_SRC_KUBECONFIG:-${CACHE_DIR}/prod.kubeconfig}" ;; + *) return 1 ;; + esac +} +CLUSTERS=(dev prod) + +die() { printf '\033[31mERROR: kubeconfig.sh: %s\033[0m\n' "$*" >&2; exit 1; } + +TARGET="${1:-}" +[ -n "${TARGET}" ] || die "usage: kubeconfig.sh <$(IFS='|'; echo "${CLUSTERS[*]}")>" +src_for "${TARGET}" >/dev/null 2>&1 || die "unknown cluster '${TARGET}' (known: ${CLUSTERS[*]})" + +for bin in kubectl yq; do + command -v "$bin" >/dev/null 2>&1 || die "'$bin' not found — run this inside infra/devcontainer." +done + +# --- FAIL LOUDLY, do not fall back ------------------------------------------------------------- +target_src="$(src_for "${TARGET}")" +if [ ! -f "${target_src}" ]; then + die "cluster '${TARGET}' was requested but its kubeconfig is MISSING: ${target_src} + Refusing to continue. Falling back to the other cluster would run your '${TARGET}' + commands against a DIFFERENT LIVE CLUSTER. Fetch the kubeconfig first (see + infra/PHASE1-STATUS.md, 'break-glass cert kubeconfig')." +fi + +mkdir -p "${CACHE_DIR}" +tmp="$(mktemp -d)" +chmod 700 "${tmp}" +trap 'rm -rf "${tmp}"' EXIT + +parts=() +present=() +for c in "${CLUSTERS[@]}"; do + s="$(src_for "$c")" + [ -f "$s" ] || { printf ' kubeconfig.sh: %-4s kubeconfig absent (%s) — omitted from the merged config\n' "$c" "$s" >&2; continue; } + ctx="k3s-${c}"; usr="k3s-${c}-admin" + # Structural rename. Every kubeconfig k3s emits here has exactly one cluster/user/context, but + # [] indexing renames all of them consistently rather than assuming index 0 exists. + ctx="$ctx" usr="$usr" yq ' + .clusters[].name = env(ctx) + | .users[].name = env(usr) + | .contexts[].name = env(ctx) + | .contexts[].context.cluster = env(ctx) + | .contexts[].context.user = env(usr) + | ."current-context" = env(ctx) + ' "$s" > "${tmp}/${c}.yaml" + chmod 600 "${tmp}/${c}.yaml" + parts+=("${tmp}/${c}.yaml") + present+=("$c") +done + +[ "${#parts[@]}" -gt 0 ] || die "no kubeconfigs found under ${CACHE_DIR}" + +# Names are distinct now, so kubectl's merge cannot silently drop one. --raw keeps the embedded +# credential data (a merged config without it authenticates as nobody). +KUBECONFIG="$(IFS=:; echo "${parts[*]}")" kubectl config view --flatten --raw > "${tmp}/merged.yaml" +chmod 600 "${tmp}/merged.yaml" +KUBECONFIG="${tmp}/merged.yaml" kubectl config use-context "k3s-${TARGET}" >/dev/null + +install -m 600 "${tmp}/merged.yaml" "${OUT}" + +# Post-condition: the requested context must exist and be current, and must NOT be `default`. +cur="$(KUBECONFIG="${OUT}" kubectl config current-context)" +[ "${cur}" = "k3s-${TARGET}" ] || die "internal: current-context is '${cur}', expected 'k3s-${TARGET}'" +if KUBECONFIG="${OUT}" kubectl config get-contexts -o name | grep -qx default; then + die "internal: a context named 'default' survived the rename — refusing to hand back an ambiguous config" +fi + +printf ' kubeconfig.sh: merged [%s] -> %s (current-context: %s)\n' "${present[*]}" "${OUT}" "${cur}" >&2 +echo "${OUT}"