feat(devcontainer): dev.sh can target prod — and cannot silently target the wrong cluster
Operator report: dev.sh only reached the dev cluster. Line 72 hardcoded
`KUBECONFIG=${DEV_KUBECONFIG:-…/dev.kubeconfig}`, and the only escape was an
undocumented override whose NAME misleads — you set `DEV_KUBECONFIG` to a *prod*
path. infra/PHASE1-STATUS.md had recorded this as a "Known gap" for months.
WHY THIS WAS NOT A ONE-LINER. Both kubeconfigs name their context, cluster AND
user `default`. kubectl's merge keeps the FIRST occurrence of a duplicated name,
so the obvious `KUBECONFIG=dev.kubeconfig:prod.kubeconfig` resolves `default` to
DEV and `--context default` can never select prod — an operator or agent would
believe they were on prod and be issuing commands against the LIVE dev cluster
that runs the operator's platform. Same collision class as ~/.kube/config an hour
earlier, and worse here because agents run in this container.
Interface: `CLUSTER=dev|prod` as an env prefix, default **dev** (back-compat: every
existing invocation, CI job and habit keeps meaning dev).
infra/devcontainer/dev.sh run kubectl get nodes # dev
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # prod
Its one weakness — `CLUSTER` can be inherited from an exported shell var — is
covered by a banner printed on EVERY invocation before the container starts:
`==> dev.sh: cluster=prod (kube context k3s-prod)`.
New infra/devcontainer/kubeconfig.sh generates a merged config with DISTINCT names
(contexts k3s-dev/k3s-prod, users k3s-*-admin), renaming structurally with `yq`
on .clusters[].name / .users[].name / .contexts[].{name,context.cluster,
context.user} before letting `kubectl config view --flatten` merge them. NOT sed —
sed renaming corrupted a config earlier today because cluster and user entries are
both `- name: default` at identical indent. Output is 0600 in the gitignored
.cache/, regenerated every run, and post-conditions assert the current context is
`k3s-<target>` and that NO context named `default` survived.
FAILS LOUDLY AT BOTH LAYERS, and never falls back to the other cluster — that
fallback is exactly how a prod command ends up hitting dev. Host-side refusal if
the requested cluster's kubeconfig is missing; in-container refusal likewise;
unknown CLUSTER exits 2. A missing source for the cluster you did NOT ask for is
tolerated (merges the other one only).
VERIFIED BY ME, independently of the agent — the two BASE_DOMAIN reads are the
unambiguous proof of which cluster was reached, since a node list is not:
default (no selector) -> home.keithjackson.dev (DEV, unchanged)
CLUSTER=prod -> prod.home.keithjackson.dev (PROD)
contexts -> k3s-dev, k3s-prod; NO `default` context survives
exit codes -> invalid CLUSTER=2, valid dev/prod=0
standup.sh -> `--kubeconfig` still wins over the new default
(preflight against prod: target confirmed 192.168.2.9)
credentials -> dev.kubeconfig and prod.kubeconfig byte-intact and each
still reaching 3 nodes after the agent's move-aside test
git -> no kubeconfig committable; clusters.kubeconfig ignored
Full CI gate PASS.
NOT VERIFIED: `dev.sh build` and the interactive shell path (existing image reused,
no TTY available). The shell path shares run()'s code; only the TTY differs.
DEV_KUBECONFIG still wins verbatim, now with a deprecation warning. All existing
mounts are untouched: SOPS age keyring rw, ~/.ssh ro, ansible-vault-pass ro, spec
repo ro + SPEC_REPO, --network host, label=disable.
Closes the PHASE1-STATUS "Known gap".
Refs R-ENV-02, R-TEST-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
577c62136b
commit
21e111b08c
5 files changed
+268
-17
No files matched your search
@@ -33,6 +33,57 @@ cd infra/devcontainer
|
||||
```
|
||||
`ENGINE=docker ./dev.sh` to use docker instead of rootless podman.
|
||||
|
||||
## Choosing a cluster — `CLUSTER=dev|prod`
|
||||
|
||||
```bash
|
||||
./dev.sh run kubectl get nodes # DEV — the default, unchanged
|
||||
CLUSTER=prod ./dev.sh run kubectl get nodes # PROD
|
||||
CLUSTER=prod ./dev.sh # interactive shell pointed at prod
|
||||
```
|
||||
|
||||
An env-var **prefix**, so the target sits on the same line as the command. Every invocation prints
|
||||
a banner naming the cluster before the container starts, because `CLUSTER` can also be inherited
|
||||
from an exported shell variable.
|
||||
|
||||
Inside the container `KUBECONFIG` points at a **merged** config holding both clusters under
|
||||
unambiguous names, with `CLUSTER`'s context current:
|
||||
|
||||
```
|
||||
CURRENT NAME CLUSTER AUTHINFO
|
||||
* k3s-dev k3s-dev k3s-dev-admin
|
||||
k3s-prod k3s-prod k3s-prod-admin
|
||||
```
|
||||
|
||||
So `kubectl config current-context` answers "which cluster am I on?" honestly, and
|
||||
`kubectl --context k3s-prod …` works from a dev-defaulted shell.
|
||||
|
||||
**Why a merge script and not `KUBECONFIG=dev.kubeconfig:prod.kubeconfig`.** k3s names the context,
|
||||
the cluster *and* the user in **both** files `default`. kubectl's merge keeps the **first**
|
||||
occurrence of a duplicated name, so a colon-joined `KUBECONFIG` silently resolves `default` to
|
||||
**dev** and `--context default` can never reach prod — you would believe you were on prod while
|
||||
issuing commands against the live dev cluster. `kubeconfig.sh` therefore **renames on merge**
|
||||
(`k3s-dev` / `k3s-prod`, `*-admin` users) with `yq`, structurally — not with `sed`, which corrupts
|
||||
these files because the cluster and the user entry are both `- name: default` at identical indent.
|
||||
|
||||
- Generator: [`kubeconfig.sh`](kubeconfig.sh). Output: `infra/ansible/.cache/clusters.kubeconfig`,
|
||||
mode 0600, **regenerated on every `dev.sh run`** — a build artifact, never hand-edited, never
|
||||
committed (`.cache/` and `*.kubeconfig` are both gitignored; it embeds cluster-admin certs).
|
||||
- A cluster whose source kubeconfig is missing is simply omitted from the merge, so a machine with
|
||||
only `dev.kubeconfig` still works.
|
||||
- **Asking for a cluster whose kubeconfig is absent is a hard error.** `dev.sh` refuses on the host
|
||||
before starting a container and never falls back to the other cluster — that fallback is exactly
|
||||
how a prod command lands on dev.
|
||||
- `bootstrap/standup.sh` is unaffected: it exports its own `KUBECONFIG` from the mandatory
|
||||
`--kubeconfig` flag, which overrides what `dev.sh` sets. Its `--expect-server` guard remains the
|
||||
thing that makes it safe, not `CLUSTER`.
|
||||
|
||||
### `DEV_KUBECONFIG` — superseded, still honoured
|
||||
|
||||
`DEV_KUBECONFIG=<path> ./dev.sh run …` still pins `KUBECONFIG` verbatim and takes precedence over
|
||||
`CLUSTER` (nothing depending on it breaks), but it is **deprecated**: the name says "dev" while the
|
||||
established workaround set it to a *prod* path, and it produces a single-cluster config whose
|
||||
context is the ambiguous `default`. Use `CLUSTER=dev|prod`. Invocations that use it print a warning.
|
||||
|
||||
## What `dev.sh` mounts, and why each one matters
|
||||
|
||||
- `--network host` — Ansible SSHes to the nodes through the pfSense NAT (`192.168.2.102:225x`) and
|
||||
@@ -61,18 +112,16 @@ SSH, which `--network host` provides.
|
||||
|
||||
So: run everything in here. The host needs only podman (or docker).
|
||||
|
||||
## kubectl inside the container — known gap
|
||||
## kubectl inside the container — was a known gap, now fixed
|
||||
|
||||
`dev.sh` sets `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`, a **libvirt-era path
|
||||
that no longer exists**, and does not mount `~/.kube`. Override it explicitly:
|
||||
`dev.sh` used to hard-wire `KUBECONFIG` to `dev.kubeconfig` with no supported way to reach prod, and
|
||||
it does not mount `~/.kube`. It now selects between `infra/ansible/.cache/dev.kubeconfig` and
|
||||
`prod.kubeconfig` via `CLUSTER=dev|prod` (see above) and merges both under distinct context names.
|
||||
|
||||
```bash
|
||||
DEV_KUBECONFIG=/work/path/to/break-glass.kubeconfig ./dev.sh run kubectl get nodes
|
||||
```
|
||||
|
||||
Use a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login needs a browser,
|
||||
which a container shell does not have. See [`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both
|
||||
access paths. (Not yet fixed in `dev.sh`.)
|
||||
The container still needs a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login
|
||||
needs a browser, which a container shell does not have. See
|
||||
[`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both access paths and for how to regenerate these
|
||||
two files from `/etc/rancher/k3s/k3s.yaml`.
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
Reference in new issue
Block a user