jakman85andClaude Opus 5
21e111b08c
Operator report: dev.sh only reached the dev cluster. Line 72 hardcoded
`KUBECONFIG=${DEV_KUBECONFIG:-…/dev.kubeconfig}`, and the only escape was an
undocumented override whose NAME misleads — you set `DEV_KUBECONFIG` to a *prod*
path. infra/PHASE1-STATUS.md had recorded this as a "Known gap" for months.
WHY THIS WAS NOT A ONE-LINER. Both kubeconfigs name their context, cluster AND
user `default`. kubectl's merge keeps the FIRST occurrence of a duplicated name,
so the obvious `KUBECONFIG=dev.kubeconfig:prod.kubeconfig` resolves `default` to
DEV and `--context default` can never select prod — an operator or agent would
believe they were on prod and be issuing commands against the LIVE dev cluster
that runs the operator's platform. Same collision class as ~/.kube/config an hour
earlier, and worse here because agents run in this container.
Interface: `CLUSTER=dev|prod` as an env prefix, default **dev** (back-compat: every
existing invocation, CI job and habit keeps meaning dev).
infra/devcontainer/dev.sh run kubectl get nodes # dev
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # prod
Its one weakness — `CLUSTER` can be inherited from an exported shell var — is
covered by a banner printed on EVERY invocation before the container starts:
`==> dev.sh: cluster=prod (kube context k3s-prod)`.
New infra/devcontainer/kubeconfig.sh generates a merged config with DISTINCT names
(contexts k3s-dev/k3s-prod, users k3s-*-admin), renaming structurally with `yq`
on .clusters[].name / .users[].name / .contexts[].{name,context.cluster,
context.user} before letting `kubectl config view --flatten` merge them. NOT sed —
sed renaming corrupted a config earlier today because cluster and user entries are
both `- name: default` at identical indent. Output is 0600 in the gitignored
.cache/, regenerated every run, and post-conditions assert the current context is
`k3s-<target>` and that NO context named `default` survived.
FAILS LOUDLY AT BOTH LAYERS, and never falls back to the other cluster — that
fallback is exactly how a prod command ends up hitting dev. Host-side refusal if
the requested cluster's kubeconfig is missing; in-container refusal likewise;
unknown CLUSTER exits 2. A missing source for the cluster you did NOT ask for is
tolerated (merges the other one only).
VERIFIED BY ME, independently of the agent — the two BASE_DOMAIN reads are the
unambiguous proof of which cluster was reached, since a node list is not:
default (no selector) -> home.keithjackson.dev (DEV, unchanged)
CLUSTER=prod -> prod.home.keithjackson.dev (PROD)
contexts -> k3s-dev, k3s-prod; NO `default` context survives
exit codes -> invalid CLUSTER=2, valid dev/prod=0
standup.sh -> `--kubeconfig` still wins over the new default
(preflight against prod: target confirmed 192.168.2.9)
credentials -> dev.kubeconfig and prod.kubeconfig byte-intact and each
still reaching 3 nodes after the agent's move-aside test
git -> no kubeconfig committable; clusters.kubeconfig ignored
Full CI gate PASS.
NOT VERIFIED: `dev.sh build` and the interactive shell path (existing image reused,
no TTY available). The shell path shares run()'s code; only the TTY differs.
DEV_KUBECONFIG still wins verbatim, now with a deprecation warning. All existing
mounts are untouched: SOPS age keyring rw, ~/.ssh ro, ansible-vault-pass ro, spec
repo ro + SPEC_REPO, --network host, label=disable.
Closes the PHASE1-STATUS "Known gap".
Refs R-ENV-02, R-TEST-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
133 lines
7.3 KiB
Markdown
133 lines
7.3 KiB
Markdown
# Dev toolbox (containerized baseline)
|
|
|
|
The toolchain for this repo is **not all Python** (OpenTofu, `age`, `sops`, `flux`, `chainsaw`,
|
|
`kyverno` are Go binaries; only Ansible and a couple of linters are Python). Per the operator's
|
|
rule it is therefore delivered as a **pinned dev container** — no host pollution, one reproducible
|
|
baseline, and the *same* binaries CI uses.
|
|
|
|
**This is the supported way to run anything in this repo.** Don't install the toolchain on the host.
|
|
|
|
## What's inside (pinned in `versions.env`)
|
|
|
|
| Group | Tools |
|
|
|---|---|
|
|
| Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
|
|
| GitOps / cluster | `flux`, `kubectl`, `helm` |
|
|
| Secrets | `sops`, `age` + `age-keygen` |
|
|
| **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
|
|
| Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |
|
|
|
|
The conformance-tool versions are kept in lockstep with `ci/versions.env` so
|
|
`make -C ci gate` in here is byte-identical to the CI runner (**R-TEST-02**). The image records
|
|
the resolved set at `/etc/dev-toolchain-versions`.
|
|
|
|
## Use
|
|
|
|
```bash
|
|
cd infra/devcontainer
|
|
./dev.sh build # build the image (needs network)
|
|
./dev.sh # build if needed + shell in /work (the repo)
|
|
./dev.sh run make -C ci gate # the merge gate, exactly as CI runs it
|
|
./dev.sh run sops -d path/to.sops.yaml
|
|
./dev.sh run tofu -chdir=/work/infra/tofu plan
|
|
```
|
|
`ENGINE=docker ./dev.sh` to use docker instead of rootless podman.
|
|
|
|
## Choosing a cluster — `CLUSTER=dev|prod`
|
|
|
|
```bash
|
|
./dev.sh run kubectl get nodes # DEV — the default, unchanged
|
|
CLUSTER=prod ./dev.sh run kubectl get nodes # PROD
|
|
CLUSTER=prod ./dev.sh # interactive shell pointed at prod
|
|
```
|
|
|
|
An env-var **prefix**, so the target sits on the same line as the command. Every invocation prints
|
|
a banner naming the cluster before the container starts, because `CLUSTER` can also be inherited
|
|
from an exported shell variable.
|
|
|
|
Inside the container `KUBECONFIG` points at a **merged** config holding both clusters under
|
|
unambiguous names, with `CLUSTER`'s context current:
|
|
|
|
```
|
|
CURRENT NAME CLUSTER AUTHINFO
|
|
* k3s-dev k3s-dev k3s-dev-admin
|
|
k3s-prod k3s-prod k3s-prod-admin
|
|
```
|
|
|
|
So `kubectl config current-context` answers "which cluster am I on?" honestly, and
|
|
`kubectl --context k3s-prod …` works from a dev-defaulted shell.
|
|
|
|
**Why a merge script and not `KUBECONFIG=dev.kubeconfig:prod.kubeconfig`.** k3s names the context,
|
|
the cluster *and* the user in **both** files `default`. kubectl's merge keeps the **first**
|
|
occurrence of a duplicated name, so a colon-joined `KUBECONFIG` silently resolves `default` to
|
|
**dev** and `--context default` can never reach prod — you would believe you were on prod while
|
|
issuing commands against the live dev cluster. `kubeconfig.sh` therefore **renames on merge**
|
|
(`k3s-dev` / `k3s-prod`, `*-admin` users) with `yq`, structurally — not with `sed`, which corrupts
|
|
these files because the cluster and the user entry are both `- name: default` at identical indent.
|
|
|
|
- Generator: [`kubeconfig.sh`](kubeconfig.sh). Output: `infra/ansible/.cache/clusters.kubeconfig`,
|
|
mode 0600, **regenerated on every `dev.sh run`** — a build artifact, never hand-edited, never
|
|
committed (`.cache/` and `*.kubeconfig` are both gitignored; it embeds cluster-admin certs).
|
|
- A cluster whose source kubeconfig is missing is simply omitted from the merge, so a machine with
|
|
only `dev.kubeconfig` still works.
|
|
- **Asking for a cluster whose kubeconfig is absent is a hard error.** `dev.sh` refuses on the host
|
|
before starting a container and never falls back to the other cluster — that fallback is exactly
|
|
how a prod command lands on dev.
|
|
- `bootstrap/standup.sh` is unaffected: it exports its own `KUBECONFIG` from the mandatory
|
|
`--kubeconfig` flag, which overrides what `dev.sh` sets. Its `--expect-server` guard remains the
|
|
thing that makes it safe, not `CLUSTER`.
|
|
|
|
### `DEV_KUBECONFIG` — superseded, still honoured
|
|
|
|
`DEV_KUBECONFIG=<path> ./dev.sh run …` still pins `KUBECONFIG` verbatim and takes precedence over
|
|
`CLUSTER` (nothing depending on it breaks), but it is **deprecated**: the name says "dev" while the
|
|
established workaround set it to a *prod* path, and it produces a single-cluster config whose
|
|
context is the ambiguous `default`. Use `CLUSTER=dev|prod`. Invocations that use it print a warning.
|
|
|
|
## What `dev.sh` mounts, and why each one matters
|
|
|
|
- `--network host` — Ansible SSHes to the nodes through the pfSense NAT (`192.168.2.102:225x`) and
|
|
`kubectl`/`flux`/`tofu` reach the kube-API (`192.168.2.50:6443`) and the Proxmox API
|
|
(`192.168.2.102:8006`) on the host's network.
|
|
- `-v $REPO:/work` (workdir) — the repo.
|
|
- `-v ~/.config/sops:/root/.config/sops` — **the age keyring, read-write**. This is deliberate: the
|
|
private key generated by `bootstrap/sops/age-keygen.sh` must land on the *host*, or it dies with
|
|
the `--rm` container and every SOPS-encrypted secret in the repo becomes undecryptable.
|
|
- `-v ~/.ssh:/root/.ssh:ro` — node SSH key (`home_network`).
|
|
- `-v ~/.config/home-k3s/ansible-vault-pass:...:ro` — the Ansible vault passphrase, so vaulted
|
|
`group_vars` decrypt unattended.
|
|
- `-v ../home-network-k3s:/home-network-k3s:ro` + `SPEC_REPO` — the **spec repo**, read-only, when
|
|
it exists beside this one. Without it `ci/spec-immutability.sh` honestly *skips*
|
|
rather than passing — mount it, or that gate silently verifies nothing.
|
|
- `--security-opt label=disable` — SELinux hosts, so the bind mounts work without relabeling.
|
|
|
|
## tofu and ansible now run **inside** the container
|
|
|
|
This reverses the old Phase-1 limitation. When the dev cluster was on host libvirt, OpenTofu's
|
|
`dmacvicar/libvirt` provider could not do a read-write `qemu:///system` connect from inside a
|
|
container (polkit/PID-identity: `Cannot find start time for pid`), so tofu had to be extracted and
|
|
run on the host. The cluster moved to **Proxmox VE** and the provider is now `bpg/proxmox`, which
|
|
talks to the Proxmox **HTTP API** — nothing host-local is required. Ansible likewise only needs
|
|
SSH, which `--network host` provides.
|
|
|
|
So: run everything in here. The host needs only podman (or docker).
|
|
|
|
## kubectl inside the container — was a known gap, now fixed
|
|
|
|
`dev.sh` used to hard-wire `KUBECONFIG` to `dev.kubeconfig` with no supported way to reach prod, and
|
|
it does not mount `~/.kube`. It now selects between `infra/ansible/.cache/dev.kubeconfig` and
|
|
`prod.kubeconfig` via `CLUSTER=dev|prod` (see above) and merges both under distinct context names.
|
|
|
|
The container still needs a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login
|
|
needs a browser, which a container shell does not have. See
|
|
[`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both access paths and for how to regenerate these
|
|
two files from `/etc/rancher/k3s/k3s.yaml`.
|
|
|
|
## Notes
|
|
|
|
- Versions are **best-effort pins**; if a download 404s at build, bump the value in `versions.env`
|
|
and rebuild. Keep the conformance-tool block aligned with `ci/versions.env`.
|
|
- **Secrets never enter the image** — they arrive only as the bind mounts listed above.
|
|
- This container is the toolchain, not the cluster. The cluster runs on Proxmox VE; see
|
|
[`../PHASE1-STATUS.md`](../PHASE1-STATUS.md).
|