chore(backup): R2 credentials for the off-cluster DR target

Written by platform/backup/set-r2-credentials.sh (operator, 2026-09-07T05:41:34Z)
into all four consumers, in the two shapes they need: velero wants an AWS INI
file, CNPG/barman wants discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY. All four
verified to carry the SAME 32-character key id, so no consumer was missed.

Proven from a pod INSIDE prod against
https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com:
head-bucket on k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both
succeed. Byte-integrity of GET is NOT yet proven (the aws-cli image has no cmp);
s3check is what will establish it.

Credentials only — no endpoint or bucket wiring. That is feat/r2-backup-prod.

Co-Authored-By: Claude Opus 5 <[email protected]>

infra/devcontainer/README.md

@@ -33,6 +33,84 @@ cd infra/devcontainer
 ```
 `ENGINE=docker ./dev.sh` to use docker instead of rootless podman.
 
+## Choosing a cluster — `CLUSTER=dev|prod`
+
+```bash
+./dev.sh run kubectl get nodes                 # DEV — the default, unchanged
+CLUSTER=prod ./dev.sh run kubectl get nodes    # PROD
+CLUSTER=prod ./dev.sh                          # interactive shell pointed at prod
+```
+
+An env-var **prefix**, so the target sits on the same line as the command. Every invocation prints
+a banner naming the cluster before the container starts, because `CLUSTER` can also be inherited
+from an exported shell variable.
+
+Inside the container `KUBECONFIG` is a two-file **chain**,
+`<per-invocation selector>:<shared merged config>`:
+
+| file | shared? | holds |
+|---|---|---|
+| selector, in a private `mktemp` dir | no — one per `dev.sh` invocation | only `current-context: k3s-$CLUSTER`; no credentials |
+| `infra/ansible/.cache/clusters.kubeconfig` | yes — bind-mounted, read by every container | both clusters under unambiguous names; **no `current-context`** |
+
+kubectl merges the chain first-wins, so the selector supplies this invocation's default:
+
+```
+CURRENT   NAME       CLUSTER    AUTHINFO
+*         k3s-dev    k3s-dev    k3s-dev-admin
+          k3s-prod   k3s-prod   k3s-prod-admin
+```
+
+So `kubectl config current-context` answers "which cluster am I on?" honestly, and
+`kubectl --context k3s-prod …` works from a dev-defaulted shell.
+
+**Why the default context is not stored in the merged file.** It used to be, and that was a
+cross-cluster data-corruption bug. The merged file lives in the repo's `.cache`, which every
+container bind-mounts, so `current-context` was **shared mutable state**: two concurrent `dev.sh`
+runs with different `CLUSTER` values overwrote each other's default *mid-command*. A
+`CLUSTER=prod` script read prod for its first half and **dev** for its second; it was caught only
+because a dev-only PVC name showed up in a prod query. Moving selection into a per-invocation file
+removes the shared field rather than locking it — there is no window left to lose. Read on its own
+the merged file now resolves to nothing (`error: current-context is not set`), which is loud and
+safe; `--context k3s-dev` / `--context k3s-prod` against it are unaffected and remain the pattern
+to prefer in scripts. Concurrent `CLUSTER=dev` and `CLUSTER=prod` invocations are now safe.
+
+Consequences worth knowing:
+
+- Export what `kubeconfig.sh` prints **verbatim** — it is a colon-joined chain, not one path.
+- `kubectl config use-context …` in an interactive toolbox shell writes to the **first** file in
+  the chain (the selector), so a manual switch stays local to that container.
+
+**Why a merge script and not `KUBECONFIG=dev.kubeconfig:prod.kubeconfig`.** k3s names the context,
+the cluster *and* the user in **both** files `default`. kubectl's merge keeps the **first**
+occurrence of a duplicated name, so a colon-joined `KUBECONFIG` silently resolves `default` to
+**dev** and `--context default` can never reach prod — you would believe you were on prod while
+issuing commands against the live dev cluster. `kubeconfig.sh` therefore **renames on merge**
+(`k3s-dev` / `k3s-prod`, `*-admin` users) with `yq`, structurally — not with `sed`, which corrupts
+these files because the cluster and the user entry are both `- name: default` at identical indent.
+
+- Generator: [`kubeconfig.sh`](kubeconfig.sh). Output: `infra/ansible/.cache/clusters.kubeconfig`,
+  mode 0600, **regenerated on every `dev.sh run`** — a build artifact, never hand-edited, never
+  committed (`.cache/` and `*.kubeconfig` are both gitignored; it embeds cluster-admin certs).
+  Its content no longer depends on which cluster was asked for, so concurrent regenerations write
+  identical bytes; the write is published by atomic rename so a concurrent reader can never load a
+  half-written kubeconfig.
+- A cluster whose source kubeconfig is missing is simply omitted from the merge, so a machine with
+  only `dev.kubeconfig` still works.
+- **Asking for a cluster whose kubeconfig is absent is a hard error.** `dev.sh` refuses on the host
+  before starting a container and never falls back to the other cluster — that fallback is exactly
+  how a prod command lands on dev.
+- `bootstrap/standup.sh` is unaffected: it exports its own `KUBECONFIG` from the mandatory
+  `--kubeconfig` flag, which overrides what `dev.sh` sets. Its `--expect-server` guard remains the
+  thing that makes it safe, not `CLUSTER`.
+
+### `DEV_KUBECONFIG` — superseded, still honoured
+
+`DEV_KUBECONFIG=<path> ./dev.sh run …` still pins `KUBECONFIG` verbatim and takes precedence over
+`CLUSTER` (nothing depending on it breaks), but it is **deprecated**: the name says "dev" while the
+established workaround set it to a *prod* path, and it produces a single-cluster config whose
+context is the ambiguous `default`. Use `CLUSTER=dev|prod`. Invocations that use it print a warning.
+
 ## What `dev.sh` mounts, and why each one matters
 
 - `--network host` — Ansible SSHes to the nodes through the pfSense NAT (`192.168.2.102:225x`) and
@@ -61,18 +139,16 @@ SSH, which `--network host` provides.
 
 So: run everything in here. The host needs only podman (or docker).
 
-## kubectl inside the container — known gap
+## kubectl inside the container — was a known gap, now fixed
 
-`dev.sh` sets `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`, a **libvirt-era path
-that no longer exists**, and does not mount `~/.kube`. Override it explicitly:
+`dev.sh` used to hard-wire `KUBECONFIG` to `dev.kubeconfig` with no supported way to reach prod, and
+it does not mount `~/.kube`. It now selects between `infra/ansible/.cache/dev.kubeconfig` and
+`prod.kubeconfig` via `CLUSTER=dev|prod` (see above) and merges both under distinct context names.
 
-```bash
-DEV_KUBECONFIG=/work/path/to/break-glass.kubeconfig ./dev.sh run kubectl get nodes
-```
-
-Use a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login needs a browser,
-which a container shell does not have. See [`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both
-access paths. (Not yet fixed in `dev.sh`.)
+The container still needs a **break-glass cert kubeconfig**, not the `k3s-oidc` context — OIDC login
+needs a browser, which a container shell does not have. See
+[`../PHASE1-STATUS.md`](../PHASE1-STATUS.md) for both access paths and for how to regenerate these
+two files from `/etc/rancher/k3s/k3s.yaml`.
 
 ## Notes