0 Fork 0

Files

T

jakman85andClaude Opus 5 21e111b08c feat(devcontainer): dev.sh can target prod — and cannot silently target the wrong cluster

Operator report: dev.sh only reached the dev cluster. Line 72 hardcoded
`KUBECONFIG=${DEV_KUBECONFIG:-…/dev.kubeconfig}`, and the only escape was an
undocumented override whose NAME misleads — you set `DEV_KUBECONFIG` to a *prod*
path. infra/PHASE1-STATUS.md had recorded this as a "Known gap" for months.

WHY THIS WAS NOT A ONE-LINER. Both kubeconfigs name their context, cluster AND
user `default`. kubectl's merge keeps the FIRST occurrence of a duplicated name,
so the obvious `KUBECONFIG=dev.kubeconfig:prod.kubeconfig` resolves `default` to
DEV and `--context default` can never select prod — an operator or agent would
believe they were on prod and be issuing commands against the LIVE dev cluster
that runs the operator's platform. Same collision class as ~/.kube/config an hour
earlier, and worse here because agents run in this container.

Interface: `CLUSTER=dev|prod` as an env prefix, default **dev** (back-compat: every
existing invocation, CI job and habit keeps meaning dev).

    infra/devcontainer/dev.sh run kubectl get nodes                # dev
    CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes   # prod

Its one weakness — `CLUSTER` can be inherited from an exported shell var — is
covered by a banner printed on EVERY invocation before the container starts:
`==> dev.sh: cluster=prod (kube context k3s-prod)`.

New infra/devcontainer/kubeconfig.sh generates a merged config with DISTINCT names
(contexts k3s-dev/k3s-prod, users k3s-*-admin), renaming structurally with `yq`
on .clusters[].name / .users[].name / .contexts[].{name,context.cluster,
context.user} before letting `kubectl config view --flatten` merge them. NOT sed —
sed renaming corrupted a config earlier today because cluster and user entries are
both `- name: default` at identical indent. Output is 0600 in the gitignored
.cache/, regenerated every run, and post-conditions assert the current context is
`k3s-<target>` and that NO context named `default` survived.

FAILS LOUDLY AT BOTH LAYERS, and never falls back to the other cluster — that
fallback is exactly how a prod command ends up hitting dev. Host-side refusal if
the requested cluster's kubeconfig is missing; in-container refusal likewise;
unknown CLUSTER exits 2. A missing source for the cluster you did NOT ask for is
tolerated (merges the other one only).

VERIFIED BY ME, independently of the agent — the two BASE_DOMAIN reads are the
unambiguous proof of which cluster was reached, since a node list is not:
  default (no selector)  -> home.keithjackson.dev        (DEV, unchanged)
  CLUSTER=prod           -> prod.home.keithjackson.dev   (PROD)
  contexts               -> k3s-dev, k3s-prod; NO `default` context survives
  exit codes             -> invalid CLUSTER=2, valid dev/prod=0
  standup.sh             -> `--kubeconfig` still wins over the new default
                            (preflight against prod: target confirmed 192.168.2.9)
  credentials            -> dev.kubeconfig and prod.kubeconfig byte-intact and each
                            still reaching 3 nodes after the agent's move-aside test
  git                    -> no kubeconfig committable; clusters.kubeconfig ignored
Full CI gate PASS.

NOT VERIFIED: `dev.sh build` and the interactive shell path (existing image reused,
no TTY available). The shell path shares run()'s code; only the TTY differs.

DEV_KUBECONFIG still wins verbatim, now with a deprecation warning. All existing
mounts are untouched: SOPS age keyring rw, ~/.ssh ro, ansible-vault-pass ro, spec
repo ro + SPEC_REPO, --network host, label=disable.

Closes the PHASE1-STATUS "Known gap".

Refs R-ENV-02, R-TEST-02.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-05 16:38:32 -05:00

9.0 KiB

Raw Blame History

Dev cluster — current state & reconnect guide

Last verified: 2026-08-31. Phase 1 is complete and long since exited; the cluster has been running the full platform (phases 2–6) for some time. This doc is how a session — human or agent — reconnects to it and re-establishes the toolchain.

New PM / agent taking over? Read this doc (reconnect + toolchain), then the latest DECISIONS-LOG.md entry (2026-08-31) — it is the current-state + gotchas + prioritized Next Steps handoff — and the plans under docs/plans/. Start at Next Steps.

Phase 7 (prod) is OPEN as of 2026-08-31 — prep only. The first prod node (k3s-prod-1, 192.168.2.10) is on the bench. This document covers the dev cluster; for prod see PROD-PROVISIONING.md (T-7.1 checklist) and the Phase-7 entry in DECISIONS-LOG.md. Nothing is installed on prod hardware until the R-ENV-02 tokenization blockers close and the gate is green.

History note. The cluster was originally 3 libvirt/KVM VMs on the workstation (10.73.0.0/24 NAT). That was retired after the workstation ran out of RAM/disk; the cluster now lives on Proxmox VE. Anything mentioning virsh, qemu:///system, tofu+dmacvicar/libvirt, or 10.73.0.x for the cluster is historical — including cluster.sh (retired) and PHASE1-SCOPE.md (historical scoping doc).

The cluster


Substrate	Proxmox VE host, node `proxmox`, disks on the `Storage` ZFS datastore, bridge `vmbr0`
Nodes	`k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 8 vCPU, 24 GiB, Debian 13 (trixie)
k3s	v1.36.4+k3s1, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik
CNI	Cilium 1.20.1 (kube-proxy replacement, L2 announcements, egress gateway)
kube-API	kube-vip VIP `192.168.1.49:6443`, tls-san'd on the serving cert
Ingress VIP	`192.168.1.53` (shared Gateway). Pool `192.168.1.53–.60` for LB VIPs; UniFi holds `.54`
Storage	Longhorn 1.12.1 — classes `longhorn` (default), `longhorn-static`, `longhorn-vm` (RWX/Block, migratable), `longhorn-scratch`. `bulk` (NFS CSI) is a non-provisioning stub pending OQ-2
Nested virt	`cpu_type = host` → real KVM in-guest; KubeVirt + CDI run on it

Everything above is provisioned by IaC: tofu/ creates the VMs, cloud-init/ baselines first boot, ansible/ installs k3s, Cilium, kube-vip and the VPN egress gateway. Flux then reconciles the whole platform from clusters/dev.

Network path from the workstation (there is a NAT in the way)

The workstation is not on the cluster LAN. A pfSense NAT bridges the two:

Purpose	Workstation-side target	Forwards to
kube-API	`https://192.168.2.50:6443`	kube-vip VIP `192.168.1.49:6443`
Node SSH (Ansible only)	`192.168.2.102:2250 / 2251 / 2252`	`192.168.1.50 / .51 / .52` port 22
Proxmox API	`https://192.168.2.102:8006/`	the Proxmox host

TLS through the NAT validates because the kubeconfig sets tls-server-name: 192.168.1.49 — the VIP is in the serving cert's SAN list, so the NAT address does not have to be.

There is no SSH tunnel any more; the old tunnel was retired when kube-vip + the NAT landed.

Access

kubectl — Keycloak SSO (normal path). ~/.kube/config context k3s-oidc authenticates against the homelab realm via kubelogin (auth-code + PKCE, public client, no secret in the kubeconfig). Group membership maps to RBAC by platform/keycloak/k8s-oidc-rbac.yaml:

kubectl config use-context k3s-oidc
kubectl auth whoami     # -> oidc:<you>, groups [oidc:platform-admins ...]

oidc:platform-admins → cluster-admin, oidc:viewers → view. First use opens a browser, so this path needs an interactive desktop.

kubectl — cert kubeconfig (break-glass). The k3s admin cert kubeconfig from /etc/rancher/k3s/k3s.yaml on k3s-dev-1 (rewrite the server to https://192.168.2.50:6443 and add tls-server-name: 192.168.1.49). Fully independent of Keycloak — use this when OIDC, the gateway, or Keycloak itself is down. Headless contexts (CI jobs, containers, agents) also need this, since they cannot complete a browser login.

Node SSH. ssh -i ~/.ssh/home_network -p 2250 [email protected] (user k3s, NOPASSWD sudo). Ports 2251/2252 for nodes 2/3.

Toolchain — use the dev container, not the host

Every pinned tool (tofu, ansible, sops, age, flux, kubectl, helm and the full CI gate toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image:

infra/devcontainer/dev.sh build                    # once
infra/devcontainer/dev.sh run make -C ci gate      # exactly what CI runs
infra/devcontainer/dev.sh                          # interactive shell at /work

infra/devcontainer/dev.sh run kubectl get nodes                # DEV (the default)
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes   # PROD

Unlike the libvirt era, tofu and ansible now run fine inside the container — the bpg/proxmox provider talks to the Proxmox API over the network and needs no host libvirt socket, which is what broke the old dmacvicar/libvirt provider in a container.

dev.sh mounts the repo at /work, ~/.ssh (read-only), ~/.config/sops (the age keyring — persisted on the host, or you lose access to every SOPS secret), the ansible-vault passphrase file, and the sibling spec repo read-only at /home-network-k3s so the spec-immutability check runs instead of silently skipping.

Closed (was a known gap): dev.sh can now target either cluster. It used to hard-wire KUBECONFIG to dev.kubeconfig, leaving an undocumented — and misleadingly named — DEV_KUBECONFIG=<prod path> as the only route to prod. Select with CLUSTER=dev|prod; dev remains the default, so every existing invocation means what it always meant. Inside the container KUBECONFIG is a merged config carrying both clusters as contexts k3s-dev and k3s-prod (never default), regenerated each run by infra/devcontainer/kubeconfig.sh into the gitignored infra/ansible/.cache/clusters.kubeconfig. Naming matters: k3s writes both source kubeconfigs with context, cluster and user all called default, and kubectl's merge keeps the first duplicate — so a plain KUBECONFIG=dev:prod would silently resolve default to dev while you believed you were on prod. Requesting a cluster whose kubeconfig is absent is a hard error, never a fallback. DEV_KUBECONFIG still works and still wins, but is superseded — it is opaque about its target and yields an ambiguous default context. ~/.kube is still not mounted and the OIDC context still cannot work in a container (no browser); use the break-glass cert kubeconfigs above. Details: devcontainer/README.md.

Off-repo state you must not lose

What	Where
SOPS age private key	`~/.config/sops/age/keys.txt` — losing it locks you out of every encrypted secret
Ansible vault passphrase	`~/.config/home-k3s/ansible-vault-pass`
Node SSH key	`~/.ssh/home_network`
Proxmox API token	operator env only (`PROXMOX_VE_API_TOKEN`) — never a tofu var, never in state
tofu state	`infra/tofu/terraform.tfstate` (gitignored) — absent on a fresh workstation; a `tofu apply` without it would try to recreate the live nodes
Generated SSO secrets	`~/.config/home-k3s/keycloak-sso-secrets.txt`

Health check

kubectl get nodes                                  # 3/3 Ready
flux get kustomizations                            # flux-system, platform, platform-config,
                                                   # apps-wave-0/1/2, apps-vms — all Ready
kubectl get helmreleases -A                        # all Ready
kubectl get cluster -A                             # CNPG: keycloak-db, nextcloud-db, semaphore-db
kubectl get vmi -A                                 # unifi Running

Flux's git source is inside the cluster — mind the loop

GitRepository/flux-system points at https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git: the in-cluster Gitea, which is itself an app Flux deploys in apps/wave-0. Resolution goes through split-horizon CoreDNS → Gateway VIP 192.168.1.53, so a Flux reconcile depends on Gitea, the Gateway, CoreDNS and the wildcard cert all being up.

In practice this is stable — Gitea's Deployment and PVC survive a reboot and come back without Flux's help, so a Proxmox host reboot self-heals (VMs auto-start, cluster reconverges). The loop only bites if Gitea is lost rather than restarted. The recovery path is the host bootstrap Gitea (bash bootstrap/gitea/bootstrap-gitea.sh) plus repointing the GitRepository at it — that is what the bootstrap/ tree exists for, and it is the reason the bootstrap Gitea is kept.

Push target. origin is that same in-cluster Gitea, so git push origin main is the deploy. There is no separate "apply" step and no GitHub remote.

9.0 KiB Raw Blame History Unescape Escape