jakman85andClaude Opus 5
21e111b08c
Operator report: dev.sh only reached the dev cluster. Line 72 hardcoded
`KUBECONFIG=${DEV_KUBECONFIG:-…/dev.kubeconfig}`, and the only escape was an
undocumented override whose NAME misleads — you set `DEV_KUBECONFIG` to a *prod*
path. infra/PHASE1-STATUS.md had recorded this as a "Known gap" for months.
WHY THIS WAS NOT A ONE-LINER. Both kubeconfigs name their context, cluster AND
user `default`. kubectl's merge keeps the FIRST occurrence of a duplicated name,
so the obvious `KUBECONFIG=dev.kubeconfig:prod.kubeconfig` resolves `default` to
DEV and `--context default` can never select prod — an operator or agent would
believe they were on prod and be issuing commands against the LIVE dev cluster
that runs the operator's platform. Same collision class as ~/.kube/config an hour
earlier, and worse here because agents run in this container.
Interface: `CLUSTER=dev|prod` as an env prefix, default **dev** (back-compat: every
existing invocation, CI job and habit keeps meaning dev).
infra/devcontainer/dev.sh run kubectl get nodes # dev
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # prod
Its one weakness — `CLUSTER` can be inherited from an exported shell var — is
covered by a banner printed on EVERY invocation before the container starts:
`==> dev.sh: cluster=prod (kube context k3s-prod)`.
New infra/devcontainer/kubeconfig.sh generates a merged config with DISTINCT names
(contexts k3s-dev/k3s-prod, users k3s-*-admin), renaming structurally with `yq`
on .clusters[].name / .users[].name / .contexts[].{name,context.cluster,
context.user} before letting `kubectl config view --flatten` merge them. NOT sed —
sed renaming corrupted a config earlier today because cluster and user entries are
both `- name: default` at identical indent. Output is 0600 in the gitignored
.cache/, regenerated every run, and post-conditions assert the current context is
`k3s-<target>` and that NO context named `default` survived.
FAILS LOUDLY AT BOTH LAYERS, and never falls back to the other cluster — that
fallback is exactly how a prod command ends up hitting dev. Host-side refusal if
the requested cluster's kubeconfig is missing; in-container refusal likewise;
unknown CLUSTER exits 2. A missing source for the cluster you did NOT ask for is
tolerated (merges the other one only).
VERIFIED BY ME, independently of the agent — the two BASE_DOMAIN reads are the
unambiguous proof of which cluster was reached, since a node list is not:
default (no selector) -> home.keithjackson.dev (DEV, unchanged)
CLUSTER=prod -> prod.home.keithjackson.dev (PROD)
contexts -> k3s-dev, k3s-prod; NO `default` context survives
exit codes -> invalid CLUSTER=2, valid dev/prod=0
standup.sh -> `--kubeconfig` still wins over the new default
(preflight against prod: target confirmed 192.168.2.9)
credentials -> dev.kubeconfig and prod.kubeconfig byte-intact and each
still reaching 3 nodes after the agent's move-aside test
git -> no kubeconfig committable; clusters.kubeconfig ignored
Full CI gate PASS.
NOT VERIFIED: `dev.sh build` and the interactive shell path (existing image reused,
no TTY available). The shell path shares run()'s code; only the TTY differs.
DEV_KUBECONFIG still wins verbatim, now with a deprecation warning. All existing
mounts are untouched: SOPS age keyring rw, ~/.ssh ro, ansible-vault-pass ro, spec
repo ro + SPEC_REPO, --network host, label=disable.
Closes the PHASE1-STATUS "Known gap".
Refs R-ENV-02, R-TEST-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
154 lines
9.0 KiB
Markdown
154 lines
9.0 KiB
Markdown
# Dev cluster — current state & reconnect guide
|
||
|
||
**Last verified: 2026-08-31.** Phase 1 is **complete and long since exited**; the cluster has been
|
||
running the full platform (phases 2–6) for some time. This doc is how a session — human or agent —
|
||
reconnects to it and re-establishes the toolchain.
|
||
|
||
> **New PM / agent taking over?** Read this doc (reconnect + toolchain), then the **latest
|
||
> `DECISIONS-LOG.md` entry (2026-08-31)** — it is the current-state + gotchas + prioritized **Next
|
||
> Steps** handoff — and the plans under `docs/plans/`. Start at Next Steps.
|
||
|
||
> **Phase 7 (prod) is OPEN as of 2026-08-31 — prep only.** The first prod node
|
||
> (`k3s-prod-1`, `192.168.2.10`) is on the bench. This document covers the **dev** cluster;
|
||
> for prod see [`PROD-PROVISIONING.md`](PROD-PROVISIONING.md) (T-7.1 checklist) and the
|
||
> Phase-7 entry in `DECISIONS-LOG.md`. Nothing is installed on prod hardware until the
|
||
> R-ENV-02 tokenization blockers close and the gate is green.
|
||
|
||
> **History note.** The cluster was originally 3 libvirt/KVM VMs on the workstation
|
||
> (`10.73.0.0/24` NAT). That was **retired** after the workstation ran out of RAM/disk; the cluster
|
||
> now lives on **Proxmox VE**. Anything mentioning `virsh`, `qemu:///system`, `tofu`+`dmacvicar/libvirt`,
|
||
> or `10.73.0.x` for the cluster is historical — including [`cluster.sh`](cluster.sh) (retired) and
|
||
> [`PHASE1-SCOPE.md`](PHASE1-SCOPE.md) (historical scoping doc).
|
||
|
||
## The cluster
|
||
|
||
| | |
|
||
|---|---|
|
||
| Substrate | Proxmox VE host, node `proxmox`, disks on the `Storage` ZFS datastore, bridge `vmbr0` |
|
||
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 8 vCPU, 24 GiB, Debian 13 (trixie) |
|
||
| k3s | **v1.36.4+k3s1**, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik |
|
||
| CNI | Cilium 1.20.1 (kube-proxy replacement, L2 announcements, egress gateway) |
|
||
| kube-API | kube-vip VIP **`192.168.1.49:6443`**, tls-san'd on the serving cert |
|
||
| Ingress VIP | **`192.168.1.53`** (shared Gateway). Pool `192.168.1.53–.60` for LB VIPs; UniFi holds `.54` |
|
||
| Storage | Longhorn 1.12.1 — classes `longhorn` (default), `longhorn-static`, `longhorn-vm` (RWX/Block, migratable), `longhorn-scratch`. `bulk` (NFS CSI) is a **non-provisioning stub** pending OQ-2 |
|
||
| Nested virt | `cpu_type = host` → real KVM in-guest; KubeVirt + CDI run on it |
|
||
|
||
Everything above is provisioned by IaC: [`tofu/`](tofu/) creates the VMs,
|
||
[`cloud-init/`](cloud-init/) baselines first boot, [`ansible/`](ansible/) installs k3s, Cilium,
|
||
kube-vip and the VPN egress gateway. Flux then reconciles the whole platform from `clusters/dev`.
|
||
|
||
## Network path from the workstation (there is a NAT in the way)
|
||
|
||
The workstation is **not** on the cluster LAN. A pfSense NAT bridges the two:
|
||
|
||
| Purpose | Workstation-side target | Forwards to |
|
||
|---|---|---|
|
||
| kube-API | `https://192.168.2.50:6443` | kube-vip VIP `192.168.1.49:6443` |
|
||
| Node SSH (Ansible only) | `192.168.2.102:2250 / 2251 / 2252` | `192.168.1.50 / .51 / .52` port 22 |
|
||
| Proxmox API | `https://192.168.2.102:8006/` | the Proxmox host |
|
||
|
||
TLS through the NAT validates because the kubeconfig sets `tls-server-name: 192.168.1.49` — the
|
||
VIP is in the serving cert's SAN list, so the NAT address does not have to be.
|
||
|
||
There is **no SSH tunnel** any more; the old tunnel was retired when kube-vip + the NAT landed.
|
||
|
||
## Access
|
||
|
||
**kubectl — Keycloak SSO (normal path).** `~/.kube/config` context **`k3s-oidc`** authenticates
|
||
against the `homelab` realm via `kubelogin` (auth-code + PKCE, public client, no secret in the
|
||
kubeconfig). Group membership maps to RBAC by `platform/keycloak/k8s-oidc-rbac.yaml`:
|
||
|
||
```
|
||
kubectl config use-context k3s-oidc
|
||
kubectl auth whoami # -> oidc:<you>, groups [oidc:platform-admins ...]
|
||
```
|
||
|
||
`oidc:platform-admins` → `cluster-admin`, `oidc:viewers` → `view`. First use opens a browser, so
|
||
this path needs an interactive desktop.
|
||
|
||
**kubectl — cert kubeconfig (break-glass).** The k3s admin cert kubeconfig from
|
||
`/etc/rancher/k3s/k3s.yaml` on `k3s-dev-1` (rewrite the server to `https://192.168.2.50:6443` and
|
||
add `tls-server-name: 192.168.1.49`). Fully independent of Keycloak — **use this when OIDC, the
|
||
gateway, or Keycloak itself is down.** Headless contexts (CI jobs, containers, agents) also need
|
||
this, since they cannot complete a browser login.
|
||
|
||
**Node SSH.** `ssh -i ~/.ssh/home_network -p 2250 [email protected]` (user `k3s`, NOPASSWD sudo).
|
||
Ports 2251/2252 for nodes 2/3.
|
||
|
||
## Toolchain — use the dev container, not the host
|
||
|
||
Every pinned tool (tofu, ansible, sops, age, flux, kubectl, helm **and** the full CI gate
|
||
toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image:
|
||
|
||
```bash
|
||
infra/devcontainer/dev.sh build # once
|
||
infra/devcontainer/dev.sh run make -C ci gate # exactly what CI runs
|
||
infra/devcontainer/dev.sh # interactive shell at /work
|
||
|
||
infra/devcontainer/dev.sh run kubectl get nodes # DEV (the default)
|
||
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # PROD
|
||
```
|
||
|
||
Unlike the libvirt era, **tofu and ansible now run fine inside the container** — the
|
||
`bpg/proxmox` provider talks to the Proxmox API over the network and needs no host libvirt socket,
|
||
which is what broke the old `dmacvicar/libvirt` provider in a container.
|
||
|
||
`dev.sh` mounts the repo at `/work`, `~/.ssh` (read-only), `~/.config/sops` (the age keyring —
|
||
**persisted on the host, or you lose access to every SOPS secret**), the ansible-vault passphrase
|
||
file, and the sibling spec repo read-only at `/home-network-k3s` so the spec-immutability
|
||
check runs instead of silently skipping.
|
||
|
||
> **Closed (was a known gap): `dev.sh` can now target either cluster.** It used to hard-wire
|
||
> `KUBECONFIG` to `dev.kubeconfig`, leaving an undocumented — and misleadingly named —
|
||
> `DEV_KUBECONFIG=<prod path>` as the only route to prod. Select with **`CLUSTER=dev|prod`**;
|
||
> **dev remains the default**, so every existing invocation means what it always meant. Inside the
|
||
> container `KUBECONFIG` is a merged config carrying both clusters as contexts **`k3s-dev`** and
|
||
> **`k3s-prod`** (never `default`), regenerated each run by
|
||
> [`infra/devcontainer/kubeconfig.sh`](devcontainer/kubeconfig.sh) into the gitignored
|
||
> `infra/ansible/.cache/clusters.kubeconfig`. Naming matters: k3s writes *both* source kubeconfigs
|
||
> with context, cluster and user all called `default`, and kubectl's merge keeps the first
|
||
> duplicate — so a plain `KUBECONFIG=dev:prod` would silently resolve `default` to **dev** while you
|
||
> believed you were on prod. Requesting a cluster whose kubeconfig is absent is a hard error, never
|
||
> a fallback. `DEV_KUBECONFIG` still works and still wins, but is **superseded** — it is opaque
|
||
> about its target and yields an ambiguous `default` context. `~/.kube` is still not mounted and the
|
||
> OIDC context still cannot work in a container (no browser); use the break-glass cert kubeconfigs
|
||
> above. Details: [`devcontainer/README.md`](devcontainer/README.md).
|
||
|
||
## Off-repo state you must not lose
|
||
|
||
| What | Where |
|
||
|---|---|
|
||
| SOPS **age private key** | `~/.config/sops/age/keys.txt` — losing it locks you out of every encrypted secret |
|
||
| Ansible vault passphrase | `~/.config/home-k3s/ansible-vault-pass` |
|
||
| Node SSH key | `~/.ssh/home_network` |
|
||
| Proxmox API token | operator env only (`PROXMOX_VE_API_TOKEN`) — never a tofu var, never in state |
|
||
| tofu state | `infra/tofu/terraform.tfstate` (gitignored) — **absent on a fresh workstation**; a `tofu apply` without it would try to recreate the live nodes |
|
||
| Generated SSO secrets | `~/.config/home-k3s/keycloak-sso-secrets.txt` |
|
||
|
||
## Health check
|
||
|
||
```bash
|
||
kubectl get nodes # 3/3 Ready
|
||
flux get kustomizations # flux-system, platform, platform-config,
|
||
# apps-wave-0/1/2, apps-vms — all Ready
|
||
kubectl get helmreleases -A # all Ready
|
||
kubectl get cluster -A # CNPG: keycloak-db, nextcloud-db, semaphore-db
|
||
kubectl get vmi -A # unifi Running
|
||
```
|
||
|
||
## Flux's git source is inside the cluster — mind the loop
|
||
|
||
`GitRepository/flux-system` points at **`https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`**:
|
||
the **in-cluster** Gitea, which is itself an app Flux deploys in `apps/wave-0`. Resolution goes
|
||
through split-horizon CoreDNS → Gateway VIP `192.168.1.53`, so a Flux reconcile depends on Gitea,
|
||
the Gateway, CoreDNS and the wildcard cert all being up.
|
||
|
||
In practice this is stable — Gitea's Deployment and PVC survive a reboot and come back without
|
||
Flux's help, so a Proxmox host reboot self-heals (VMs auto-start, cluster reconverges). The loop
|
||
only bites if Gitea is **lost** rather than restarted. The recovery path is the host bootstrap
|
||
Gitea (`bash bootstrap/gitea/bootstrap-gitea.sh`) plus repointing the GitRepository at it — that
|
||
is what the `bootstrap/` tree exists for, and it is the reason the bootstrap Gitea is kept.
|
||
|
||
**Push target.** `origin` is that same in-cluster Gitea, so `git push origin main` *is* the deploy.
|
||
There is no separate "apply" step and no GitHub remote.
|