# Dev cluster — current state & reconnect guide

**Last verified: 2026-08-31.** Phase 1 is **complete and long since exited**; the cluster has been
running the full platform (phases 2–6) for some time. This doc is how a session — human or agent —
reconnects to it and re-establishes the toolchain.

> **New PM / agent taking over?** Read this doc (reconnect + toolchain), then the **latest
> `DECISIONS-LOG.md` entry (2026-08-31)** — it is the current-state + gotchas + prioritized **Next
> Steps** handoff — and the plans under `docs/plans/`. Start at Next Steps.

> **Phase 7 (prod) is OPEN as of 2026-08-31 — prep only.** The first prod node
> (`k3s-prod-1`, `192.168.2.10`) is on the bench. This document covers the **dev** cluster;
> for prod see [`PROD-PROVISIONING.md`](PROD-PROVISIONING.md) (T-7.1 checklist) and the
> Phase-7 entry in `DECISIONS-LOG.md`. Nothing is installed on prod hardware until the
> R-ENV-02 tokenization blockers close and the gate is green.

> **History note.** The cluster was originally 3 libvirt/KVM VMs on the workstation
> (`10.73.0.0/24` NAT). That was **retired** after the workstation ran out of RAM/disk; the cluster
> now lives on **Proxmox VE**. Anything mentioning `virsh`, `qemu:///system`, `tofu`+`dmacvicar/libvirt`,
> or `10.73.0.x` for the cluster is historical — including [`cluster.sh`](cluster.sh) (retired) and
> [`PHASE1-SCOPE.md`](PHASE1-SCOPE.md) (historical scoping doc).

## The cluster

| | |
|---|---|
| Substrate | Proxmox VE host, node `proxmox`, disks on the `Storage` ZFS datastore, bridge `vmbr0` |
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 8 vCPU, 24 GiB, Debian 13 (trixie) |
| k3s | **v1.36.4+k3s1**, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik |
| CNI | Cilium 1.20.1 (kube-proxy replacement, L2 announcements, egress gateway) |
| kube-API | kube-vip VIP **`192.168.1.49:6443`**, tls-san'd on the serving cert |
| Ingress VIP | **`192.168.1.53`** (shared Gateway). Pool `192.168.1.53–.60` for LB VIPs; UniFi holds `.54` |
| Storage | Longhorn 1.12.1 — classes `longhorn` (default), `longhorn-static`, `longhorn-vm` (RWX/Block, migratable), `longhorn-scratch`. `bulk` (NFS CSI) is a **non-provisioning stub** pending OQ-2 |
| Nested virt | `cpu_type = host` → real KVM in-guest; KubeVirt + CDI run on it |

Everything above is provisioned by IaC: [`tofu/`](tofu/) creates the VMs,
[`cloud-init/`](cloud-init/) baselines first boot, [`ansible/`](ansible/) installs k3s, Cilium,
kube-vip and the VPN egress gateway. Flux then reconciles the whole platform from `clusters/dev`.

## Network path from the workstation (there is a NAT in the way)

The workstation is **not** on the cluster LAN. A pfSense NAT bridges the two:

| Purpose | Workstation-side target | Forwards to |
|---|---|---|
| kube-API | `https://192.168.2.50:6443` | kube-vip VIP `192.168.1.49:6443` |
| Node SSH (Ansible only) | `192.168.2.102:2250 / 2251 / 2252` | `192.168.1.50 / .51 / .52` port 22 |
| Proxmox API | `https://192.168.2.102:8006/` | the Proxmox host |

TLS through the NAT validates because the kubeconfig sets `tls-server-name: 192.168.1.49` — the
VIP is in the serving cert's SAN list, so the NAT address does not have to be.

There is **no SSH tunnel** any more; the old tunnel was retired when kube-vip + the NAT landed.

## Access

**kubectl — Keycloak SSO (normal path).** `~/.kube/config` context **`k3s-oidc`** authenticates
against the `homelab` realm via `kubelogin` (auth-code + PKCE, public client, no secret in the
kubeconfig). Group membership maps to RBAC by `platform/keycloak/k8s-oidc-rbac.yaml`:

```
kubectl config use-context k3s-oidc
kubectl auth whoami     # -> oidc:<you>, groups [oidc:platform-admins ...]
```

`oidc:platform-admins` → `cluster-admin`, `oidc:viewers` → `view`. First use opens a browser, so
this path needs an interactive desktop.

**kubectl — cert kubeconfig (break-glass).** The k3s admin cert kubeconfig from
`/etc/rancher/k3s/k3s.yaml` on `k3s-dev-1` (rewrite the server to `https://192.168.2.50:6443` and
add `tls-server-name: 192.168.1.49`). Fully independent of Keycloak — **use this when OIDC, the
gateway, or Keycloak itself is down.** Headless contexts (CI jobs, containers, agents) also need
this, since they cannot complete a browser login.

**Node SSH.** `ssh -i ~/.ssh/home_network -p 2250 k3s@192.168.2.102` (user `k3s`, NOPASSWD sudo).
Ports 2251/2252 for nodes 2/3.

## Toolchain — use the dev container, not the host

Every pinned tool (tofu, ansible, sops, age, flux, kubectl, helm **and** the full CI gate
toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image:

```bash
infra/devcontainer/dev.sh build                    # once
infra/devcontainer/dev.sh run make -C ci gate      # exactly what CI runs
infra/devcontainer/dev.sh                          # interactive shell at /work

infra/devcontainer/dev.sh run kubectl get nodes                # DEV (the default)
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes   # PROD
```

Unlike the libvirt era, **tofu and ansible now run fine inside the container** — the
`bpg/proxmox` provider talks to the Proxmox API over the network and needs no host libvirt socket,
which is what broke the old `dmacvicar/libvirt` provider in a container.

`dev.sh` mounts the repo at `/work`, `~/.ssh` (read-only), `~/.config/sops` (the age keyring —
**persisted on the host, or you lose access to every SOPS secret**), the ansible-vault passphrase
file, and the sibling spec repo read-only at `/home-network-k3s` so the spec-immutability
check runs instead of silently skipping.

> **Closed (was a known gap): `dev.sh` can now target either cluster.** It used to hard-wire
> `KUBECONFIG` to `dev.kubeconfig`, leaving an undocumented — and misleadingly named —
> `DEV_KUBECONFIG=<prod path>` as the only route to prod. Select with **`CLUSTER=dev|prod`**;
> **dev remains the default**, so every existing invocation means what it always meant. Inside the
> container `KUBECONFIG` is a merged config carrying both clusters as contexts **`k3s-dev`** and
> **`k3s-prod`** (never `default`), regenerated each run by
> [`infra/devcontainer/kubeconfig.sh`](devcontainer/kubeconfig.sh) into the gitignored
> `infra/ansible/.cache/clusters.kubeconfig`. Naming matters: k3s writes *both* source kubeconfigs
> with context, cluster and user all called `default`, and kubectl's merge keeps the first
> duplicate — so a plain `KUBECONFIG=dev:prod` would silently resolve `default` to **dev** while you
> believed you were on prod. Requesting a cluster whose kubeconfig is absent is a hard error, never
> a fallback. `DEV_KUBECONFIG` still works and still wins, but is **superseded** — it is opaque
> about its target and yields an ambiguous `default` context. `~/.kube` is still not mounted and the
> OIDC context still cannot work in a container (no browser); use the break-glass cert kubeconfigs
> above. Details: [`devcontainer/README.md`](devcontainer/README.md).

## Off-repo state you must not lose

| What | Where |
|---|---|
| SOPS **age private key** | `~/.config/sops/age/keys.txt` — losing it locks you out of every encrypted secret |
| Ansible vault passphrase | `~/.config/home-k3s/ansible-vault-pass` |
| Node SSH key | `~/.ssh/home_network` |
| Proxmox API token | operator env only (`PROXMOX_VE_API_TOKEN`) — never a tofu var, never in state |
| tofu state | `infra/tofu/terraform.tfstate` (gitignored) — **absent on a fresh workstation**; a `tofu apply` without it would try to recreate the live nodes |
| Generated SSO secrets | `~/.config/home-k3s/keycloak-sso-secrets.txt` |

## Health check

```bash
kubectl get nodes                                  # 3/3 Ready
flux get kustomizations                            # flux-system, platform, platform-config,
                                                   # apps-wave-0/1/2, apps-vms — all Ready
kubectl get helmreleases -A                        # all Ready
kubectl get cluster -A                             # CNPG: keycloak-db, nextcloud-db, semaphore-db
kubectl get vmi -A                                 # unifi Running
```

## Flux's git source is inside the cluster — mind the loop

`GitRepository/flux-system` points at **`https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`**:
the **in-cluster** Gitea, which is itself an app Flux deploys in `apps/wave-0`. Resolution goes
through split-horizon CoreDNS → Gateway VIP `192.168.1.53`, so a Flux reconcile depends on Gitea,
the Gateway, CoreDNS and the wildcard cert all being up.

In practice this is stable — Gitea's Deployment and PVC survive a reboot and come back without
Flux's help, so a Proxmox host reboot self-heals (VMs auto-start, cluster reconverges). The loop
only bites if Gitea is **lost** rather than restarted. The recovery path is the host bootstrap
Gitea (`bash bootstrap/gitea/bootstrap-gitea.sh`) plus repointing the GitRepository at it — that
is what the `bootstrap/` tree exists for, and it is the reason the bootstrap Gitea is kept.

**Push target.** `origin` is that same in-cluster Gitea, so `git push origin main` *is* the deploy.
There is no separate "apply" step and no GitHub remote.