jakman85andClaude Opus 5
574088dfeb
The operator added the first production node (k3s-prod-1, 192.168.2.10 — HP EliteDesk 800 G5 Mini, i7-9700T/31GiB, Debian 13, VT-x + /dev/kvm) and opened roadmap Phase 7 for PREP WORK ONLY: no k3s install until the R-ENV-02 blockers close and the gate is green. Records the Phase-7 entry-gate position honestly: the operator has declared hardware ready, but "Phases 1-6 proven in dev/test" is NOT met (37/51 conformance; R-ENV-01/02/03 themselves uncovered). Proceeding is an explicit, accepted risk — not an oversight. Adds infra/PROD-PROVISIONING.md (T-7.1, design-only): firmware/BIOS, disk layout, OS, network, access, Ansible reuse, repo blockers, build sequence, Phase-7 exit criteria. Key findings recorded in DECISIONS-LOG: - R-ENV-02 is violated in the base — dev LAN addresses hardcoded into base manifests (cilium k8sServiceHost, the Cilium LB pool, crafty/UniFi LB IPs, 3 S3 backup endpoints). Prod is on 192.168.2.x, so prod would build WRONG today. - clusters/prod never sets GATEWAY_VIP -> would inherit the base default 10.73.0.240, an address on the RETIRED libvirt network. - No NOPASSWD sudo on the prod node -> all Ansible fails at its first become. - node_baseline device defaults are virtio (/dev/vdb,vdc) and absent on bare metal. - ESCALATION: agents/task-board.md is structurally unmaintainable — AGENTS.md designates it the one agent-editable surface, but ci/spec-immutability.sh fails the gate on ANY spec-repo dirt with no path exception. Explains the all-todo board after six phases. NOT worked around: exempting it would weaken an R-TEST-06 gate. Operator decision. Operator decisions: Phase 7 open (prep only); wait for all 3 nodes before cluster-init; check drive health before choosing the single-SSD disk layout. Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03. Task: T-7.1. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
7.9 KiB
Dev cluster — current state & reconnect guide
Last verified: 2026-08-31. Phase 1 is complete and long since exited; the cluster has been running the full platform (phases 2–6) for some time. This doc is how a session — human or agent — reconnects to it and re-establishes the toolchain.
New PM / agent taking over? Read this doc (reconnect + toolchain), then the latest
DECISIONS-LOG.mdentry (2026-08-31) — it is the current-state + gotchas + prioritized Next Steps handoff — and the plans underdocs/plans/. Start at Next Steps.
Phase 7 (prod) is OPEN as of 2026-08-31 — prep only. The first prod node (
k3s-prod-1,192.168.2.10) is on the bench. This document covers the dev cluster; for prod seePROD-PROVISIONING.md(T-7.1 checklist) and the Phase-7 entry inDECISIONS-LOG.md. Nothing is installed on prod hardware until the R-ENV-02 tokenization blockers close and the gate is green.
History note. The cluster was originally 3 libvirt/KVM VMs on the workstation (
10.73.0.0/24NAT). That was retired after the workstation ran out of RAM/disk; the cluster now lives on Proxmox VE. Anything mentioningvirsh,qemu:///system,tofu+dmacvicar/libvirt, or10.73.0.xfor the cluster is historical — includingcluster.sh(retired) andPHASE1-SCOPE.md(historical scoping doc).
The cluster
| Substrate | Proxmox VE host, node proxmox, disks on the Storage ZFS datastore, bridge vmbr0 |
| Nodes | k3s-dev-1/2/3 at 192.168.1.50 / .51 / .52 — 8 vCPU, 24 GiB, Debian 13 (trixie) |
| k3s | v1.36.4+k3s1, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik |
| CNI | Cilium 1.20.1 (kube-proxy replacement, L2 announcements, egress gateway) |
| kube-API | kube-vip VIP 192.168.1.49:6443, tls-san'd on the serving cert |
| Ingress VIP | 192.168.1.53 (shared Gateway). Pool 192.168.1.53–.60 for LB VIPs; UniFi holds .54 |
| Storage | Longhorn 1.12.1 — classes longhorn (default), longhorn-static, longhorn-vm (RWX/Block, migratable), longhorn-scratch. bulk (NFS CSI) is a non-provisioning stub pending OQ-2 |
| Nested virt | cpu_type = host → real KVM in-guest; KubeVirt + CDI run on it |
Everything above is provisioned by IaC: tofu/ creates the VMs,
cloud-init/ baselines first boot, ansible/ installs k3s, Cilium,
kube-vip and the VPN egress gateway. Flux then reconciles the whole platform from clusters/dev.
Network path from the workstation (there is a NAT in the way)
The workstation is not on the cluster LAN. A pfSense NAT bridges the two:
| Purpose | Workstation-side target | Forwards to |
|---|---|---|
| kube-API | https://192.168.2.50:6443 |
kube-vip VIP 192.168.1.49:6443 |
| Node SSH (Ansible only) | 192.168.2.102:2250 / 2251 / 2252 |
192.168.1.50 / .51 / .52 port 22 |
| Proxmox API | https://192.168.2.102:8006/ |
the Proxmox host |
TLS through the NAT validates because the kubeconfig sets tls-server-name: 192.168.1.49 — the
VIP is in the serving cert's SAN list, so the NAT address does not have to be.
There is no SSH tunnel any more; the old tunnel was retired when kube-vip + the NAT landed.
Access
kubectl — Keycloak SSO (normal path). ~/.kube/config context k3s-oidc authenticates
against the homelab realm via kubelogin (auth-code + PKCE, public client, no secret in the
kubeconfig). Group membership maps to RBAC by platform/keycloak/k8s-oidc-rbac.yaml:
kubectl config use-context k3s-oidc
kubectl auth whoami # -> oidc:<you>, groups [oidc:platform-admins ...]
oidc:platform-admins → cluster-admin, oidc:viewers → view. First use opens a browser, so
this path needs an interactive desktop.
kubectl — cert kubeconfig (break-glass). The k3s admin cert kubeconfig from
/etc/rancher/k3s/k3s.yaml on k3s-dev-1 (rewrite the server to https://192.168.2.50:6443 and
add tls-server-name: 192.168.1.49). Fully independent of Keycloak — use this when OIDC, the
gateway, or Keycloak itself is down. Headless contexts (CI jobs, containers, agents) also need
this, since they cannot complete a browser login.
Node SSH. ssh -i ~/.ssh/home_network -p 2250 [email protected] (user k3s, NOPASSWD sudo).
Ports 2251/2252 for nodes 2/3.
Toolchain — use the dev container, not the host
Every pinned tool (tofu, ansible, sops, age, flux, kubectl, helm and the full CI gate toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # exactly what CI runs
infra/devcontainer/dev.sh # interactive shell at /work
Unlike the libvirt era, tofu and ansible now run fine inside the container — the
bpg/proxmox provider talks to the Proxmox API over the network and needs no host libvirt socket,
which is what broke the old dmacvicar/libvirt provider in a container.
dev.sh mounts the repo at /work, ~/.ssh (read-only), ~/.config/sops (the age keyring —
persisted on the host, or you lose access to every SOPS secret), the ansible-vault passphrase
file, and the sibling spec repo read-only at /home-network-k3s so the spec-immutability
check runs instead of silently skipping.
Known gap.
dev.shdefaultsKUBECONFIGto/work/infra/ansible/.cache/dev.kubeconfig, a libvirt-era path that no longer exists, and does not mount~/.kube. In-containerkubectltherefore needs an explicitDEV_KUBECONFIG=<path to a break-glass cert kubeconfig>— the OIDC context cannot work there anyway (no browser). Not yet fixed.
Off-repo state you must not lose
| What | Where |
|---|---|
| SOPS age private key | ~/.config/sops/age/keys.txt — losing it locks you out of every encrypted secret |
| Ansible vault passphrase | ~/.config/home-k3s/ansible-vault-pass |
| Node SSH key | ~/.ssh/home_network |
| Proxmox API token | operator env only (PROXMOX_VE_API_TOKEN) — never a tofu var, never in state |
| tofu state | infra/tofu/terraform.tfstate (gitignored) — absent on a fresh workstation; a tofu apply without it would try to recreate the live nodes |
| Generated SSO secrets | ~/.config/home-k3s/keycloak-sso-secrets.txt |
Health check
kubectl get nodes # 3/3 Ready
flux get kustomizations # flux-system, platform, platform-config,
# apps-wave-0/1/2, apps-vms — all Ready
kubectl get helmreleases -A # all Ready
kubectl get cluster -A # CNPG: keycloak-db, nextcloud-db, semaphore-db
kubectl get vmi -A # unifi Running
Flux's git source is inside the cluster — mind the loop
GitRepository/flux-system points at https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git:
the in-cluster Gitea, which is itself an app Flux deploys in apps/wave-0. Resolution goes
through split-horizon CoreDNS → Gateway VIP 192.168.1.53, so a Flux reconcile depends on Gitea,
the Gateway, CoreDNS and the wildcard cert all being up.
In practice this is stable — Gitea's Deployment and PVC survive a reboot and come back without
Flux's help, so a Proxmox host reboot self-heals (VMs auto-start, cluster reconverges). The loop
only bites if Gitea is lost rather than restarted. The recovery path is the host bootstrap
Gitea (bash bootstrap/gitea/bootstrap-gitea.sh) plus repointing the GitRepository at it — that
is what the bootstrap/ tree exists for, and it is the reason the bootstrap Gitea is kept.
Push target. origin is that same in-cluster Gitea, so git push origin main is the deploy.
There is no separate "apply" step and no GitHub remote.