jakman85andClaude Opus 5
6b88670129
Two runbook corrections from executing step A of the dev migration. 1. P1 said "only NOW unsuspend prod", gated on D8 (dev's apex vacated). That silently assumed prod would be serving the bare apex. IT IS NOT. Prod runsc8c1d29, where BASE_DOMAIN=prod.home.keithjackson.dev and EXTERNALDNS_TXT_OWNER_ID=prod — verified live in prod's applied cluster-vars. Prod's names are a DISJOINT SUBTREE under a DISTINCT registry owner, so neither §10a incident mode applies: it cannot claim dev's `default`-owned records, and *.prod.home.keithjackson.dev is a DIFFERENT name set from dev's, so it has its own Let's Encrypt duplicate-cert bucket and races no _acme-challenge with dev. Prod can therefore be brought up IN FULL while dev still holds the apex — and should be, because it proves the whole stack before the flip, turning the cutover into a DNS/TLS change on a known-good cluster rather than a bring-up and a cutover simultaneously. D8 gates ONLY the later move of prod's BASE_DOMAIN to the apex. 2. D1 was executed via the REPOINT option, not suspend — repointing removes the dependency that causes the strand instead of merely deferring it. Dev now fetches from http://192.168.2.132:3000/... branch `dev-live`: IP-addressed, plain HTTP, dependent on NEITHER DNS NOR TLS, which is the property a DNS/TLS migration needs. A merge to `main` is now inert for dev. THE TRAP, recorded because it is not obvious: a bare copy of main@4809199 on the bootstrap Gitea would NOT have held. The root renders ./clusters/dev, and clusters/dev/kustomization.yaml CONTAINS the patch that sets GitRepository.spec.url — so Flux would have fetched from the bootstrap Gitea and then applied manifests that set the url straight back, reverting the repoint and re-arming the strand. The branch must declare its own source. dev-live (a4a3c2e) carries exactly one content change on4809199, proven by `kubectl kustomize clusters/dev` differing in EXACTLY TWO of 357 lines. NEW DEPENDENCY, and it is the weak link: 192.168.2.132 is a DHCP lease on the workstation's WIFI (wlp0s20f3, NetworkManager ipv4.method=auto — confirmed by inspection, not assumed). A lease change or the workstation going offline stalls dev's FETCHING; the cluster keeps serving and the fix is a kubectl patch. Hence: a static reservation is worth having, and the window should be kept short — the exit is D4. Pointing Flux at gitea.gitea.svc.cluster.local would drop the DNS, TLS and workstation dependencies at once and is the better long-term shape, but namespace `gitea` has default-deny plus a Cilium policy admitting only the Gateway, so it needs a new authored policy — deliberately, not mid-migration. Post-cutover verification: 13/13 Kustomizations Ready on dev-live@a4a3c2e, cluster-vars byte-identical to the D0 snapshot (BASE_DOMAIN still the apex), 18/18 Longhorn volumes attached+healthy, all 16 HTTPRoute hostnames unchanged, and the Cloudflare record set re-queried and diffed byte-for-byte against D0 — identical, 57 records, 36 TXT all owner=default. Refs R-ENV-02, R-GITOPS-01. Co-Authored-By: Claude Opus 5 <[email protected]>
infra/ — node provisioning and the toolchain
OpenTofu (Proxmox VE) + cloud-init + Ansible bring up the 3-node k3s/Cilium dev cluster.
Spec: ../home-network-k3s/docs/02-environments-and-provisioning.md.
| Path | What |
|---|---|
tofu/ |
The "Create" stage: 3 cloud-init'd VMs on the Proxmox box (bpg/proxmox provider) |
cloud-init/ |
First-boot baseline shipped as a Proxmox user-data snippet |
ansible/ |
The "Baseline"/"Configure" stages: k3s HA, Cilium, kube-vip, the VPN egress gateway |
devcontainer/ |
The pinned toolchain (tofu, ansible, sops, age, flux, kubectl, helm and the whole CI gate toolchain) |
cluster.sh |
RETIRED — drove the old libvirt VMs; does not work against Proxmox |
PHASE1-STATUS.md |
Current state of the cluster and how to reconnect to it |
PHASE1-SCOPE.md |
Historical Phase-1 scoping doc (libvirt era — superseded) |
Where the cluster actually lives
The dev cluster is not on this workstation. It runs on a Proxmox VE host; the nodes sit on the Proxmox LAN and are reached across a pfSense NAT:
| Nodes | k3s-dev-1/2/3 at 192.168.1.50/.51/.52 — 6 vCPU, 24 GiB, Debian 13, k3s v1.36.4 |
| kube-API | kube-vip VIP 192.168.1.49:6443, reached from the workstation via NAT 192.168.2.50:6443 |
| Node SSH | pfSense NAT 192.168.2.102:2250/2251/2252 → node :22 (Ansible only) |
| Ingress VIP | 192.168.1.53 (Cilium L2, shared Gateway); .53–.60 reserved for LB VIPs |
The earlier libvirt/KVM cluster on 10.73.0.0/24 is retired. Any doc or command referencing
virsh, qemu:///system, or 10.73.0.x for the cluster is historical.