jakman85andClaude Opus 5
4f37fc375e
Prod is up and serving. 113 pods, 0 not Running/Completed. wildcard-home Certificate Ready (Let's Encrypt PROD, *.prod.home.keithjackson.dev). Gateway VIP 192.168.2.20, crafty L4 .22, unifi L4 .21. 36 DNS records published under prod.home.keithjackson.dev, every TXT registry entry owner=prod. End-to-end HTTPS with certificate validation (no -k), pinned to prod's gateway: apex 200, keycloak 302 (login), gitea 200, grafana 302 (SSO). DEV UNTOUCHED: all 57 pre-existing records byte-identical to the pre-standup baseline. velero remains suspended — S3_BACKUP_ENDPOINT is still the TBD placeholder, and resuming it would create a BackupStorageLocation that can never reach a bucket: a tier reporting healthy while backups silently do not exist. TWO BUGS IN MY OWN VERIFICATION, both of which would have made the safety check worthless. Recording them because a check you cannot trust is worse than none — it trains you to skip the one alarm that matters. 1. SNAP_DIR defaulted to $TMPDIR. Every `dev.sh run` is a FRESH container, so the DNS baseline was discarded between steps and `verify`'s before/after comparison — the check that detects damage to the other cluster's records — could never run across invocations. Now repo-relative (bind-mounted, so it persists on the host) and gitignored as a run artifact. 2. The first real comparison cried wolf: "RECORDS OUTSIDE prod.home CHANGED", listing pairs of IDENTICAL lines. Cause was locale-dependent sort collation — the host ordered `*.keithjackson.dev` differently from the container, so a diff of two identical sets reported changes. Proved it was noise by re-comparing under LC_ALL=C: all 57 records identical. dns_snapshot now strips CRs and sorts under LC_ALL=C so ordering is reproducible everywhere. ACTION REQUIRED (operator, pfSense): the local override is a wildcard on *.home.keithjackson.dev and now also swallows the new prod.home.* subtree, resolving EVERY prod hostname to 192.168.2.50 (dev). Cloudflare correctly returns 192.168.2.20. Prod is reachable from anywhere using public DNS, but NOT from behind pfSense. Narrow the override, or add one pointing prod.home.* at 192.168.2.20. That override is also the trap that produced a false healthy reading during the 2026-09-04 incident, and it is subtle because it PARTIALLY works: dev's *.home.keithjackson.dev wildcard matches prod.home.keithjackson.dev (one label) so the apex returns 200 over valid TLS, while keycloak.prod.home... (two labels) fails with a certificate-name error. I hit this myself during verification and mistook it for a broken prod cert. Never verify DNS or TLS here through a local resolver. Refs R-ENV-02, R-GITOPS-01, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]>
infra/ — node provisioning and the toolchain
OpenTofu (Proxmox VE) + cloud-init + Ansible bring up the 3-node k3s/Cilium dev cluster.
Spec: ../home-network-k3s/docs/02-environments-and-provisioning.md.
| Path | What |
|---|---|
tofu/ |
The "Create" stage: 3 cloud-init'd VMs on the Proxmox box (bpg/proxmox provider) |
cloud-init/ |
First-boot baseline shipped as a Proxmox user-data snippet |
ansible/ |
The "Baseline"/"Configure" stages: k3s HA, Cilium, kube-vip, the VPN egress gateway |
devcontainer/ |
The pinned toolchain (tofu, ansible, sops, age, flux, kubectl, helm and the whole CI gate toolchain) |
cluster.sh |
RETIRED — drove the old libvirt VMs; does not work against Proxmox |
PHASE1-STATUS.md |
Current state of the cluster and how to reconnect to it |
PHASE1-SCOPE.md |
Historical Phase-1 scoping doc (libvirt era — superseded) |
Where the cluster actually lives
The dev cluster is not on this workstation. It runs on a Proxmox VE host; the nodes sit on the Proxmox LAN and are reached across a pfSense NAT:
| Nodes | k3s-dev-1/2/3 at 192.168.1.50/.51/.52 — 6 vCPU, 24 GiB, Debian 13, k3s v1.36.4 |
| kube-API | kube-vip VIP 192.168.1.49:6443, reached from the workstation via NAT 192.168.2.50:6443 |
| Node SSH | pfSense NAT 192.168.2.102:2250/2251/2252 → node :22 (Ansible only) |
| Ingress VIP | 192.168.1.53 (Cilium L2, shared Gateway); .53–.60 reserved for LB VIPs |
The earlier libvirt/KVM cluster on 10.73.0.0/24 is retired. Any doc or command referencing
virsh, qemu:///system, or 10.73.0.x for the cluster is historical.