jakman85andClaude Opus 5
6d62aca0d2
OPERATOR CORRECTION (2026-09-05). I had been calling the resolver that shadows
prod "a pfSense override" throughout this repo. Both halves of that were wrong:
- It is the WORKSTATION'S OWN dnsmasq, not pfSense. The live resolver is the
one NetworkManager spawns (dns=dnsmasq, --conf-dir=/etc/NetworkManager/
dnsmasq.d); the standalone dnsmasq.service is INACTIVE, so /etc/dnsmasq.d is
dormant. The dev profile's `address=/.home.keithjackson.dev/192.168.2.50`
matches the domain AND EVERY SUBDOMAIN, so it also swallowed the whole new
prod.home.* tree and answered it with dev's address.
- The k3s prod cluster is NOT behind pfSense. It sits on the SOHO router
network 192.168.2.0/24, which is pfSense's WAN; pfSense CONTAINS the dev LAN
192.168.1.0/24. This workstation is on 192.168.2.0/24 too, which is why the
rules below use prod's real VIPs and need no NAT hop, unlike dev's .50.
New infra/workstation/dnsmasq/ — authored, not click-ops:
20-prod.home.keithjackson.dev.conf prod.home.* -> 192.168.2.20 (Gateway),
unifi -> .21, crafty-mc -> .22 (L4 VIPs)
install.sh installs + reloads; refuses non-root
dnsmasq resolves `address=` by LONGEST MATCH, so the prod rules win WITHOUT
editing or reordering the dev profile. PROVEN, not assumed: ran a throwaway
dnsmasq on 127.0.0.1:5354 loaded with both profiles —
prod.home / gitea.prod / keycloak.prod -> 192.168.2.20
unifi.prod -> 192.168.2.21 crafty-mc.prod -> 192.168.2.22
gitea.home / home.keithjackson.dev -> 192.168.2.50 (DEV, UNCHANGED)
`dnsmasq --test` on the new file: syntax OK.
install.sh pins the destination because getting it wrong is SILENT: a file in
/etc/dnsmasq.d exists, looks installed, and changes nothing. It also uses
`systemctl reload NetworkManager` rather than SIGHUP — dnsmasq does not re-read
its config on SIGHUP, so a HUP would look like it worked and not apply anything.
WHY THE MISLABEL MATTERED: the trap PARTIALLY works. Dev serves a
*.home.keithjackson.dev wildcard cert, which matches prod.home.keithjackson.dev
(one label) so the prod apex returns 200 over VALID TLS, while
keycloak.prod.home.keithjackson.dev (two labels) fails with a certificate-name
error. It reads as a broken prod cert; it is the wrong server answering. I made
exactly that misreading during verification.
Also recorded, because it changes what the apex flip IS: the SOHO router NATs
inbound 443 to the pfSense WAN (192.168.2.102) to serve the EXISTING NON-K3S
production environment, which must stay live until the switch is deliberately
made. So §10a sequence P is not just a k3s-side DNS/TLS change — it is the
legacy-prod cutover, and P as written does NOT cover repointing that NAT. The
two must be planned together or the apex resolves to k3s while inbound 443 still
lands on the legacy box. Phase C touches none of it: prod's Gateway is
192.168.2.20:443 (a different address) and its cert is DNS-01, needing no
inbound 443. And the MinIO timeout is now correctly described as expected
topology — crossing INTO pfSense from its WAN — not a misconfiguration.
VERIFIED: full CI gate PASS (spec-immutability, lint, l0, render-golden, l1).
Refs R-DNS-02, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
infra/ — node provisioning and the toolchain
OpenTofu (Proxmox VE) + cloud-init + Ansible bring up the 3-node k3s/Cilium dev cluster.
Spec: ../home-network-k3s/docs/02-environments-and-provisioning.md.
| Path | What |
|---|---|
tofu/ |
The "Create" stage: 3 cloud-init'd VMs on the Proxmox box (bpg/proxmox provider) |
cloud-init/ |
First-boot baseline shipped as a Proxmox user-data snippet |
ansible/ |
The "Baseline"/"Configure" stages: k3s HA, Cilium, kube-vip, the VPN egress gateway |
devcontainer/ |
The pinned toolchain (tofu, ansible, sops, age, flux, kubectl, helm and the whole CI gate toolchain) |
cluster.sh |
RETIRED — drove the old libvirt VMs; does not work against Proxmox |
PHASE1-STATUS.md |
Current state of the cluster and how to reconnect to it |
PHASE1-SCOPE.md |
Historical Phase-1 scoping doc (libvirt era — superseded) |
Where the cluster actually lives
The dev cluster is not on this workstation. It runs on a Proxmox VE host; the nodes sit on the Proxmox LAN and are reached across a pfSense NAT:
| Nodes | k3s-dev-1/2/3 at 192.168.1.50/.51/.52 — 6 vCPU, 24 GiB, Debian 13, k3s v1.36.4 |
| kube-API | kube-vip VIP 192.168.1.49:6443, reached from the workstation via NAT 192.168.2.50:6443 |
| Node SSH | pfSense NAT 192.168.2.102:2250/2251/2252 → node :22 (Ansible only) |
| Ingress VIP | 192.168.1.53 (Cilium L2, shared Gateway); .53–.60 reserved for LB VIPs |
The earlier libvirt/KVM cluster on 10.73.0.0/24 is retired. Any doc or command referencing
virsh, qemu:///system, or 10.73.0.x for the cluster is historical.