jakman85andClaude Opus 4.8
7a34590b19
Root cause: the k3s nodes shipped with a SINGLE upstream resolver, PiHole
(192.168.1.18), set by cloud-init/netplan. PiHole answers the local jackson.lan
zone instantly but its external/recursive resolution flaps — verified 0/25 for
public names (registry-1.docker.io, api.cloudflare.com, quay.io, ghcr.io,
google.com) while gitea.jackson.lan resolved 5/5. A single upstream is a hard
SPOF: both containerd image pulls ("lookup registry-1.docker.io: Try again")
and CoreDNS ("forward . /etc/resolv.conf" -> pods' external DNS, e.g.
external-dns -> Cloudflare) fail whenever PiHole recursion is down. CoreDNS
logs were flooded with "read udp ...->192.168.1.18:53: i/o timeout".
Fix (additive, no policy bypass): keep PiHole PRIMARY (preserves jackson.lan +
network ad-blocking) and ADD the pfSense gateway (192.168.1.1) as a FALLBACK,
which resolves the public zones reliably (verified). The resolver fails over to
the fallback when PiHole times out.
- node_baseline: netplan drop-in (99-k3s-dns.yaml) with both nameservers +
"Apply netplan" handler; new node_baseline_dns_* defaults; tagged `dns`.
Applied to all 3 nodes via playbooks/node-baseline.yml --tags dns.
- tofu: initialization.dns.servers now lists [dns_server, network_gateway] so a
fresh cloud-init/netplan build stays in parity with the drop-in.
Verified: all external names resolve cluster-wide (10/10, zero timeouts) from
multiple pods after a CoreDNS restart; CoreDNS i/o-timeout floods stopped;
split-horizon (keycloak.home.keithjackson.dev -> Gateway VIP) intact.
NOTE: a SEPARATE, pre-existing WAN-egress outage (ICMP + TCP 443 to the internet
time out from every node while the LAN/pfSense resolver work) still blocks image
pulls, external-dns->Cloudflare, and Flux remote bases. That is a pfSense
NAT/firewall/WAN problem outside this repo, flagged for PM follow-up.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
infra/ — node provisioning and the toolchain
OpenTofu (Proxmox VE) + cloud-init + Ansible bring up the 3-node k3s/Cilium dev cluster.
Spec: ../home-network-k3s/docs/02-environments-and-provisioning.md.
| Path | What |
|---|---|
tofu/ |
The "Create" stage: 3 cloud-init'd VMs on the Proxmox box (bpg/proxmox provider) |
cloud-init/ |
First-boot baseline shipped as a Proxmox user-data snippet |
ansible/ |
The "Baseline"/"Configure" stages: k3s HA, Cilium, kube-vip, the VPN egress gateway |
devcontainer/ |
The pinned toolchain (tofu, ansible, sops, age, flux, kubectl, helm and the whole CI gate toolchain) |
cluster.sh |
RETIRED — drove the old libvirt VMs; does not work against Proxmox |
PHASE1-STATUS.md |
Current state of the cluster and how to reconnect to it |
PHASE1-SCOPE.md |
Historical Phase-1 scoping doc (libvirt era — superseded) |
Where the cluster actually lives
The dev cluster is not on this workstation. It runs on a Proxmox VE host; the nodes sit on the Proxmox LAN and are reached across a pfSense NAT:
| Nodes | k3s-dev-1/2/3 at 192.168.1.50/.51/.52 — 6 vCPU, 16 GiB, Debian 12, k3s v1.31.5 |
| kube-API | kube-vip VIP 192.168.1.49:6443, reached from the workstation via NAT 192.168.2.50:6443 |
| Node SSH | pfSense NAT 192.168.2.102:2250/2251/2252 → node :22 (Ansible only) |
| Ingress VIP | 192.168.1.53 (Cilium L2, shared Gateway); .53–.60 reserved for LB VIPs |
The earlier libvirt/KVM cluster on 10.73.0.0/24 is retired. Any doc or command referencing
virsh, qemu:///system, or 10.73.0.x for the cluster is historical.