jakman85andClaude Opus 4.8
7a34590b19
Root cause: the k3s nodes shipped with a SINGLE upstream resolver, PiHole
(192.168.1.18), set by cloud-init/netplan. PiHole answers the local jackson.lan
zone instantly but its external/recursive resolution flaps — verified 0/25 for
public names (registry-1.docker.io, api.cloudflare.com, quay.io, ghcr.io,
google.com) while gitea.jackson.lan resolved 5/5. A single upstream is a hard
SPOF: both containerd image pulls ("lookup registry-1.docker.io: Try again")
and CoreDNS ("forward . /etc/resolv.conf" -> pods' external DNS, e.g.
external-dns -> Cloudflare) fail whenever PiHole recursion is down. CoreDNS
logs were flooded with "read udp ...->192.168.1.18:53: i/o timeout".
Fix (additive, no policy bypass): keep PiHole PRIMARY (preserves jackson.lan +
network ad-blocking) and ADD the pfSense gateway (192.168.1.1) as a FALLBACK,
which resolves the public zones reliably (verified). The resolver fails over to
the fallback when PiHole times out.
- node_baseline: netplan drop-in (99-k3s-dns.yaml) with both nameservers +
"Apply netplan" handler; new node_baseline_dns_* defaults; tagged `dns`.
Applied to all 3 nodes via playbooks/node-baseline.yml --tags dns.
- tofu: initialization.dns.servers now lists [dns_server, network_gateway] so a
fresh cloud-init/netplan build stays in parity with the drop-in.
Verified: all external names resolve cluster-wide (10/10, zero timeouts) from
multiple pods after a CoreDNS restart; CoreDNS i/o-timeout floods stopped;
split-horizon (keycloak.home.keithjackson.dev -> Gateway VIP) intact.
NOTE: a SEPARATE, pre-existing WAN-egress outage (ICMP + TCP 443 to the internet
time out from every node while the LAN/pfSense resolver work) still blocks image
pulls, external-dns->Cloudflare, and Flux remote bases. That is a pfSense
NAT/firewall/WAN problem outside this repo, flagged for PM follow-up.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
27 lines
1.0 KiB
YAML
27 lines
1.0 KiB
YAML
---
|
|
# node_baseline — first-boot floor for k3s/Cilium/Longhorn, moved out of cloud-init
|
|
# (Proxmox provisioning is API-only). Runs before k3s (site.yml). Idempotent.
|
|
node_baseline_packages:
|
|
- qemu-guest-agent
|
|
- open-iscsi
|
|
- nfs-common
|
|
- curl
|
|
node_baseline_kernel_modules:
|
|
- overlay
|
|
- br_netfilter
|
|
- iscsi_tcp
|
|
- dm_crypt
|
|
- nfs
|
|
node_baseline_longhorn_device: /dev/vdb
|
|
node_baseline_longhorn_mount: /var/lib/longhorn
|
|
node_baseline_longhorn_fs_label: longhorn
|
|
|
|
# DNS resilience (see tasks/main.yml "DNS resilience"). PiHole stays PRIMARY (local
|
|
# jackson.lan zone + network ad-blocking); the pfSense gateway is an additive FALLBACK so
|
|
# external resolution (containerd image pulls, CoreDNS→pods) survives PiHole recursion
|
|
# outages. Keep these aligned with tofu variables dns_server / network_gateway.
|
|
node_baseline_primary_iface: eth0
|
|
node_baseline_dns_primary: 192.168.1.18 # PiHole — local zone + ad-block (PRIMARY)
|
|
node_baseline_dns_fallback: 192.168.1.1 # pfSense gateway — reliable external recursion
|
|
node_baseline_dns_search: jackson.lan
|