fix(dns): add resilient fallback resolver to k3s nodes (PiHole primary + pfSense fallback)
Root cause: the k3s nodes shipped with a SINGLE upstream resolver, PiHole
(192.168.1.18), set by cloud-init/netplan. PiHole answers the local jackson.lan
zone instantly but its external/recursive resolution flaps — verified 0/25 for
public names (registry-1.docker.io, api.cloudflare.com, quay.io, ghcr.io,
google.com) while gitea.jackson.lan resolved 5/5. A single upstream is a hard
SPOF: both containerd image pulls ("lookup registry-1.docker.io: Try again")
and CoreDNS ("forward . /etc/resolv.conf" -> pods' external DNS, e.g.
external-dns -> Cloudflare) fail whenever PiHole recursion is down. CoreDNS
logs were flooded with "read udp ...->192.168.1.18:53: i/o timeout".
Fix (additive, no policy bypass): keep PiHole PRIMARY (preserves jackson.lan +
network ad-blocking) and ADD the pfSense gateway (192.168.1.1) as a FALLBACK,
which resolves the public zones reliably (verified). The resolver fails over to
the fallback when PiHole times out.
- node_baseline: netplan drop-in (99-k3s-dns.yaml) with both nameservers +
"Apply netplan" handler; new node_baseline_dns_* defaults; tagged `dns`.
Applied to all 3 nodes via playbooks/node-baseline.yml --tags dns.
- tofu: initialization.dns.servers now lists [dns_server, network_gateway] so a
fresh cloud-init/netplan build stays in parity with the drop-in.
Verified: all external names resolve cluster-wide (10/10, zero timeouts) from
multiple pods after a CoreDNS restart; CoreDNS i/o-timeout floods stopped;
split-horizon (keycloak.home.keithjackson.dev -> Gateway VIP) intact.
NOTE: a SEPARATE, pre-existing WAN-egress outage (ICMP + TCP 443 to the internet
time out from every node while the LAN/pfSense resolver work) still blocks image
pulls, external-dns->Cloudflare, and Flux remote bases. That is a pfSense
NAT/firewall/WAN problem outside this repo, flagged for PM follow-up.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
1 parent
b5d65e9bc8
commit
7a34590b19
4 files changed
+62
-1
No files matched your search
+7
-1
@@ -148,8 +148,14 @@ resource "proxmox_virtual_environment_vm" "node" {
|
||||
}
|
||||
}
|
||||
|
||||
# DNS resilience: PiHole (var.dns_server) stays PRIMARY for the local jackson.lan zone +
|
||||
# network ad-blocking; the pfSense gateway (var.network_gateway) is an additive FALLBACK so
|
||||
# a fresh node's containerd pulls + CoreDNS survive PiHole external-recursion outages (a
|
||||
# single upstream was a hard SPOF — see the node_baseline "DNS resilience" task and
|
||||
# DECISIONS-LOG). Existing nodes get the same two servers from the node_baseline netplan
|
||||
# drop-in; this keeps cloud-init/netplan on rebuild in parity.
|
||||
dns {
|
||||
servers = [var.dns_server]
|
||||
servers = [var.dns_server, var.network_gateway]
|
||||
}
|
||||
|
||||
user_account {
|
||||
|
||||
Reference in new issue
Block a user