diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index fed53d1..a9d846a 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3654,3 +3654,51 @@ only if that path proves unreliable, not on principle. Consequence for the roadmap: the DR drill is unblocked and Phase 7's remaining exit criterion is simply to run it on prod and record the proof. + +--- + +## 2026-09-14 — LAN split-horizon is documented but NOT implemented; pfSense strips the answers + +**Symptom.** The operator could not reach `sonarr.home.keithjackson.dev` from the workstation. + +**First cause (the obvious one).** `/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf` +still carried the pre-migration override: + + local=/home.keithjackson.dev/ + address=/.home.keithjackson.dev/192.168.2.50 + +`192.168.2.50` was the **dev** cluster's pfSense NAT address. Dev was decommissioned 2026-09-12 and +prod moved onto the apex, so every name in the zone resolved to a dead host — and `local=` stops +dnsmasq forwarding the zone, so there was no fallback. A sibling file pinned the obsolete +`prod.home.keithjackson.dev` naming. + +**Second cause — and why deleting the override would NOT have fixed it.** I was about to recommend +simply removing both files, on the grounds that public DNS already answers correctly (it does: +Cloudflare returns `192.168.2.20` for the apex and every HTTP name, `.21` for unifi, `.22` for +crafty-mc). Checking the upstream first showed that is not enough: **pfSense at `192.168.2.1` +returns `NOERROR` with `ANSWER: 0`** for `home.keithjackson.dev`, `sonarr.home.keithjackson.dev` and +even the bare `keithjackson.dev`, while resolving `cloudflare.com` fine and correctly returning the +zone's Cloudflare NS records. That is the signature of **DNS-rebind / private-address filtering**: +the resolver discards RFC1918 answers for public domains, and this zone legitimately points at +`192.168.2.x`. With the override removed, the workstation would have had no source of truth at all. + +**The real gap.** `platform/coredns/coredns-custom.yaml` is explicit that its split-horizon is for +pods — *"External/LAN clients keep their own answers (PiHole split-horizon, docs/09)"*. But **Pi-hole +serves no DNS**: its Service is `ClusterIP` with ports 80 and 4180 only, no port 53 and no +LoadBalancer. So the LAN half of split-horizon referenced in the design and in README's Phase 2 row +does not exist. Each LAN client is silently relying on a hand-maintained dnsmasq file, which is +exactly why this broke — and why it will break again on the next address change. + +**Done now:** the workstation override was corrected rather than deleted — wildcard to the Gateway +VIP `192.168.2.20`, with the two L4 services overriding it (dnsmasq applies the most specific match), +all three verified against public DNS. Requires the operator's sudo; the file was staged for them. + +**Proper fixes, none applied (operator's call):** +1. **pfSense domain override / `private-domain: "keithjackson.dev"`** on the DNS Resolver — one + change, fixes every LAN client, and removes the need for per-client files. Preferred. +2. **Expose Pi-hole's DNS to the LAN** (port 53 via a LoadBalancer IP) and make it authoritative for + the zone, which is what the design already claims. Bigger, but it is the documented architecture. +3. Keep per-client dnsmasq overrides — works, but it is N hand-maintained copies of one fact, and + this incident is what that costs. + +Until 1 or 2 lands, README's Phase 2 "split-horizon CoreDNS" should be read as **in-cluster only**.