From 8891df07b0804806c69c14b65cf4860d111e3019 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 21:31:03 -0500 Subject: [PATCH] docs: LAN split-horizon DNS is documented but unimplemented; pfSense strips the answers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The workstation could not reach sonarr.home.keithjackson.dev. The stale dnsmasq override pinning the zone to dev's dead 192.168.2.50 was the obvious cause, but removing it — which is what I was about to recommend — would not have fixed it: pfSense returns NOERROR/ANSWER:0 for this zone while resolving cloudflare.com fine, the signature of private-address rebind filtering, since the zone legitimately points at RFC1918 space. The underlying gap: coredns-custom.yaml defers LAN clients to 'PiHole split-horizon', but Pi-hole's Service is ClusterIP with ports 80/4180 — it serves no DNS at all. So the LAN half of split-horizon does not exist and each client silently depends on a hand-maintained file. Records the corrected override (staged for the operator's sudo) and the three proper fixes, none applied. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index fed53d1..a9d846a 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3654,3 +3654,51 @@ only if that path proves unreliable, not on principle. Consequence for the roadmap: the DR drill is unblocked and Phase 7's remaining exit criterion is simply to run it on prod and record the proof. + +--- + +## 2026-09-14 — LAN split-horizon is documented but NOT implemented; pfSense strips the answers + +**Symptom.** The operator could not reach `sonarr.home.keithjackson.dev` from the workstation. + +**First cause (the obvious one).** `/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf` +still carried the pre-migration override: + + local=/home.keithjackson.dev/ + address=/.home.keithjackson.dev/192.168.2.50 + +`192.168.2.50` was the **dev** cluster's pfSense NAT address. Dev was decommissioned 2026-09-12 and +prod moved onto the apex, so every name in the zone resolved to a dead host — and `local=` stops +dnsmasq forwarding the zone, so there was no fallback. A sibling file pinned the obsolete +`prod.home.keithjackson.dev` naming. + +**Second cause — and why deleting the override would NOT have fixed it.** I was about to recommend +simply removing both files, on the grounds that public DNS already answers correctly (it does: +Cloudflare returns `192.168.2.20` for the apex and every HTTP name, `.21` for unifi, `.22` for +crafty-mc). Checking the upstream first showed that is not enough: **pfSense at `192.168.2.1` +returns `NOERROR` with `ANSWER: 0`** for `home.keithjackson.dev`, `sonarr.home.keithjackson.dev` and +even the bare `keithjackson.dev`, while resolving `cloudflare.com` fine and correctly returning the +zone's Cloudflare NS records. That is the signature of **DNS-rebind / private-address filtering**: +the resolver discards RFC1918 answers for public domains, and this zone legitimately points at +`192.168.2.x`. With the override removed, the workstation would have had no source of truth at all. + +**The real gap.** `platform/coredns/coredns-custom.yaml` is explicit that its split-horizon is for +pods — *"External/LAN clients keep their own answers (PiHole split-horizon, docs/09)"*. But **Pi-hole +serves no DNS**: its Service is `ClusterIP` with ports 80 and 4180 only, no port 53 and no +LoadBalancer. So the LAN half of split-horizon referenced in the design and in README's Phase 2 row +does not exist. Each LAN client is silently relying on a hand-maintained dnsmasq file, which is +exactly why this broke — and why it will break again on the next address change. + +**Done now:** the workstation override was corrected rather than deleted — wildcard to the Gateway +VIP `192.168.2.20`, with the two L4 services overriding it (dnsmasq applies the most specific match), +all three verified against public DNS. Requires the operator's sudo; the file was staged for them. + +**Proper fixes, none applied (operator's call):** +1. **pfSense domain override / `private-domain: "keithjackson.dev"`** on the DNS Resolver — one + change, fixes every LAN client, and removes the need for per-client files. Preferred. +2. **Expose Pi-hole's DNS to the LAN** (port 53 via a LoadBalancer IP) and make it authoritative for + the zone, which is what the design already claims. Bigger, but it is the documented architecture. +3. Keep per-client dnsmasq overrides — works, but it is N hand-maintained copies of one fact, and + this incident is what that costs. + +Until 1 or 2 lands, README's Phase 2 "split-horizon CoreDNS" should be read as **in-cluster only**.