--- date: 2026-10-18 branch: fix/pihole-local-ttl title: "The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes" --- ## What changed The operator reported DNS "flapping between the cluster ingress and the router (public IP)". PiHole was not at fault — both replicas answer correctly and consistently. The bug is an **asymmetry in cacheability** that we created. Measured on prod 2026-10-18: ``` dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0 dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300 dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143 ``` dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The SOHO router forwards LAN queries to PiHole and **caches** the result — the decrementing TTL above is the proof it caches rather than forwarding live. So: * the **correct** answer is **uncacheable** (TTL 0): the router must re-query PiHole on *every* lookup, which is maximum exposure to any momentary gap; * the **wrong** answer is **cacheable for 300s**: the instant the router falls back to its upstream (standard forwarder behaviour when the forward target does not answer) it latches the public WAN address and serves it to **every LAN client** for five minutes. So **one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP** — then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed. That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip. Fix: `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines` in `catalog/pihole.yaml`, so the private answer is cacheable and a short gap is invisible to clients. `apps/wave-0/pihole.yaml` regenerated with the canonical render command. ## What I verified * **PiHole is NOT the fault.** Both replicas, queried directly by pod IP, answer `192.168.2.20`; the VIP returned it 6/6; this workstation 8/8. DHCP hands out only `192.168.2.23`. * **The router caches rather than forwards live** — three successive queries returned TTL 153, 148, 143 (decrementing), which a pure forwarder could not do. * **The TTL asymmetry**, as quoted above: 0 from PiHole, 300 from public DNS. * **The router does answer DNS and returns the public chain** (`wan.home.keithjackson.dev` → the WAN address), so a fallback always has a wrong answer available. * **`local-ttl` is the right directive**: dnsmasq applies it to replies made from configuration, `/etc/hosts` and DHCP leases, which covers all five `address=` names with one directive. It is a standalone option, not a sub-field of `address=`, so prepending it to the `;`-separated `FTLCONF_misc_dnsmasq_lines` list is syntactically correct. * **The render is clean and scoped**: `catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` (the canonical command `ci/render-golden.sh` asserts against, and deliberately **not** `--emit-waves`) changed exactly two files — `catalog/pihole.yaml` and `apps/wave-0/pihole.yaml` — and the rendered diff is the single `local-ttl=60;` prefix. ## What I did NOT verify * **UNAPPLIED.** Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL, nor the router caching the private answer, nor the flap stopping. **Verify after merge with `dig home.keithjackson.dev @192.168.2.23` (expect `TTL=60`, not 0) and then the same query at the router (expect `192.168.2.20` with a decrementing TTL, not `wan.…`).** * **I did not confirm that PiHole/FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header records that FTL has previously written a bogus line into `dnsmasq.conf` and reported "0 invalid" — so a malformed passthrough fails *silently*. If the TTL is still 0 after merge, that is the cause, not this reasoning. Check the pod's `/etc/pihole/dnsmasq.conf`. * **60s is chosen, not derived.** Long enough for a pod restart and a lease re-election (both seconds); it does **nothing** for an outage longer than the TTL, where the cache expires and the public answer latches again. That is a real availability problem, not a caching one. * **This does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services (`pihole-dns-tcp`, `pihole-dns-udp`) with **two independently-elected L2 announcement leases**, so two nodes can ARP for the same address at once — they sat on different nodes earlier the same day (tcp on prod-2, udp on prod-1). The same pattern applies to crafty (`.22`) and unifi (`.21`). Merging each pair into one mixed-protocol Service would remove the class; the operator has parked that decision, and it is deliberately **not** in this change. * **Client-side bypasses are unexamined.** A device with its own resolver, or a browser with DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help there. Only this one workstation's resolver chain was inspected. * **No alert fires if the split-horizon answer regresses to the public address.** Nothing probes `home.keithjackson.dev` from a LAN vantage point and compares it against `${GATEWAY_VIP}`, so the next recurrence would again be found by a human noticing. That gap is not closed here.