home-network-k3s-impl/decisions/2026-10-18-pihole-local-ttl.md

date, branch, title

date	branch	title
2026-10-18	fix/pihole-local-ttl	The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes

What changed

The operator reported DNS "flapping between the cluster ingress and the router (public IP)". PiHole was not at fault — both replicas answer correctly and consistently. The bug is an asymmetry in cacheability that we created.

Measured on prod 2026-10-18:

dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143

dnsmasq answers address= entries with TTL 0 unless told otherwise. The SOHO router forwards LAN queries to PiHole and caches the result — the decrementing TTL above is the proof it caches rather than forwarding live. So:

• the correct answer is uncacheable (TTL 0): the router must re-query PiHole on every lookup, which is maximum exposure to any momentary gap;
• the wrong answer is cacheable for 300s: the instant the router falls back to its upstream (standard forwarder behaviour when the forward target does not answer) it latches the public WAN address and serves it to every LAN client for five minutes.

So one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP — then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed. That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip.

Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines in catalog/pihole.yaml, so the private answer is cacheable and a short gap is invisible to clients. apps/wave-0/pihole.yaml regenerated with the canonical render command.

What I verified

• PiHole is NOT the fault. Both replicas, queried directly by pod IP, answer 192.168.2.20; the VIP returned it 6/6; this workstation 8/8. DHCP hands out only 192.168.2.23.
• The router caches rather than forwards live — three successive queries returned TTL 153, 148, 143 (decrementing), which a pure forwarder could not do.
• The TTL asymmetry, as quoted above: 0 from PiHole, 300 from public DNS.
• The router does answer DNS and returns the public chain (wan.home.keithjackson.dev → the WAN address), so a fallback always has a wrong answer available.
• local-ttl is the right directive: dnsmasq applies it to replies made from configuration, /etc/hosts and DHCP leases, which covers all five address= names with one directive. It is a standalone option, not a sub-field of address=, so prepending it to the ;-separated FTLCONF_misc_dnsmasq_lines list is syntactically correct.
• The render is clean and scoped: catalog/render.py --all catalog -o apps --wave-layout --no-image-automation (the canonical command ci/render-golden.sh asserts against, and deliberately not --emit-waves) changed exactly two files — catalog/pihole.yaml and apps/wave-0/pihole.yaml — and the rendered diff is the single local-ttl=60; prefix.

What I did NOT verify

• UNAPPLIED. Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL, nor the router caching the private answer, nor the flap stopping. Verify after merge with dig home.keithjackson.dev @192.168.2.23 (expect TTL=60, not 0) and then the same query at the router (expect 192.168.2.20 with a decrementing TTL, not wan.…).
• I did not confirm that PiHole/FTL accepts local-ttl via FTLCONF_misc_dnsmasq_lines. The directive is valid dnsmasq and the field is a passthrough, but catalog/pihole.yaml's own header records that FTL has previously written a bogus line into dnsmasq.conf and reported "0 invalid" — so a malformed passthrough fails silently. If the TTL is still 0 after merge, that is the cause, not this reasoning. Check the pod's /etc/pihole/dnsmasq.conf.
• 60s is chosen, not derived. Long enough for a pod restart and a lease re-election (both seconds); it does nothing for an outage longer than the TTL, where the cache expires and the public answer latches again. That is a real availability problem, not a caching one.
• This does not remove the trigger. 192.168.2.23 is served by two LoadBalancer Services (pihole-dns-tcp, pihole-dns-udp) with two independently-elected L2 announcement leases, so two nodes can ARP for the same address at once — they sat on different nodes earlier the same day (tcp on prod-2, udp on prod-1). The same pattern applies to crafty (.22) and unifi (.21). Merging each pair into one mixed-protocol Service would remove the class; the operator has parked that decision, and it is deliberately not in this change.
• Client-side bypasses are unexamined. A device with its own resolver, or a browser with DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help there. Only this one workstation's resolver chain was inspected.
• No alert fires if the split-horizon answer regresses to the public address. Nothing probes home.keithjackson.dev from a LAN vantage point and compares it against ${GATEWAY_VIP}, so the next recurrence would again be found by a human noticing. That gap is not closed here.