fix(pihole): make the split-horizon answer cacheable (local-ttl=60) #78

2026-10-18T08:05:36Z

gitea-admin commented

2026-10-18 08:05:36 +00:00

Scope note: this does not fix the operator's reported flap. Their workstation resolves via 8.8.8.8/1.1.1.1 and never consults PiHole, so this change is invisible to it. The Cloudflare-record flap is a separate issue (external-dns policy=sync with no RFC1918 target guard, plus two HTTPRoutes claiming the apex) and is not addressed here. This PR fixes a real but different defect affecting clients that forward to PiHole — principally the SOHO router, which does.

The defect: our authoritative answer is uncacheable, the wrong one caches for 5 minutes

Measured on prod 2026-10-18:

dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143

dnsmasq answers address= entries with TTL 0 unless told otherwise. The router forwards LAN queries to PiHole and caches the result — the decrementing TTL above is proof it caches rather than forwarding live. So:

the correct answer is uncacheable — the router must re-query PiHole on every lookup, maximum exposure to any momentary gap;
the wrong answer is cacheable for 300s — the instant the router falls back to upstream (standard forwarder behaviour when the target does not answer) it latches the public WAN address and serves it to every forwarding client for five minutes.

One sub-second PiHole gap becomes five minutes of wrong answers for those clients. Then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed. That asymmetry, not the gap, is what makes it visible.

Fix

local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies local-ttl to replies made from configuration, so one directive covers all five address= names.

60s is a deliberate floor: long enough to ride out a pod restart and an L2 lease re-election (both seconds), short enough that a genuine VIP change propagates inside a minute. These VIPs are pinned by lbipam.cilium.io/ips and effectively static, so the propagation cost is theoretical while the flap it prevents is measured.

apps/wave-0/pihole.yaml regenerated with the canonical command (--all catalog -o apps --wave-layout --no-image-automation, deliberately not --emit-waves); exactly two files changed and the rendered diff is the single local-ttl=60; prefix.

Not verified

UNAPPLIED. Verify after merge with dig home.keithjackson.dev @192.168.2.23 — expect TTL=60, not 0.
I did not confirm FTL accepts local-ttl via FTLCONF_misc_dnsmasq_lines. The directive is valid dnsmasq and the field is a passthrough, but catalog/pihole.yaml's own header records FTL previously writing a bogus line into dnsmasq.conf and reporting "0 invalid" — a malformed passthrough fails silently. If the TTL is still 0 after merge, check the pod's /etc/pihole/dnsmasq.conf; that is the cause, not this reasoning.
It does not remove the trigger. 192.168.2.23 is served by two LoadBalancer Services with two independently-elected L2 announcement leases, so two nodes can ARP for it simultaneously — they sat on different nodes earlier the same day. Same pattern on crafty (.22) and unifi (.21). Merging each pair into one mixed-protocol Service would remove the class; that decision is parked.
No help for an outage longer than the TTL — the cache expires and the public answer latches again. That is an availability problem, not a caching one.
No alert fires if the split-horizon answer regresses to the public address. Nothing probes the name from a LAN vantage point and compares against ${GATEWAY_VIP}.

Local gate: rc 3, zero failures; only gitea-settings and L2/L3 conformance did not run.

Full detail: decisions/2026-10-18-pihole-local-ttl.md

🤖 Generated with Claude Code

> **Scope note:** this does **not** fix the operator's reported flap. Their workstation resolves via 8.8.8.8/1.1.1.1 and never consults PiHole, so this change is invisible to it. The Cloudflare-record flap is a separate issue (external-dns `policy=sync` with no RFC1918 target guard, plus two HTTPRoutes claiming the apex) and is **not** addressed here. This PR fixes a real but *different* defect affecting clients that forward to PiHole — principally the SOHO router, which does. ## The defect: our authoritative answer is uncacheable, the wrong one caches for 5 minutes Measured on prod 2026-10-18: ``` dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0 dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300 dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143 ``` dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The router forwards LAN queries to PiHole and **caches** the result — the decrementing TTL above is proof it caches rather than forwarding live. So: - the **correct** answer is **uncacheable** — the router must re-query PiHole on *every* lookup, maximum exposure to any momentary gap; - the **wrong** answer is **cacheable for 300s** — the instant the router falls back to upstream (standard forwarder behaviour when the target does not answer) it latches the public WAN address and serves it to **every forwarding client** for five minutes. **One sub-second PiHole gap becomes five minutes of wrong answers** for those clients. Then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed. That asymmetry, not the gap, is what makes it visible. ## Fix `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines`. dnsmasq applies `local-ttl` to replies made from configuration, so one directive covers all five `address=` names. 60s is a deliberate floor: long enough to ride out a pod restart and an L2 lease re-election (both seconds), short enough that a genuine VIP change propagates inside a minute. These VIPs are pinned by `lbipam.cilium.io/ips` and effectively static, so the propagation cost is theoretical while the flap it prevents is measured. `apps/wave-0/pihole.yaml` regenerated with the canonical command (`--all catalog -o apps --wave-layout --no-image-automation`, deliberately **not** `--emit-waves`); exactly two files changed and the rendered diff is the single `local-ttl=60;` prefix. ## Not verified - **UNAPPLIED.** Verify after merge with `dig home.keithjackson.dev @192.168.2.23` — expect `TTL=60`, not 0. - **I did not confirm FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header records FTL previously writing a bogus line into `dnsmasq.conf` and reporting "0 invalid" — a malformed passthrough fails **silently**. If the TTL is still 0 after merge, check the pod's `/etc/pihole/dnsmasq.conf`; that is the cause, not this reasoning. - **It does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services with **two independently-elected L2 announcement leases**, so two nodes can ARP for it simultaneously — they sat on different nodes earlier the same day. Same pattern on crafty (`.22`) and unifi (`.21`). Merging each pair into one mixed-protocol Service would remove the class; that decision is parked. - **No help for an outage longer than the TTL** — the cache expires and the public answer latches again. That is an availability problem, not a caching one. - **No alert fires if the split-horizon answer regresses** to the public address. Nothing probes the name from a LAN vantage point and compares against `${GATEWAY_VIP}`. Local gate: **rc 3, zero failures**; only `gitea-settings` and L2/L3 `conformance` did not run. Full detail: `decisions/2026-10-18-pihole-local-ttl.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-18 08:05:36 +00:00

fix(pihole): make the split-horizon answer cacheable, or one blip latches the WAN IP for 5 minutes

gate / gate (pull_request) Successful in 1m56s

Details

gate / conformance (pull_request) Has been skipped

Details

fda5c31a94

Operator reported DNS flapping between the cluster ingress and the router's
public IP. PiHole is not at fault -- both replicas answer 192.168.2.20
consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in
CACHEABILITY that we created. Measured:

  pihole  192.168.2.23 -> 192.168.2.20               TTL=0
  public  1.1.1.1      -> wan.home.keithjackson.dev  TTL=300
  router  192.168.2.1  -> wan....  TTL=153 -> 148 -> 143   (decrementing = caching)

dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router
forwards LAN queries to PiHole and caches the result, so the CORRECT answer is
uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while
the WRONG answer caches for 300s. The instant the router falls back to upstream
(standard forwarder behaviour when the target does not answer) it latches the
WAN address for the whole LAN for five minutes, then expires, re-queries PiHole,
gets TTL 0, and is immediately re-exposed.

So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured
reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That
asymmetry, not the gap, is why it reads as "flapping".

Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies
local-ttl to replies made from configuration, so one directive covers all five
address= names. 60s rides out a pod restart and a lease re-election while a real
VIP change still propagates inside a minute; these VIPs are pinned and static.

apps/wave-0/pihole.yaml regenerated with the canonical render command
(--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves);
exactly two files changed and the rendered diff is the single local-ttl prefix.

UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected
L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If
the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough
line -- see the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 approved these changes

2026-10-18 08:19:19 +00:00

jakman85 merged commit 84bb0f0acd into main

2026-10-18 08:19:28 +00:00

jakman85 referenced this issue from a commit

2026-10-18 08:19:29 +00:00

Merge pull request 'fix(pihole): make the split-horizon answer cacheable (local-ttl=60)' (#78) from fix/pihole-local-ttl into main

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#78