fix(pihole): make the split-horizon answer cacheable, or one blip latches the WAN IP for 5 minutes

Operator reported DNS flapping between the cluster ingress and the router's
public IP. PiHole is not at fault -- both replicas answer 192.168.2.20
consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in
CACHEABILITY that we created. Measured:

  pihole  192.168.2.23 -> 192.168.2.20               TTL=0
  public  1.1.1.1      -> wan.home.keithjackson.dev  TTL=300
  router  192.168.2.1  -> wan....  TTL=153 -> 148 -> 143   (decrementing = caching)

dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router
forwards LAN queries to PiHole and caches the result, so the CORRECT answer is
uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while
the WRONG answer caches for 300s. The instant the router falls back to upstream
(standard forwarder behaviour when the target does not answer) it latches the
WAN address for the whole LAN for five minutes, then expires, re-queries PiHole,
gets TTL 0, and is immediately re-exposed.

So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured
reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That
asymmetry, not the gap, is why it reads as "flapping".

Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies
local-ttl to replies made from configuration, so one directive covers all five
address= names. 60s rides out a pod restart and a lease re-election while a real
VIP change still propagates inside a minute; these VIPs are pinned and static.

apps/wave-0/pihole.yaml regenerated with the canonical render command
(--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves);
exactly two files changed and the rendered diff is the single local-ttl prefix.

UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected
L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If
the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough
line -- see the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

catalog/pihole.yaml

@@ -143,8 +143,39 @@ env:
   # resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
   # failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
   # every client that takes the ADVERTISED connection cannot reach the server at all.
+  #
+  # ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
+  # MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
+  #
+  #   dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+  #   dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+  #   dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+  #
+  # dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
+  # bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
+  #   * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
+  #     single lookup, which is maximum exposure to any momentary gap; while
+  #   * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
+  #     upstream (standard forwarder behaviour when the forward target does not answer) it latches
+  #     the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
+  # The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
+  # live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
+  # flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
+  # That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
+  #
+  # 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
+  # re-election (both seconds), short enough that a genuine VIP change still propagates inside a
+  # minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
+  # propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
+  # LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
+  # the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
+  # cache expires, the router falls back, and the public answer latches again. That is a real
+  # outage, and the fix for it is PiHole availability, not DNS caching.
+  #
+  # Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
+  # configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
   - name: FTLCONF_misc_dnsmasq_lines
-    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
+    value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
   # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
   # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
   # carries the full rationale and the measurement that found it.