Merge pull request 'fix(pihole): make the split-horizon answer cacheable (local-ttl=60)' (#78) from fix/pihole-local-ttl into main

Reviewed-on: #78
Reviewed-by: Keith Jackson <[email protected]>

apps/wave-0/pihole.yaml

@@ -73,7 +73,7 @@ spec:
         - name: FTLCONF_dns_rateLimit_count
           value: '0'
         - name: FTLCONF_misc_dnsmasq_lines
-          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
+          value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
         - name: FTLCONF_webserver_api_password
           valueFrom:
             secretKeyRef:

catalog/pihole.yaml

@@ -143,8 +143,39 @@ env:
   # resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
   # failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
   # every client that takes the ADVERTISED connection cannot reach the server at all.
+  #
+  # ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
+  # MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
+  #
+  #   dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+  #   dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+  #   dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+  #
+  # dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
+  # bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
+  #   * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
+  #     single lookup, which is maximum exposure to any momentary gap; while
+  #   * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
+  #     upstream (standard forwarder behaviour when the forward target does not answer) it latches
+  #     the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
+  # The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
+  # live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
+  # flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
+  # That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
+  #
+  # 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
+  # re-election (both seconds), short enough that a genuine VIP change still propagates inside a
+  # minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
+  # propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
+  # LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
+  # the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
+  # cache expires, the router falls back, and the public answer latches again. That is a real
+  # outage, and the fix for it is PiHole availability, not DNS caching.
+  #
+  # Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
+  # configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
   - name: FTLCONF_misc_dnsmasq_lines
-    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
+    value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
   # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
   # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
   # carries the full rationale and the measurement that found it.

decisions/2026-10-18-pihole-local-ttl.md

@@ -0,0 +1,82 @@
+---
+date: 2026-10-18
+branch: fix/pihole-local-ttl
+title: "The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes"
+---
+
+## What changed
+
+The operator reported DNS "flapping between the cluster ingress and the router (public IP)".
+PiHole was not at fault — both replicas answer correctly and consistently. The bug is an
+**asymmetry in cacheability** that we created.
+
+Measured on prod 2026-10-18:
+
+```
+dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+```
+
+dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The SOHO router forwards
+LAN queries to PiHole and **caches** the result — the decrementing TTL above is the proof it caches
+rather than forwarding live. So:
+
+* the **correct** answer is **uncacheable** (TTL 0): the router must re-query PiHole on *every*
+  lookup, which is maximum exposure to any momentary gap;
+* the **wrong** answer is **cacheable for 300s**: the instant the router falls back to its upstream
+  (standard forwarder behaviour when the forward target does not answer) it latches the public WAN
+  address and serves it to **every LAN client** for five minutes.
+
+So **one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP** —
+then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed.
+That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip.
+
+Fix: `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines` in `catalog/pihole.yaml`, so the
+private answer is cacheable and a short gap is invisible to clients. `apps/wave-0/pihole.yaml`
+regenerated with the canonical render command.
+
+## What I verified
+
+* **PiHole is NOT the fault.** Both replicas, queried directly by pod IP, answer `192.168.2.20`;
+  the VIP returned it 6/6; this workstation 8/8. DHCP hands out only `192.168.2.23`.
+* **The router caches rather than forwards live** — three successive queries returned TTL 153, 148,
+  143 (decrementing), which a pure forwarder could not do.
+* **The TTL asymmetry**, as quoted above: 0 from PiHole, 300 from public DNS.
+* **The router does answer DNS and returns the public chain** (`wan.home.keithjackson.dev` → the WAN
+  address), so a fallback always has a wrong answer available.
+* **`local-ttl` is the right directive**: dnsmasq applies it to replies made from configuration,
+  `/etc/hosts` and DHCP leases, which covers all five `address=` names with one directive. It is a
+  standalone option, not a sub-field of `address=`, so prepending it to the `;`-separated
+  `FTLCONF_misc_dnsmasq_lines` list is syntactically correct.
+* **The render is clean and scoped**: `catalog/render.py --all catalog -o apps --wave-layout
+  --no-image-automation` (the canonical command `ci/render-golden.sh` asserts against, and
+  deliberately **not** `--emit-waves`) changed exactly two files — `catalog/pihole.yaml` and
+  `apps/wave-0/pihole.yaml` — and the rendered diff is the single `local-ttl=60;` prefix.
+
+## What I did NOT verify
+
+* **UNAPPLIED.** Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL,
+  nor the router caching the private answer, nor the flap stopping. **Verify after merge with
+  `dig home.keithjackson.dev @192.168.2.23` (expect `TTL=60`, not 0) and then the same query at the
+  router (expect `192.168.2.20` with a decrementing TTL, not `wan.…`).**
+* **I did not confirm that PiHole/FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The
+  directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header
+  records that FTL has previously written a bogus line into `dnsmasq.conf` and reported "0 invalid"
+  — so a malformed passthrough fails *silently*. If the TTL is still 0 after merge, that is the
+  cause, not this reasoning. Check the pod's `/etc/pihole/dnsmasq.conf`.
+* **60s is chosen, not derived.** Long enough for a pod restart and a lease re-election (both
+  seconds); it does **nothing** for an outage longer than the TTL, where the cache expires and the
+  public answer latches again. That is a real availability problem, not a caching one.
+* **This does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services
+  (`pihole-dns-tcp`, `pihole-dns-udp`) with **two independently-elected L2 announcement leases**, so
+  two nodes can ARP for the same address at once — they sat on different nodes earlier the same day
+  (tcp on prod-2, udp on prod-1). The same pattern applies to crafty (`.22`) and unifi (`.21`).
+  Merging each pair into one mixed-protocol Service would remove the class; the operator has parked
+  that decision, and it is deliberately **not** in this change.
+* **Client-side bypasses are unexamined.** A device with its own resolver, or a browser with
+  DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help
+  there. Only this one workstation's resolver chain was inspected.
+* **No alert fires if the split-horizon answer regresses to the public address.** Nothing probes
+  `home.keithjackson.dev` from a LAN vantage point and compares it against `${GATEWAY_VIP}`, so the
+  next recurrence would again be found by a human noticing. That gap is not closed here.