Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

fix(pihole): make the split-horizon answer cacheable, or one blip latches the WAN IP for 5 minutes
gate / gate (pull_request) Successful in 1m56s
Details
gate / conformance (pull_request) Has been skipped
Details

Browse Source 
Operator reported DNS flapping between the cluster ingress and the router's
public IP. PiHole is not at fault -- both replicas answer 192.168.2.20
consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in
CACHEABILITY that we created. Measured:

  pihole  192.168.2.23 -> 192.168.2.20               TTL=0
  public  1.1.1.1      -> wan.home.keithjackson.dev  TTL=300
  router  192.168.2.1  -> wan....  TTL=153 -> 148 -> 143   (decrementing = caching)

dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router
forwards LAN queries to PiHole and caches the result, so the CORRECT answer is
uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while
the WRONG answer caches for 300s. The instant the router falls back to upstream
(standard forwarder behaviour when the target does not answer) it latches the
WAN address for the whole LAN for five minutes, then expires, re-queries PiHole,
gets TTL 0, and is immediately re-exposed.

So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured
reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That
asymmetry, not the gap, is why it reads as "flapping".

Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies
local-ttl to replies made from configuration, so one directive covers all five
address= names. 60s rides out a pod restart and a lease re-election while a real
VIP change still propagates inside a minute; these VIPs are pinned and static.

apps/wave-0/pihole.yaml regenerated with the canonical render command
(--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves);
exactly two files changed and the rendered diff is the single local-ttl prefix.

UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected
L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If
the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough
line -- see the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-18 02:56:49 -05:00
1 parent c340bbfc4b
commit fda5c31a94
3 files changed
+115 -2
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

apps/wave-0/pihole.yaml
+1 -1
View File
@@ -73,7 +73,7 @@ spec:
- name: FTLCONF_dns_rateLimit_count
value: '0'
- name: FTLCONF_misc_dnsmasq_lines
value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
- name: FTLCONF_webserver_api_password
valueFrom:
secretKeyRef:
catalog/pihole.yaml
+32 -1
View File
@@ -143,8 +143,39 @@ env:
# resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
# failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
# every client that takes the ADVERTISED connection cannot reach the server at all.
#
# ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
# MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
#
# dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0
# dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300
# dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143
#
# dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
# bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
# * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
# single lookup, which is maximum exposure to any momentary gap; while
# * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
# upstream (standard forwarder behaviour when the forward target does not answer) it latches
# the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
# The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
# live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
# flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
# That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
#
# 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
# re-election (both seconds), short enough that a genuine VIP change still propagates inside a
# minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
# propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
# LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
# the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
# cache expires, the router falls back, and the public answer latches again. That is a real
# outage, and the fix for it is PiHole availability, not DNS caching.
#
# Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
# configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
- name: FTLCONF_misc_dnsmasq_lines
value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
# carries the full rationale and the measurement that found it.
decisions/2026-10-18-pihole-local-ttl.md
+82
View File
@@ -0,0 +1,82 @@
---
date: 2026-10-18
branch: fix/pihole-local-ttl
title: "The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes"
---
## What changed
The operator reported DNS "flapping between the cluster ingress and the router (public IP)".
PiHole was not at fault — both replicas answer correctly and consistently. The bug is an
**asymmetry in cacheability** that we created.
Measured on prod 2026-10-18:
```
dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0
dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300
dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143
```
dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The SOHO router forwards
LAN queries to PiHole and **caches** the result — the decrementing TTL above is the proof it caches
rather than forwarding live. So:
* the **correct** answer is **uncacheable** (TTL 0): the router must re-query PiHole on *every*
lookup, which is maximum exposure to any momentary gap;
* the **wrong** answer is **cacheable for 300s**: the instant the router falls back to its upstream
(standard forwarder behaviour when the forward target does not answer) it latches the public WAN
address and serves it to **every LAN client** for five minutes.
So **one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP** —
then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed.
That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip.
Fix: `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines` in `catalog/pihole.yaml`, so the
private answer is cacheable and a short gap is invisible to clients. `apps/wave-0/pihole.yaml`
regenerated with the canonical render command.
## What I verified
* **PiHole is NOT the fault.** Both replicas, queried directly by pod IP, answer `192.168.2.20`;
the VIP returned it 6/6; this workstation 8/8. DHCP hands out only `192.168.2.23`.
* **The router caches rather than forwards live** — three successive queries returned TTL 153, 148,
143 (decrementing), which a pure forwarder could not do.
* **The TTL asymmetry**, as quoted above: 0 from PiHole, 300 from public DNS.
* **The router does answer DNS and returns the public chain** (`wan.home.keithjackson.dev` → the WAN
address), so a fallback always has a wrong answer available.
* **`local-ttl` is the right directive**: dnsmasq applies it to replies made from configuration,
`/etc/hosts` and DHCP leases, which covers all five `address=` names with one directive. It is a
standalone option, not a sub-field of `address=`, so prepending it to the `;`-separated
`FTLCONF_misc_dnsmasq_lines` list is syntactically correct.
* **The render is clean and scoped**: `catalog/render.py --all catalog -o apps --wave-layout
--no-image-automation` (the canonical command `ci/render-golden.sh` asserts against, and
deliberately **not** `--emit-waves`) changed exactly two files — `catalog/pihole.yaml` and
`apps/wave-0/pihole.yaml` — and the rendered diff is the single `local-ttl=60;` prefix.
## What I did NOT verify
* **UNAPPLIED.** Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL,
nor the router caching the private answer, nor the flap stopping. **Verify after merge with
`dig home.keithjackson.dev @192.168.2.23` (expect `TTL=60`, not 0) and then the same query at the
router (expect `192.168.2.20` with a decrementing TTL, not `wan.…`).**
* **I did not confirm that PiHole/FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The
directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header
records that FTL has previously written a bogus line into `dnsmasq.conf` and reported "0 invalid"
— so a malformed passthrough fails *silently*. If the TTL is still 0 after merge, that is the
cause, not this reasoning. Check the pod's `/etc/pihole/dnsmasq.conf`.
* **60s is chosen, not derived.** Long enough for a pod restart and a lease re-election (both
seconds); it does **nothing** for an outage longer than the TTL, where the cache expires and the
public answer latches again. That is a real availability problem, not a caching one.
* **This does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services
(`pihole-dns-tcp`, `pihole-dns-udp`) with **two independently-elected L2 announcement leases**, so
two nodes can ARP for the same address at once — they sat on different nodes earlier the same day
(tcp on prod-2, udp on prod-1). The same pattern applies to crafty (`.22`) and unifi (`.21`).
Merging each pair into one mixed-protocol Service would remove the class; the operator has parked
that decision, and it is deliberately **not** in this change.
* **Client-side bypasses are unexamined.** A device with its own resolver, or a browser with
DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help
there. Only this one workstation's resolver chain was inspected.
* **No alert fires if the split-horizon answer regresses to the public address.** Nothing probes
`home.keithjackson.dev` from a LAN vantage point and compares it against `${GATEWAY_VIP}`, so the
next recurrence would again be found by a human noticing. That gap is not closed here.