fix(pihole): make the split-horizon answer cacheable, or one blip latches the WAN IP for 5 minutes

Operator reported DNS flapping between the cluster ingress and the router's
public IP. PiHole is not at fault -- both replicas answer 192.168.2.20
consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in
CACHEABILITY that we created. Measured:

  pihole  192.168.2.23 -> 192.168.2.20               TTL=0
  public  1.1.1.1      -> wan.home.keithjackson.dev  TTL=300
  router  192.168.2.1  -> wan....  TTL=153 -> 148 -> 143   (decrementing = caching)

dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router
forwards LAN queries to PiHole and caches the result, so the CORRECT answer is
uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while
the WRONG answer caches for 300s. The instant the router falls back to upstream
(standard forwarder behaviour when the target does not answer) it latches the
WAN address for the whole LAN for five minutes, then expires, re-queries PiHole,
gets TTL 0, and is immediately re-exposed.

So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured
reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That
asymmetry, not the gap, is why it reads as "flapping".

Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies
local-ttl to replies made from configuration, so one directive covers all five
address= names. 60s rides out a pod restart and a lease re-election while a real
VIP change still propagates inside a minute; these VIPs are pinned and static.

apps/wave-0/pihole.yaml regenerated with the canonical render command
(--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves);
exactly two files changed and the rendered diff is the single local-ttl prefix.

UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected
L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If
the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough
line -- see the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

apps/wave-0/pihole.yaml

@@ -73,7 +73,7 @@ spec:
         - name: FTLCONF_dns_rateLimit_count
           value: '0'
         - name: FTLCONF_misc_dnsmasq_lines
-          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
+          value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
         - name: FTLCONF_webserver_api_password
           valueFrom:
             secretKeyRef:

catalog/pihole.yaml

@@ -143,8 +143,39 @@ env:
   # resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
   # failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
   # every client that takes the ADVERTISED connection cannot reach the server at all.
+  #
+  # ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
+  # MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
+  #
+  #   dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+  #   dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+  #   dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+  #
+  # dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
+  # bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
+  #   * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
+  #     single lookup, which is maximum exposure to any momentary gap; while
+  #   * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
+  #     upstream (standard forwarder behaviour when the forward target does not answer) it latches
+  #     the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
+  # The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
+  # live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
+  # flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
+  # That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
+  #
+  # 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
+  # re-election (both seconds), short enough that a genuine VIP change still propagates inside a
+  # minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
+  # propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
+  # LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
+  # the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
+  # cache expires, the router falls back, and the public answer latches again. That is a real
+  # outage, and the fix for it is PiHole availability, not DNS caching.
+  #
+  # Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
+  # configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
   - name: FTLCONF_misc_dnsmasq_lines
-    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
+    value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
   # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
   # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
   # carries the full rationale and the measurement that found it.

conformance/tests/l0-static/ha-control-plane-as-code/check.sh

@@ -445,8 +445,21 @@ join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
 check(join is not None,
       "roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
       "— without a join target the servers are separate single-node clusters")
-join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
-                      K3S_TEXT, re.S)
+# LINE-BOUNDED ON PURPOSE -- DO NOT "SIMPLIFY" BACK TO `.*` + re.S.
+# This was `--server https://.*?\n(?:.*\n){0,10}?\s*when:...` with re.S, which made every `.`
+# match newlines too. That is nested unbounded repetition (a `{0,10}` of a quantifier that can each
+# swallow the whole document), so a FAILED or late match backtracks exponentially. It cost
+# **19m54s of CPU in CI** (measured 2026-10-18 via `docker top` on the live job container) and made
+# the whole merge gate take ~28 minutes instead of ~2 on any change touching `infra/**` -- because
+# `ci/touched-requirements.sh` maps `infra/*` to R-ENV-01, which pulls in this whole-repo check.
+# It was latent for as long as the regex existed and only crossed the threshold when
+# roles/k3s_server grew by ~100 lines (the kubelet graceful-shutdown config, PR #71): K3S_TEXT is
+# every *.yml in that role concatenated, so the role growing is the input growing.
+# `[^\n]*` cannot cross a line, so the `{0,10}` is a genuine ten-line window with nothing to
+# backtrack into. Measured on the same input: 0.2 ms, same verdict (match found).
+join_when = re.search(
+    r"--server https://[^\n]*\n(?:[^\n]*\n){0,10}?[ \t]*when:[ \t]*k3s_server_mode[ \t]*==[ \t]*'secondary'",
+    K3S_TEXT)
 check(join_when is not None,
       "the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
       "would also try to join itself")

decisions/2026-10-18-ha-check-catastrophic-regex.md

@@ -0,0 +1,76 @@
+---
+date: 2026-10-18
+branch: fix/ha-check-catastrophic-regex
+title: "One regex made the merge gate take 28 minutes instead of 2"
+---
+
+## What changed
+
+The merge gate started taking **~28 minutes** instead of its usual **<2 minutes**, on every change
+touching `infra/**`. One regex was responsible for essentially all of it.
+
+`conformance/tests/l0-static/ha-control-plane-as-code/check.sh` contained:
+
+```python
+join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
+                      K3S_TEXT, re.S)
+```
+
+With `re.S`, every `.` also matches newlines, so `.*?` and the nested `(?:.*\n){0,10}?` can each
+consume the entire document. That is **nested unbounded repetition** — ten levels of a quantifier
+that can each swallow everything — and a failed or late match backtracks exponentially.
+
+Replaced `.` with `[^\n]*` and dropped `re.S`, so the `{0,10}` is a genuine ten-line window with
+nothing to backtrack into. Same intent, same verdict, no DOTALL.
+
+**Why it surfaced now, and it was mine.** `K3S_TEXT` is every `*.yml` in `roles/k3s_server`
+concatenated. The regex was latent for as long as it existed; **PR #71 added ~100 lines to that role
+(the kubelet graceful-shutdown config), which grew the input past the point where exponential
+backtracking stopped being survivable.** Nothing about the check changed — my change grew its input.
+
+**Why only `infra/**` changes were slow.** `ci/touched-requirements.sh:268` maps `infra/*` to
+`R-ENV-01`, and the resulting L0 run is `--include-scope whole-repo --requirements R-ENV-01`, which
+pulls in this whole-repo check. Doc-only and `platform/**` changes never selected it, which is why
+#69 (1m55s) and #72 (1m53s) stayed fast while #71 (27m44s), the #73 merge (27m46s) and #75 all ran
+~28 minutes.
+
+## What I verified
+
+* **The hot spot, by measurement rather than reading.** `docker top` on the live CI job container for
+  run #299 showed `python3 -` with **19:54 of CPU time** under
+  `ha-control-plane-as-code/check.sh`, inside
+  `conformance/run.sh --layer l0 --include-scope whole-repo --requirements R-ENV-01`.
+* **Reproduced locally**: the check did not finish within a 100-second budget on an unmodified
+  checkout.
+* **Localised to the exact line** by extracting the inline Python and running it under
+  `faulthandler.dump_traceback_later(25)`, which named `re/__init__.py:177 in search` called from
+  the `join_when` line — not inferred from reading the patterns.
+* **The pattern is unique in the repo**: `grep -rlnE "\{0,[0-9]+\}\?" conformance/tests/l0-static/*/check.sh`
+  returns only this file, so there is no sibling with the same hazard.
+* **The fix is fast AND verdict-preserving**, tested against the real `K3S_TEXT` (22,179 chars, 403
+  lines): the new regex matches (`YES`) in **0.2 ms**. Sanity-checked that the thing it looks for
+  genuinely exists — one `--server https://` and three `k3s_server_mode == 'secondary'` occurrences.
+* **The whole check now runs in 0.199s, rc 0, 115 assertions**, with the same success message as
+  before.
+
+## What I did NOT verify
+
+* **I did not re-time the full gate in CI after the fix.** The expectation is that `infra/**` changes
+  return to the ~2-minute range, but that is arithmetic from the 19m54s figure, not a measured
+  post-fix run. The PR's own CI run is the first real datapoint — read it.
+* **I did not audit the other 23 L0 checks for different performance pathologies.** This one was
+  found because it was actively burning CPU in a live job; a check that is merely slow (not
+  exponential) would not have stood out, and `ci/gate.sh` has **no per-stage timing** at all
+  (`grep -nE "SECONDS|date \+|elapsed"` finds nothing), so there is no data on where the remaining
+  time goes.
+* **Nothing prevents this class of bug returning.** No check asserts gate stage runtime, and no lint
+  forbids nested quantifiers under `re.S`. A timing budget per stage, or a `re` lint, would close it;
+  neither is written here.
+* **I did not verify my claim about *why* only `infra/**` was affected beyond the path mapping and
+  the observed durations.** The correlation is strong (5 runs, two clear clusters) and the live
+  process list confirms the `--requirements R-ENV-01` scope, but I did not read
+  `conformance/run.sh`'s scope-selection logic end to end.
+* The regex rewrite assumes the `when:` clause is indented with spaces or tabs only (`[ \t]*`). A
+  file using some other whitespace before `when:` would now fail to match where the old DOTALL
+  version would have succeeded. Unlikely in YAML, and the current input matches, but it is a
+  narrowing.

decisions/2026-10-18-pihole-local-ttl.md

@@ -0,0 +1,82 @@
+---
+date: 2026-10-18
+branch: fix/pihole-local-ttl
+title: "The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes"
+---
+
+## What changed
+
+The operator reported DNS "flapping between the cluster ingress and the router (public IP)".
+PiHole was not at fault — both replicas answer correctly and consistently. The bug is an
+**asymmetry in cacheability** that we created.
+
+Measured on prod 2026-10-18:
+
+```
+dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+```
+
+dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The SOHO router forwards
+LAN queries to PiHole and **caches** the result — the decrementing TTL above is the proof it caches
+rather than forwarding live. So:
+
+* the **correct** answer is **uncacheable** (TTL 0): the router must re-query PiHole on *every*
+  lookup, which is maximum exposure to any momentary gap;
+* the **wrong** answer is **cacheable for 300s**: the instant the router falls back to its upstream
+  (standard forwarder behaviour when the forward target does not answer) it latches the public WAN
+  address and serves it to **every LAN client** for five minutes.
+
+So **one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP** —
+then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed.
+That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip.
+
+Fix: `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines` in `catalog/pihole.yaml`, so the
+private answer is cacheable and a short gap is invisible to clients. `apps/wave-0/pihole.yaml`
+regenerated with the canonical render command.
+
+## What I verified
+
+* **PiHole is NOT the fault.** Both replicas, queried directly by pod IP, answer `192.168.2.20`;
+  the VIP returned it 6/6; this workstation 8/8. DHCP hands out only `192.168.2.23`.
+* **The router caches rather than forwards live** — three successive queries returned TTL 153, 148,
+  143 (decrementing), which a pure forwarder could not do.
+* **The TTL asymmetry**, as quoted above: 0 from PiHole, 300 from public DNS.
+* **The router does answer DNS and returns the public chain** (`wan.home.keithjackson.dev` → the WAN
+  address), so a fallback always has a wrong answer available.
+* **`local-ttl` is the right directive**: dnsmasq applies it to replies made from configuration,
+  `/etc/hosts` and DHCP leases, which covers all five `address=` names with one directive. It is a
+  standalone option, not a sub-field of `address=`, so prepending it to the `;`-separated
+  `FTLCONF_misc_dnsmasq_lines` list is syntactically correct.
+* **The render is clean and scoped**: `catalog/render.py --all catalog -o apps --wave-layout
+  --no-image-automation` (the canonical command `ci/render-golden.sh` asserts against, and
+  deliberately **not** `--emit-waves`) changed exactly two files — `catalog/pihole.yaml` and
+  `apps/wave-0/pihole.yaml` — and the rendered diff is the single `local-ttl=60;` prefix.
+
+## What I did NOT verify
+
+* **UNAPPLIED.** Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL,
+  nor the router caching the private answer, nor the flap stopping. **Verify after merge with
+  `dig home.keithjackson.dev @192.168.2.23` (expect `TTL=60`, not 0) and then the same query at the
+  router (expect `192.168.2.20` with a decrementing TTL, not `wan.…`).**
+* **I did not confirm that PiHole/FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The
+  directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header
+  records that FTL has previously written a bogus line into `dnsmasq.conf` and reported "0 invalid"
+  — so a malformed passthrough fails *silently*. If the TTL is still 0 after merge, that is the
+  cause, not this reasoning. Check the pod's `/etc/pihole/dnsmasq.conf`.
+* **60s is chosen, not derived.** Long enough for a pod restart and a lease re-election (both
+  seconds); it does **nothing** for an outage longer than the TTL, where the cache expires and the
+  public answer latches again. That is a real availability problem, not a caching one.
+* **This does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services
+  (`pihole-dns-tcp`, `pihole-dns-udp`) with **two independently-elected L2 announcement leases**, so
+  two nodes can ARP for the same address at once — they sat on different nodes earlier the same day
+  (tcp on prod-2, udp on prod-1). The same pattern applies to crafty (`.22`) and unifi (`.21`).
+  Merging each pair into one mixed-protocol Service would remove the class; the operator has parked
+  that decision, and it is deliberately **not** in this change.
+* **Client-side bypasses are unexamined.** A device with its own resolver, or a browser with
+  DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help
+  there. Only this one workstation's resolver chain was inspected.
+* **No alert fires if the split-horizon answer regresses to the public address.** Nothing probes
+  `home.keithjackson.dev` from a LAN vantage point and compares it against `${GATEWAY_VIP}`, so the
+  next recurrence would again be found by a human noticing. That gap is not closed here.