From fd43f332d694f89a8c8eefc306e7aec77df23ae1 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 14 Sep 2026 00:57:30 -0500 Subject: [PATCH] feat(pihole): authoritative LAN DNS for the zone on udp/53 + tcp/53 at a pinned VIP platform/coredns/coredns-custom.yaml has always said its split-horizon is for PODS and deferred LAN clients to "PiHole split-horizon". PiHole served no DNS: its Service was ClusterIP with ports 80 and 4180 only. The LAN half of R-DNS-03 that docs/09 and the README's Phase 2 row both claim did not exist, so LAN clients ran on hand-maintained per-client dnsmasq files - one went stale after the dev decommission and cost the operator sonarr.home.keithjackson.dev. The SOHO router that owns 192.168.2.0/24 cannot cover for it either: it strips RFC1918 answers for public domains as rebind protection and returns NOERROR with no answer. EXPOSURE (apps/wave-0/pihole-dns.yaml, hand-written, placed into wave 0 by render.py's EXTRA_UNITS - the schema has no field for a LoadBalancer Service, a pinned lbipam VIP, a sharing key, source ranges or UDP, the same reason apps/vms/unifi and platform/crafty hand-write theirs): - two Services, udp/53 and tcp/53, sharing one pinned VIP ${PIHOLE_DNS_VIP} = 192.168.2.23 on prod, the next free address in the .20-.30 pool (pool status IPsUsed 3 of 11; .20 Gateway, .21 unifi, .22 crafty; arping from a LAN host got no reply). - no external-dns hostname annotation: nothing public is created, and platform/externaldns/ is untouched. ANSWERS (catalog/pihole.yaml env - declarative, no reconciler): PiHole v6 takes every FTL setting as an FTLCONF_ environment variable and treats it as env-forced, applied ahead of pihole.toml on every cold start and read-only in the UI. That is a real declarative entry point, which the Keycloak and CNPG CronJob reconcilers exist precisely because their state lacks, so a reconciler here would be ceremony. misc.dnsmasq_lines carries a wildcard address=// plus one longer-domain override per L4 service (unifi, crafty-mc, pihole-dns); dnsmasq resolves overlaps by longest matching domain, so precedence is structural and a newly onboarded Gateway-fronted service needs no DNS change. dns.upstreams is declared too - the live pod carried hand-set upstreams that no manifest knew about and that every restart discarded. externalTrafficPolicy Cluster, NOT Local, and loadBalancerSourceRanges for the LAN restriction. Measured on prod, Cilium 1.20.1: the L2 announcer does not filter lease candidates by which nodes hold a backend - the two Services landed on k3s-prod-3 and k3s-prod-2 while the pod ran on k3s-prod-1 - and it keeps renewing after the pod moves away, so with Local a reschedule leaves LAN DNS dead indefinitely. Cluster serves the VIP from any node; the LAN restriction moves to loadBalancerSourceRanges, which Cilium enforces in eBPF on the VIP before load-balancing (verified by narrowing the range to exclude the test client and back). The cost is SNAT: forwarded queries reach PiHole from the ingress node, so PiHole's per-client rate limit is disabled in the catalog or the whole LAN would count as one client against its 1000-per-60s default. Proven from a LAN host (192.168.2.132) against 192.168.2.23 with the pod and both leases on three DIFFERENT nodes: sonarr/grafana/nextcloud/keycloak/apex -> 192.168.2.20, unifi -> .21, crafty-mc -> .22, pihole-dns -> .23, recursion (cloudflare.com, github.com, debian.org) intact over both udp and tcp, AAAA NODATA, NXDOMAIN outside the zone, and still answering after a forced prod-1 -> prod-2 reschedule. Co-Authored-By: Claude Opus 5 --- README.md | 2 +- apps/wave-0/kustomization.yaml | 1 + apps/wave-0/pihole-dns.yaml | 196 +++++++++++++++++++++++++++ apps/wave-0/pihole.yaml | 9 ++ catalog/pihole.yaml | 87 ++++++++++++ catalog/render.py | 19 +++ clusters/base/cluster-vars.yaml | 28 +++- clusters/prod/kustomization.yaml | 17 +++ platform/coredns/coredns-custom.yaml | 19 +++ 9 files changed, 376 insertions(+), 2 deletions(-) create mode 100644 apps/wave-0/pihole-dns.yaml diff --git a/README.md b/README.md index 4b224fa..4d3672d 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,7 @@ bug but the honest answer, because proofs are claims about a specific tree on a |---|---| | 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI | | 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) | -| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS | +| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files | | 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch | | 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled | | 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** | diff --git a/apps/wave-0/kustomization.yaml b/apps/wave-0/kustomization.yaml index aa3b049..eb649a6 100644 --- a/apps/wave-0/kustomization.yaml +++ b/apps/wave-0/kustomization.yaml @@ -14,6 +14,7 @@ resources: - nextcloud-db-backup-credentials-prod.sops.yaml - nextcloud-db-backup-credentials.sops.yaml - nextcloud-db.yaml +- pihole-dns.yaml - pihole-oauth2-proxy-credentials.sops.yaml - pihole.yaml - plex.yaml diff --git a/apps/wave-0/pihole-dns.yaml b/apps/wave-0/pihole-dns.yaml new file mode 100644 index 0000000..2bfe140 --- /dev/null +++ b/apps/wave-0/pihole-dns.yaml @@ -0,0 +1,196 @@ +--- +# ================================================================================================= +# PiHole DNS EXPOSURE — the LAN half of split-horizon (R-DNS-01 / R-DNS-03), and NETWORK POLICY. +# ================================================================================================= +# HAND-WRITTEN, NOT RENDERED. This file is a non-catalog app-tier unit anchored to the `pihole` +# ServiceDefinition and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS +# table (the same mechanism gitea-actions uses). It is hand-written because the ServiceDefinition +# schema genuinely cannot express it: schemas/servicedefinition.schema.json is +# `additionalProperties: false` and has NO field for a type=LoadBalancer Service, for a pinned +# lbipam VIP, for a sharing key, for loadBalancerSourceRanges, or for UDP north-south exposure at +# all. `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator and +# cannot carry DNS. The two existing L4 workloads in this repo are hand-written for exactly this +# reason and in exactly this shape — apps/vms/unifi/50-service-l4.yaml and +# platform/crafty/service-l4.yaml. The ANSWERS PiHole gives are declared in the catalog +# (catalog/pihole.yaml `env`), so the only thing living outside the catalog is the exposure the +# schema has no word for. +# +# WHY THIS EXISTS AT ALL: platform/coredns/coredns-custom.yaml is explicit that its split-horizon is +# for PODS and defers LAN clients to "PiHole split-horizon" — but PiHole served no DNS. Its Service +# was ClusterIP with ports 80 and 4180 only. The LAN half of the split-horizon that docs/09 +# (R-DNS-03) and the README's Phase 2 row both claim simply did not exist, and LAN clients ran on +# hand-maintained per-client dnsmasq files instead. Upstream cannot substitute: the SOHO router that +# owns this LAN strips RFC1918 answers for public domains as rebind protection (NOERROR, ANSWER 0). +# +# VIP: ${PIHOLE_DNS_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.56, prod 192.168.2.23), PINNED via +# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool` +# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the +# existing CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config +# is needed; these Services just consume the pool. PINNING IS NOT OPTIONAL HERE: this address is +# typed into every LAN client's resolver configuration (or handed out by the router's DHCP), so it +# can never be allowed to move. +# +# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so udp/53 and tcp/53 +# split into two Services sharing one address via lbipam.cilium.io/sharing-key — the same +# arrangement unifi and crafty use. BOTH PROTOCOLS ARE REQUIRED, not belt-and-braces: UDP is the +# normal query path, and TCP is what a resolver falls back to for any answer that does not fit in a +# UDP datagram and what some clients use unconditionally. A UDP-only resolver fails intermittently +# and confusingly. +# +# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry +# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their address as a PUBLIC +# Cloudflare record. This Service must not: `pihole-dns.${BASE_DOMAIN}` is answered by PiHole itself +# (catalog/pihole.yaml's dnsmasq override), it is an RFC1918 address that is useless in public DNS, +# and publishing a resolver's address in a public zone invites exactly the traffic the source-range +# filter below exists to refuse. Internal name, internal answer, no public record — which is also +# why this change touches nothing under platform/externaldns/. +# +# ------------------------------------------------------------------------------------------------- +# externalTrafficPolicy: Cluster — AND WHY IT IS NOT `Local`, WHICH IS WHAT unifi AND crafty USE. +# ------------------------------------------------------------------------------------------------- +# `Local` is the obvious choice: it preserves the client source IP and, in theory, pins the VIP to +# the node running the backend. The second half is NOT TRUE on this cluster, and that was measured, +# not assumed (Cilium 1.20.1, prod, 2026-09-14): +# +# * Cilium's L2 announcer elects one lease holder per Service among the nodes matching the +# CiliumL2AnnouncementPolicy's nodeSelector — `default-l2` matches ALL linux nodes — and it does +# NOT filter that candidate set by which nodes hold a backend. Created fresh, the two Services +# landed on DIFFERENT nodes (`cilium-l2announce-pihole-pihole-dns-tcp` -> k3s-prod-3, +# `-udp` -> k3s-prod-2) while the pihole pod ran on k3s-prod-1. +# * With `Local`, a node that announces the VIP but holds no backend has nowhere to send the +# packet, and the query fails with ICMP port unreachable (dig: "connection refused"). On three +# nodes with one replica that is a 1-in-3 chance of working, drawn independently per protocol. +# * It does not self-heal. The lease holder keeps RENEWING after the pod moves away: a pod +# rescheduled from k3s-prod-1 to k3s-prod-2 left the lease on k3s-prod-1, still renewing three +# minutes later, with LAN resolution dead the whole time. Only deleting the Lease forced a +# re-election — and that re-election is random, so it may pick a backend-less node again. +# +# A resolver that works one time in three and stays broken after any reschedule is not a resolver. +# `Cluster` makes every node able to serve the VIP by forwarding to wherever the pod is, which is +# what makes this survive a reschedule and a rebuild. VERIFIED in exactly the broken arrangement +# above — leases on k3s-prod-2 and k3s-prod-3, pod on k3s-prod-1 — udp/53, tcp/53 and recursion all +# answering. +# +# THE COST, STATED PLAINLY: Cilium runs in SNAT mode here (no `loadBalancer.mode: dsr` in +# platform/cilium/helmrelease.yaml), so a query forwarded between nodes reaches PiHole with the +# INGRESS NODE's address as its source, not the client's. PiHole's per-client query log, per-client +# groups and per-client rate limiting therefore see one apparent client instead of the LAN. Two +# consequences are handled: the rate limit is disabled in catalog/pihole.yaml (otherwise the whole +# LAN counts as one client against PiHole's 1000-queries-per-60s default and PiHole starts +# REFUSING), and the LAN restriction is enforced in the datapath rather than by source identity — +# see loadBalancerSourceRanges below. The fix that would restore client IPs is `loadBalancer.mode: +# dsr` on the Cilium HelmRelease, which is a platform-wide change with an Ansible [parity] contract +# and would also fix the same latent defect in unifi and crafty; it belongs in its own change. +# +# ------------------------------------------------------------------------------------------------- +# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH. +# ------------------------------------------------------------------------------------------------- +# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and +# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the +# datapath — it never reaches a backend and never reaches a network policy. This is what makes +# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the +# range to a /24 that excludes the test client, after which the same query that had been answering +# stopped being answered at all, and restoring it, after which it answered again. +# +# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this +# restriction once the source has been rewritten: measured on this cluster, the forwarded query +# arrives from the ingress node's internal router address and Cilium resolves that source to the +# reserved `world` identity, which a `fromCIDR` clause does not match even when the CIDR covers the +# address (tried: fromCIDR ${CLUSTER_POD_CIDR}, still dropped as `world`). So the peer restriction +# lives here, where it is actually enforceable, and the policy below restricts the PORTS. +apiVersion: v1 +kind: Service +metadata: + name: pihole-dns-udp + namespace: pihole + labels: + app.kubernetes.io/name: pihole + annotations: + lbipam.cilium.io/sharing-key: "pihole-dns" + lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}" +spec: + type: LoadBalancer + externalTrafficPolicy: Cluster + loadBalancerSourceRanges: + - ${LAN_CIDR} + selector: + app.kubernetes.io/name: pihole + ports: + - { name: dns-udp, port: 53, targetPort: 53, protocol: UDP } +--- +apiVersion: v1 +kind: Service +metadata: + name: pihole-dns-tcp + namespace: pihole + labels: + app.kubernetes.io/name: pihole + annotations: + lbipam.cilium.io/sharing-key: "pihole-dns" # same key + same IP => shares the VIP with -udp + lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}" +spec: + type: LoadBalancer + externalTrafficPolicy: Cluster + loadBalancerSourceRanges: + - ${LAN_CIDR} + selector: + app.kubernetes.io/name: pihole + ports: + - { name: dns-tcp, port: 53, targetPort: 53, protocol: TCP } +--- +# ================================================================================================= +# NETWORK POLICY — a NEW ingress path from off-cluster, restricted to the two DNS ports. +# ================================================================================================= +# The pihole namespace is default-deny on ingress (the baseline NetworkPolicy in the rendered +# apps/wave-0/pihole.yaml: `ingress: []`). Cilium enforces the UNION of that NetworkPolicy and the +# CiliumNetworkPolicies in the namespace, so this file ADDS one path and nothing else. Everything +# undeclared stays denied, and the existing `pihole-allow-gateway-ingress` (Gateway -> :4180, the +# oauth2-proxy gate in front of the admin UI) is untouched. PiHole's own :80 is NOT in this rule: +# the admin UI stays reachable only through the Gateway and oauth2-proxy, and a LAN client that +# talks to ${PIHOLE_DNS_VIP} on any port other than 53 gets nothing — there is no Service port for +# it and no policy for it. +# +# PORTS ARE THE NARROWING HERE; THE LAN RESTRICTION IS THE SOURCE-RANGE FILTER ABOVE. The peer is +# `world` for a mechanical reason, not a lazy one: with externalTrafficPolicy Cluster in Cilium's +# SNAT mode a forwarded query arrives from the ingress node's internal router address, which Cilium +# resolves to the reserved `world` identity — a `fromCIDR` clause does not match it, which was +# measured (a fromCIDR of ${CLUSTER_POD_CIDR}, which contains that address, still produced +# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer +# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of +# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this +# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here. +# +# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress +# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all; +# this rule admits fewer entities (no `cluster` — in-cluster pods resolve through CoreDNS and none +# needs PiHole on :53) on fewer ports, behind a filter those two do not have. +# +# WHAT REMAINS REACHABLE WITHOUT THE VIP, ACKNOWLEDGED: Cilium applies source ranges to the +# LoadBalancer address, not to the NodePorts the same Services allocate, so a host that can route to +# a node IP can still reach udp/53 and tcp/53 through the NodePort. On this LAN that is the same +# population the source range admits, and nothing at the edge forwards 53 inward, so PiHole is not +# an internet-facing open resolver. Say so out loud rather than implying the filter is total. +# +# `host` and `remote-node` are named explicitly because node-sourced traffic carries a reserved +# Cilium identity that no CIDR or `world` selector matches: the kubelet's probes and the +# node-to-node forwarding that IS the Cluster-policy datapath both need them. +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: pihole-allow-lan-dns-ingress + namespace: pihole + labels: + app.kubernetes.io/name: pihole +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: pihole + ingress: + - fromEntities: + - world # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR + - host # this node's kubelet probes + - remote-node # the node-to-node hop that forwards a query to wherever the pod runs + toPorts: + - ports: + - { port: "53", protocol: UDP } + - { port: "53", protocol: TCP } diff --git a/apps/wave-0/pihole.yaml b/apps/wave-0/pihole.yaml index edf537e..5e06cec 100644 --- a/apps/wave-0/pihole.yaml +++ b/apps/wave-0/pihole.yaml @@ -81,6 +81,15 @@ spec: image: pihole/pihole ports: - containerPort: 80 + env: + - name: FTLCONF_dns_listeningMode + value: ALL + - name: FTLCONF_dns_upstreams + value: 8.8.8.8;8.8.4.4 + - name: FTLCONF_dns_rateLimit_count + value: '0' + - name: FTLCONF_misc_dnsmasq_lines + value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP} volumeMounts: - name: config mountPath: /data/config diff --git a/catalog/pihole.yaml b/catalog/pihole.yaml index e64bae4..309a1fb 100644 --- a/catalog/pihole.yaml +++ b/catalog/pihole.yaml @@ -13,6 +13,93 @@ storage: size: 2Gi class: longhorn # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03) dependencies: [] # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02) +# ------------------------------------------------------------------------------------------------ +# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to. +# ------------------------------------------------------------------------------------------------ +# coredns-custom answers the ${BASE_DOMAIN} zone for PODS ONLY and says so in its own header ("LAN +# clients keep their own answers (PiHole split-horizon)"). Until now PiHole served no DNS at all — +# its Service was ClusterIP :80/:4180, no port 53, no LoadBalancer — so the LAN half did not exist +# and LAN clients depended on hand-maintained per-client dnsmasq files. One went stale after the dev +# decommission and cost the operator sonarr.${BASE_DOMAIN}. Upstream cannot cover for it either: the +# SOHO router that owns this LAN strips RFC1918 answers for public domains as rebind protection and +# returns NOERROR / ANSWER 0. apps/wave-0/pihole-dns.yaml exposes udp/53 + tcp/53 on +# ${PIHOLE_DNS_VIP}; these env vars are what make the ANSWERS declarative. +# +# WHY env AND NOT A RECONCILER: the repo's level-triggered CronJob reconcilers +# (platform/keycloak/client-secret-reconciler.yaml, the CNPG grant reconciler in +# apps/wave-0/nextcloud-db.yaml) exist because their state has NO declarative entry point — a +# Keycloak client secret lives in Keycloak's own database and a Postgres GRANT lives in the +# catalog, and nothing in a manifest can express either. PiHole v6 is not that case. Every FTL +# setting has an `FTLCONF_
_` environment variable, and a value supplied that way is +# ENV-FORCED: FTL logs it as "forced through environment", applies it on every start ahead of +# /etc/pihole/pihole.toml, and marks it read-only in the admin UI so a human cannot drift it. That +# is strictly better than a reconciler — it is enforced at t=0 on a cold start rather than at the +# next cron tick, it cannot be defeated by the UI, and it carries no extra RBAC, image or Job. A +# CronJob here would be ceremony around a knob the app already exposes. (Verified on the live prod +# image, PiHole Core v6.4.3 / FTL v6.7, 2026-09-14.) +# +# THE PVC IS NOT THE MECHANISM AND MUST NOT BE. The `config` volume below mounts at /data/config; +# PiHole's actual state directory is /etc/pihole, which is EPHEMERAL in this Deployment. Anything +# configured through the UI is lost on the next reschedule — which is exactly what happened to the +# upstream servers: the live pod's pihole.toml carried a hand-set `upstreams` marked "### CHANGED" +# that no manifest declared, so a restart would have left PiHole with upstreams = [] and no +# recursion at all. Declaring upstreams here closes that too: PiHole must stay a usable GENERAL +# resolver, not only authoritative for this zone, or pointing LAN clients at it breaks the internet. +env: + # Default is LOCAL, which answers only clients on the same subnet as the listening interface. The + # pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is + # required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is + # the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services, + # enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits + # nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml. + - name: FTLCONF_dns_listeningMode + value: "ALL" + # Recursion for everything that is NOT this zone. Public resolvers on purpose, NOT the SOHO + # router at the LAN gateway: that router is the box whose rebind protection strips RFC1918 + # answers, and forwarding to a device that may forward back here invites a loop. + - name: FTLCONF_dns_upstreams + value: "8.8.8.8;8.8.4.4" + # Per-client query rate limit OFF (default: 1000 queries per 60s, then REFUSED). It has to be off + # because of how the DNS Services are exposed: apps/wave-0/pihole-dns.yaml uses + # externalTrafficPolicy Cluster — it must, or LAN resolution breaks whenever Cilium's L2 + # announcement lease and the pihole pod land on different nodes, which is most of the time — and + # Cilium runs in SNAT mode here, so a query forwarded between nodes reaches PiHole carrying the + # INGRESS NODE's address instead of the client's. Every LAN device therefore counts against ONE + # apparent client's budget, and a normal evening of browsing from a handful of machines would trip + # a limit meant to catch one misbehaving host. Leaving it at the default turns the whole LAN's DNS + # off for 60 seconds at a time, intermittently, which is the worst possible failure shape. The + # limit becomes meaningful again — and should be restored — if Cilium is ever switched to + # `loadBalancer.mode: dsr`, which preserves the client address. + - name: FTLCONF_dns_rateLimit_count + value: "0" + # THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=//` entry answers that domain AND + # every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN — + # so the first line is a wildcard for the whole zone and each later line is a specific override. + # PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both + # line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order + # changes nothing. + # + # A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is + # `