jakman85andClaude Opus 5
fd43f332d6
platform/coredns/coredns-custom.yaml has always said its split-horizon is for
PODS and deferred LAN clients to "PiHole split-horizon". PiHole served no DNS:
its Service was ClusterIP with ports 80 and 4180 only. The LAN half of R-DNS-03
that docs/09 and the README's Phase 2 row both claim did not exist, so LAN
clients ran on hand-maintained per-client dnsmasq files - one went stale after
the dev decommission and cost the operator sonarr.home.keithjackson.dev. The
SOHO router that owns 192.168.2.0/24 cannot cover for it either: it strips
RFC1918 answers for public domains as rebind protection and returns NOERROR
with no answer.
EXPOSURE (apps/wave-0/pihole-dns.yaml, hand-written, placed into wave 0 by
render.py's EXTRA_UNITS - the schema has no field for a LoadBalancer Service, a
pinned lbipam VIP, a sharing key, source ranges or UDP, the same reason
apps/vms/unifi and platform/crafty hand-write theirs):
- two Services, udp/53 and tcp/53, sharing one pinned VIP
${PIHOLE_DNS_VIP} = 192.168.2.23 on prod, the next free address in the
.20-.30 pool (pool status IPsUsed 3 of 11; .20 Gateway, .21 unifi, .22
crafty; arping from a LAN host got no reply).
- no external-dns hostname annotation: nothing public is created, and
platform/externaldns/ is untouched.
ANSWERS (catalog/pihole.yaml env - declarative, no reconciler): PiHole v6 takes
every FTL setting as an FTLCONF_ environment variable and treats it as
env-forced, applied ahead of pihole.toml on every cold start and read-only in
the UI. That is a real declarative entry point, which the Keycloak and CNPG
CronJob reconcilers exist precisely because their state lacks, so a reconciler
here would be ceremony. misc.dnsmasq_lines carries a wildcard
address=/<zone>/<GATEWAY_VIP> plus one longer-domain override per L4 service
(unifi, crafty-mc, pihole-dns); dnsmasq resolves overlaps by longest matching
domain, so precedence is structural and a newly onboarded Gateway-fronted
service needs no DNS change. dns.upstreams is declared too - the live pod
carried hand-set upstreams that no manifest knew about and that every restart
discarded.
externalTrafficPolicy Cluster, NOT Local, and loadBalancerSourceRanges for the
LAN restriction. Measured on prod, Cilium 1.20.1: the L2 announcer does not
filter lease candidates by which nodes hold a backend - the two Services landed
on k3s-prod-3 and k3s-prod-2 while the pod ran on k3s-prod-1 - and it keeps
renewing after the pod moves away, so with Local a reschedule leaves LAN DNS
dead indefinitely. Cluster serves the VIP from any node; the LAN restriction
moves to loadBalancerSourceRanges, which Cilium enforces in eBPF on the VIP
before load-balancing (verified by narrowing the range to exclude the test
client and back). The cost is SNAT: forwarded queries reach PiHole from the
ingress node, so PiHole's per-client rate limit is disabled in the catalog or
the whole LAN would count as one client against its 1000-per-60s default.
Proven from a LAN host (192.168.2.132) against 192.168.2.23 with the pod and
both leases on three DIFFERENT nodes: sonarr/grafana/nextcloud/keycloak/apex
-> 192.168.2.20, unifi -> .21, crafty-mc -> .22, pihole-dns -> .23, recursion
(cloudflare.com, github.com, debian.org) intact over both udp and tcp, AAAA
NODATA, NXDOMAIN outside the zone, and still answering after a forced
prod-1 -> prod-2 reschedule.
Co-Authored-By: Claude Opus 5 <[email protected]>
121 lines
8.4 KiB
YAML
121 lines
8.4 KiB
YAML
# ServiceDefinition: pihole
|
|
# Source of truth: spec docs/13-service-catalog.md, row 6.
|
|
# internal | egress internet | config on Longhorn (DR-critical config) | auth forward-auth.
|
|
# Also a platform DNS dependency (docs/09). Validates against schemas/servicedefinition.schema.json.
|
|
name: pihole
|
|
kind: container
|
|
source:
|
|
image: pihole/pihole
|
|
exposure: internal # LAN-only admin UI (docs/13 row 6; R-DNS/R-NET)
|
|
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
|
storage:
|
|
- name: config
|
|
size: 2Gi
|
|
class: longhorn # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
|
|
dependencies: [] # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
|
|
# ------------------------------------------------------------------------------------------------
|
|
# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
|
|
# ------------------------------------------------------------------------------------------------
|
|
# coredns-custom answers the ${BASE_DOMAIN} zone for PODS ONLY and says so in its own header ("LAN
|
|
# clients keep their own answers (PiHole split-horizon)"). Until now PiHole served no DNS at all —
|
|
# its Service was ClusterIP :80/:4180, no port 53, no LoadBalancer — so the LAN half did not exist
|
|
# and LAN clients depended on hand-maintained per-client dnsmasq files. One went stale after the dev
|
|
# decommission and cost the operator sonarr.${BASE_DOMAIN}. Upstream cannot cover for it either: the
|
|
# SOHO router that owns this LAN strips RFC1918 answers for public domains as rebind protection and
|
|
# returns NOERROR / ANSWER 0. apps/wave-0/pihole-dns.yaml exposes udp/53 + tcp/53 on
|
|
# ${PIHOLE_DNS_VIP}; these env vars are what make the ANSWERS declarative.
|
|
#
|
|
# WHY env AND NOT A RECONCILER: the repo's level-triggered CronJob reconcilers
|
|
# (platform/keycloak/client-secret-reconciler.yaml, the CNPG grant reconciler in
|
|
# apps/wave-0/nextcloud-db.yaml) exist because their state has NO declarative entry point — a
|
|
# Keycloak client secret lives in Keycloak's own database and a Postgres GRANT lives in the
|
|
# catalog, and nothing in a manifest can express either. PiHole v6 is not that case. Every FTL
|
|
# setting has an `FTLCONF_<section>_<key>` environment variable, and a value supplied that way is
|
|
# ENV-FORCED: FTL logs it as "forced through environment", applies it on every start ahead of
|
|
# /etc/pihole/pihole.toml, and marks it read-only in the admin UI so a human cannot drift it. That
|
|
# is strictly better than a reconciler — it is enforced at t=0 on a cold start rather than at the
|
|
# next cron tick, it cannot be defeated by the UI, and it carries no extra RBAC, image or Job. A
|
|
# CronJob here would be ceremony around a knob the app already exposes. (Verified on the live prod
|
|
# image, PiHole Core v6.4.3 / FTL v6.7, 2026-09-14.)
|
|
#
|
|
# THE PVC IS NOT THE MECHANISM AND MUST NOT BE. The `config` volume below mounts at /data/config;
|
|
# PiHole's actual state directory is /etc/pihole, which is EPHEMERAL in this Deployment. Anything
|
|
# configured through the UI is lost on the next reschedule — which is exactly what happened to the
|
|
# upstream servers: the live pod's pihole.toml carried a hand-set `upstreams` marked "### CHANGED"
|
|
# that no manifest declared, so a restart would have left PiHole with upstreams = [] and no
|
|
# recursion at all. Declaring upstreams here closes that too: PiHole must stay a usable GENERAL
|
|
# resolver, not only authoritative for this zone, or pointing LAN clients at it breaks the internet.
|
|
env:
|
|
# Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
|
|
# pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
|
|
# required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
|
|
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
|
|
# enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
|
|
# nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
|
|
- name: FTLCONF_dns_listeningMode
|
|
value: "ALL"
|
|
# Recursion for everything that is NOT this zone. Public resolvers on purpose, NOT the SOHO
|
|
# router at the LAN gateway: that router is the box whose rebind protection strips RFC1918
|
|
# answers, and forwarding to a device that may forward back here invites a loop.
|
|
- name: FTLCONF_dns_upstreams
|
|
value: "8.8.8.8;8.8.4.4"
|
|
# Per-client query rate limit OFF (default: 1000 queries per 60s, then REFUSED). It has to be off
|
|
# because of how the DNS Services are exposed: apps/wave-0/pihole-dns.yaml uses
|
|
# externalTrafficPolicy Cluster — it must, or LAN resolution breaks whenever Cilium's L2
|
|
# announcement lease and the pihole pod land on different nodes, which is most of the time — and
|
|
# Cilium runs in SNAT mode here, so a query forwarded between nodes reaches PiHole carrying the
|
|
# INGRESS NODE's address instead of the client's. Every LAN device therefore counts against ONE
|
|
# apparent client's budget, and a normal evening of browsing from a handful of machines would trip
|
|
# a limit meant to catch one misbehaving host. Leaving it at the default turns the whole LAN's DNS
|
|
# off for 60 seconds at a time, intermittently, which is the worst possible failure shape. The
|
|
# limit becomes meaningful again — and should be restored — if Cilium is ever switched to
|
|
# `loadBalancer.mode: dsr`, which preserves the client address.
|
|
- name: FTLCONF_dns_rateLimit_count
|
|
value: "0"
|
|
# THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
|
|
# every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
|
|
# so the first line is a wildcard for the whole zone and each later line is a specific override.
|
|
# PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
|
|
# line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
|
|
# changes nothing.
|
|
#
|
|
# A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
|
|
# `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
|
|
# ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
|
|
# platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
|
|
# behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
|
|
# repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
|
|
# is the annotation that means "this Service publishes its own hostname at its own address":
|
|
# apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
|
|
# platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
|
|
# resolver address, are the complete set of exceptions today.
|
|
#
|
|
# AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
|
|
# (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
|
|
# NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
|
|
#
|
|
# Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
|
|
# then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
|
|
# characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
|
|
# while resolution for the zone does not work. Confirmed both ways against the live image.
|
|
- name: FTLCONF_misc_dnsmasq_lines
|
|
value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
|
|
ingressRoutes:
|
|
- host: pihole.${BASE_DOMAIN}
|
|
port: 80 # admin web UI, protected at the Gateway by forward-auth (docs/09)
|
|
auth:
|
|
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
|
|
groups:
|
|
- platform-admins # sensitive admin UI: authenticated AND in group (R-IDP-06)
|
|
update:
|
|
channel: stable # R-UPD-01
|
|
range: ">=2024.0.0"
|
|
cadence: "0 4 * * *"
|
|
autoRollbackOnFailedHealth: true # R-UPD-03
|
|
observability:
|
|
metrics: true
|
|
slis:
|
|
- up # R-OBS-01
|
|
backup:
|
|
policy: daily-offcluster # PiHole config is DR-critical (docs/04 line 118; R-STATE-03)
|