--- # ================================================================================================= # PiHole DNS EXPOSURE — the LAN half of split-horizon (R-DNS-01 / R-DNS-03), and NETWORK POLICY. # ================================================================================================= # HAND-WRITTEN, NOT RENDERED. This file is a non-catalog app-tier unit anchored to the `pihole` # ServiceDefinition and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS # table (the same mechanism gitea-actions uses). It is hand-written because the ServiceDefinition # schema genuinely cannot express it: schemas/servicedefinition.schema.json is # `additionalProperties: false` and has NO field for a type=LoadBalancer Service, for a pinned # lbipam VIP, for a sharing key, for loadBalancerSourceRanges, or for UDP north-south exposure at # all. `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator and # cannot carry DNS. The two existing L4 workloads in this repo are hand-written for exactly this # reason and in exactly this shape — apps/vms/unifi/50-service-l4.yaml and # platform/crafty/service-l4.yaml. The ANSWERS PiHole gives are declared in the catalog # (catalog/pihole.yaml `env`), so the only thing living outside the catalog is the exposure the # schema has no word for. # # WHY THIS EXISTS AT ALL: platform/coredns/coredns-custom.yaml is explicit that its split-horizon is # for PODS and defers LAN clients to "PiHole split-horizon" — but PiHole served no DNS. Its Service # was ClusterIP with ports 80 and 4180 only. The LAN half of the split-horizon that docs/09 # (R-DNS-03) and the README's Phase 2 row both claim simply did not exist, and LAN clients ran on # hand-maintained per-client dnsmasq files instead. Upstream cannot substitute: the SOHO router that # owns this LAN strips RFC1918 answers for public domains as rebind protection (NOERROR, ANSWER 0). # # VIP: ${PIHOLE_DNS_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.56, prod 192.168.2.23), PINNED via # lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool` # (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the # existing CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config # is needed; these Services just consume the pool. PINNING IS NOT OPTIONAL HERE: this address is # typed into every LAN client's resolver configuration (or handed out by the router's DHCP), so it # can never be allowed to move. # # ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so udp/53 and tcp/53 # split into two Services sharing one address via lbipam.cilium.io/sharing-key — the same # arrangement unifi and crafty use. BOTH PROTOCOLS ARE REQUIRED, not belt-and-braces: UDP is the # normal query path, and TCP is what a resolver falls back to for any answer that does not fit in a # UDP datagram and what some clients use unconditionally. A UDP-only resolver fails intermittently # and confusingly. # # NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry # `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their address as a PUBLIC # Cloudflare record. This Service must not: `pihole-dns.${BASE_DOMAIN}` is answered by PiHole itself # (catalog/pihole.yaml's dnsmasq override), it is an RFC1918 address that is useless in public DNS, # and publishing a resolver's address in a public zone invites exactly the traffic the source-range # filter below exists to refuse. Internal name, internal answer, no public record — which is also # why this change touches nothing under platform/externaldns/. # # ------------------------------------------------------------------------------------------------- # externalTrafficPolicy: Cluster — AND WHY IT IS NOT `Local`, WHICH IS WHAT unifi AND crafty USE. # ------------------------------------------------------------------------------------------------- # `Local` is the obvious choice: it preserves the client source IP and, in theory, pins the VIP to # the node running the backend. The second half is NOT TRUE on this cluster, and that was measured, # not assumed (Cilium 1.20.1, prod, 2026-09-14): # # * Cilium's L2 announcer elects one lease holder per Service among the nodes matching the # CiliumL2AnnouncementPolicy's nodeSelector — `default-l2` matches ALL linux nodes — and it does # NOT filter that candidate set by which nodes hold a backend. Created fresh, the two Services # landed on DIFFERENT nodes (`cilium-l2announce-pihole-pihole-dns-tcp` -> k3s-prod-3, # `-udp` -> k3s-prod-2) while the pihole pod ran on k3s-prod-1. # * With `Local`, a node that announces the VIP but holds no backend has nowhere to send the # packet, and the query fails with ICMP port unreachable (dig: "connection refused"). On three # nodes with one replica that is a 1-in-3 chance of working, drawn independently per protocol. # * It does not self-heal. The lease holder keeps RENEWING after the pod moves away: a pod # rescheduled from k3s-prod-1 to k3s-prod-2 left the lease on k3s-prod-1, still renewing three # minutes later, with LAN resolution dead the whole time. Only deleting the Lease forced a # re-election — and that re-election is random, so it may pick a backend-less node again. # # A resolver that works one time in three and stays broken after any reschedule is not a resolver. # `Cluster` makes every node able to serve the VIP by forwarding to wherever the pod is, which is # what makes this survive a reschedule and a rebuild. VERIFIED in exactly the broken arrangement # above — leases on k3s-prod-2 and k3s-prod-3, pod on k3s-prod-1 — udp/53, tcp/53 and recursion all # answering. # # THE COST, STATED PLAINLY: Cilium runs in SNAT mode here (no `loadBalancer.mode: dsr` in # platform/cilium/helmrelease.yaml), so a query forwarded between nodes reaches PiHole with the # INGRESS NODE's address as its source, not the client's. PiHole's per-client query log, per-client # groups and per-client rate limiting therefore see one apparent client instead of the LAN. Two # consequences are handled: the rate limit is disabled in catalog/pihole.yaml (otherwise the whole # LAN counts as one client against PiHole's 1000-queries-per-60s default and PiHole starts # REFUSING), and the LAN restriction is enforced in the datapath rather than by source identity — # see loadBalancerSourceRanges below. The fix that would restore client IPs is `loadBalancer.mode: # dsr` on the Cilium HelmRelease, which is a platform-wide change with an Ansible [parity] contract # and would also fix the same latent defect in unifi and crafty; it belongs in its own change. # # ------------------------------------------------------------------------------------------------- # loadBalancerSourceRanges — THE LAN + VPN RESTRICTION, ENFORCED IN THE DATAPATH. # ------------------------------------------------------------------------------------------------- # Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and # before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} and ${WG_CLIENT_CIDR} is # dropped by the datapath — it never reaches a backend and never reaches a network policy. This is what makes # "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the # range to a /24 that excludes the test client, after which the same query that had been answering # stopped being answered at all, and restoring it, after which it answered again. # # ${WG_CLIENT_CIDR} IS LISTED BECAUSE VPN CLIENTS REACH THIS CHECK UNTRANSLATED. wg0 is one of # Cilium's kube-proxy-replacement devices (measured 2026-11-08: `cilium-dbg status` lists eno1, wg0, # tun0), so a tunnel client's query is load-balanced as it enters from wg0, ahead of the iptables # MASQUERADE in platform/vpn-ingress/daemonset.yaml, and the source tested here is still 10.90.0.x # rather than the node's address. Without this entry a full-tunnel client's DNS is predicted to die # with DROP_NOT_IN_SRC_RANGE (derived from Cilium's source; NOT observed live, because no peer has # ever completed a handshake). The range is private and reachable only by a key in peers.conf, so # PiHole stays off the internet; conformance/tests/l0-static/pihole-dns-source-ranges fails the gate # if either entry goes missing or any range here stops being RFC1918. # # It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this # restriction once the source has been rewritten: measured on this cluster, the forwarded query # arrives from the ingress node's internal router address and Cilium resolves that source to the # reserved `world` identity, which a `fromCIDR` clause does not match even when the CIDR covers the # address (tried: fromCIDR ${CLUSTER_POD_CIDR}, still dropped as `world`). So the peer restriction # lives here, where it is actually enforceable, and the policy below restricts the PORTS. apiVersion: v1 kind: Service metadata: name: pihole-dns-udp namespace: pihole labels: app.kubernetes.io/name: pihole annotations: lbipam.cilium.io/sharing-key: "pihole-dns" lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}" spec: type: LoadBalancer externalTrafficPolicy: Cluster loadBalancerSourceRanges: - ${LAN_CIDR} - ${WG_CLIENT_CIDR} selector: app.kubernetes.io/name: pihole ports: - { name: dns-udp, port: 53, targetPort: 53, protocol: UDP } --- apiVersion: v1 kind: Service metadata: name: pihole-dns-tcp namespace: pihole labels: app.kubernetes.io/name: pihole annotations: lbipam.cilium.io/sharing-key: "pihole-dns" # same key + same IP => shares the VIP with -udp lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}" spec: type: LoadBalancer externalTrafficPolicy: Cluster loadBalancerSourceRanges: - ${LAN_CIDR} - ${WG_CLIENT_CIDR} selector: app.kubernetes.io/name: pihole ports: - { name: dns-tcp, port: 53, targetPort: 53, protocol: TCP } --- # ================================================================================================= # NETWORK POLICY — a NEW ingress path from off-cluster, restricted to the two DNS ports. # ================================================================================================= # The pihole namespace is default-deny on ingress (the baseline NetworkPolicy in the rendered # apps/wave-0/pihole.yaml: `ingress: []`). Cilium enforces the UNION of that NetworkPolicy and the # CiliumNetworkPolicies in the namespace, so this file ADDS one path and nothing else. Everything # undeclared stays denied, and the existing `pihole-allow-gateway-ingress` (Gateway -> :4180, the # oauth2-proxy gate in front of the admin UI) is untouched. PiHole's own :80 is NOT in this rule: # the admin UI stays reachable only through the Gateway and oauth2-proxy, and a LAN client that # talks to ${PIHOLE_DNS_VIP} on any port other than 53 gets nothing — there is no Service port for # it and no policy for it. # # PORTS ARE THE NARROWING HERE; THE LAN RESTRICTION IS THE SOURCE-RANGE FILTER ABOVE. The peer is # `world` for a mechanical reason, not a lazy one: with externalTrafficPolicy Cluster in Cilium's # SNAT mode a forwarded query arrives from the ingress node's internal router address, which Cilium # resolves to the reserved `world` identity — a `fromCIDR` clause does not match it, which was # measured (a fromCIDR of ${CLUSTER_POD_CIDR}, which contains that address, still produced # "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer # selector that can describe LAN traffic that has been through the LoadBalancer, and the set of # addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this # policy is consulted. Together: ${LAN_CIDR} + ${WG_CLIENT_CIDR} in the datapath, udp/53 + tcp/53 # here. A VPN client delivered to a backend on its own ingress node keeps its 10.90.0.x source, # which Cilium also classifies as `world`, so this rule needs no new peer for it. # # This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress # admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all; # this rule admits fewer entities (no `cluster` — in-cluster pods resolve through CoreDNS and none # needs PiHole on :53) on fewer ports, behind a filter those two do not have. # # WHAT REMAINS REACHABLE WITHOUT THE VIP, ACKNOWLEDGED: Cilium applies source ranges to the # LoadBalancer address, not to the NodePorts the same Services allocate, so a host that can route to # a node IP can still reach udp/53 and tcp/53 through the NodePort. On this LAN that is the same # population the source range admits, and nothing at the edge forwards 53 inward, so PiHole is not # an internet-facing open resolver. Say so out loud rather than implying the filter is total. # # `host` and `remote-node` are named explicitly because node-sourced traffic carries a reserved # Cilium identity that no CIDR or `world` selector matches: the kubelet's probes and the # node-to-node forwarding that IS the Cluster-policy datapath both need them. apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: pihole-allow-lan-dns-ingress namespace: pihole labels: app.kubernetes.io/name: pihole spec: endpointSelector: matchLabels: app.kubernetes.io/name: pihole ingress: - fromEntities: - world # LAN + VPN clients, post-LoadBalancer: see the block above for why not fromCIDR - host # this node's kubelet probes - remote-node # the node-to-node hop that forwards a query to wherever the pod runs toPorts: - ports: - { port: "53", protocol: UDP } - { port: "53", protocol: TCP }