jakman85andClaude Opus 5.5
f13d336630
Add the at-home peer (10.90.0.3) and list ${WG_CLIENT_CIDR} in the PiHole
DNS frontends' loadBalancerSourceRanges. wg0 is a Cilium device, so tunnel
clients reach the source-range check untranslated; a LAN-only list is
predicted to drop their DNS. Adds an L0 guard that both ranges are present
and every range is RFC1918, and corrects the README's SNAT and IPv6 claims.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
211 lines
14 KiB
YAML
211 lines
14 KiB
YAML
---
|
|
# =================================================================================================
|
|
# PiHole DNS EXPOSURE — the LAN half of split-horizon (R-DNS-01 / R-DNS-03), and NETWORK POLICY.
|
|
# =================================================================================================
|
|
# HAND-WRITTEN, NOT RENDERED. This file is a non-catalog app-tier unit anchored to the `pihole`
|
|
# ServiceDefinition and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS
|
|
# table (the same mechanism gitea-actions uses). It is hand-written because the ServiceDefinition
|
|
# schema genuinely cannot express it: schemas/servicedefinition.schema.json is
|
|
# `additionalProperties: false` and has NO field for a type=LoadBalancer Service, for a pinned
|
|
# lbipam VIP, for a sharing key, for loadBalancerSourceRanges, or for UDP north-south exposure at
|
|
# all. `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator and
|
|
# cannot carry DNS. The two existing L4 workloads in this repo are hand-written for exactly this
|
|
# reason and in exactly this shape — apps/vms/unifi/50-service-l4.yaml and
|
|
# platform/crafty/service-l4.yaml. The ANSWERS PiHole gives are declared in the catalog
|
|
# (catalog/pihole.yaml `env`), so the only thing living outside the catalog is the exposure the
|
|
# schema has no word for.
|
|
#
|
|
# WHY THIS EXISTS AT ALL: platform/coredns/coredns-custom.yaml is explicit that its split-horizon is
|
|
# for PODS and defers LAN clients to "PiHole split-horizon" — but PiHole served no DNS. Its Service
|
|
# was ClusterIP with ports 80 and 4180 only. The LAN half of the split-horizon that docs/09
|
|
# (R-DNS-03) and the README's Phase 2 row both claim simply did not exist, and LAN clients ran on
|
|
# hand-maintained per-client dnsmasq files instead. Upstream cannot substitute: the SOHO router that
|
|
# owns this LAN strips RFC1918 answers for public domains as rebind protection (NOERROR, ANSWER 0).
|
|
#
|
|
# VIP: ${PIHOLE_DNS_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.56, prod 192.168.2.23), PINNED via
|
|
# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
|
|
# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the
|
|
# existing CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config
|
|
# is needed; these Services just consume the pool. PINNING IS NOT OPTIONAL HERE: this address is
|
|
# typed into every LAN client's resolver configuration (or handed out by the router's DHCP), so it
|
|
# can never be allowed to move.
|
|
#
|
|
# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so udp/53 and tcp/53
|
|
# split into two Services sharing one address via lbipam.cilium.io/sharing-key — the same
|
|
# arrangement unifi and crafty use. BOTH PROTOCOLS ARE REQUIRED, not belt-and-braces: UDP is the
|
|
# normal query path, and TCP is what a resolver falls back to for any answer that does not fit in a
|
|
# UDP datagram and what some clients use unconditionally. A UDP-only resolver fails intermittently
|
|
# and confusingly.
|
|
#
|
|
# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
|
|
# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their address as a PUBLIC
|
|
# Cloudflare record. This Service must not: `pihole-dns.${BASE_DOMAIN}` is answered by PiHole itself
|
|
# (catalog/pihole.yaml's dnsmasq override), it is an RFC1918 address that is useless in public DNS,
|
|
# and publishing a resolver's address in a public zone invites exactly the traffic the source-range
|
|
# filter below exists to refuse. Internal name, internal answer, no public record — which is also
|
|
# why this change touches nothing under platform/externaldns/.
|
|
#
|
|
# -------------------------------------------------------------------------------------------------
|
|
# externalTrafficPolicy: Cluster — AND WHY IT IS NOT `Local`, WHICH IS WHAT unifi AND crafty USE.
|
|
# -------------------------------------------------------------------------------------------------
|
|
# `Local` is the obvious choice: it preserves the client source IP and, in theory, pins the VIP to
|
|
# the node running the backend. The second half is NOT TRUE on this cluster, and that was measured,
|
|
# not assumed (Cilium 1.20.1, prod, 2026-09-14):
|
|
#
|
|
# * Cilium's L2 announcer elects one lease holder per Service among the nodes matching the
|
|
# CiliumL2AnnouncementPolicy's nodeSelector — `default-l2` matches ALL linux nodes — and it does
|
|
# NOT filter that candidate set by which nodes hold a backend. Created fresh, the two Services
|
|
# landed on DIFFERENT nodes (`cilium-l2announce-pihole-pihole-dns-tcp` -> k3s-prod-3,
|
|
# `-udp` -> k3s-prod-2) while the pihole pod ran on k3s-prod-1.
|
|
# * With `Local`, a node that announces the VIP but holds no backend has nowhere to send the
|
|
# packet, and the query fails with ICMP port unreachable (dig: "connection refused"). On three
|
|
# nodes with one replica that is a 1-in-3 chance of working, drawn independently per protocol.
|
|
# * It does not self-heal. The lease holder keeps RENEWING after the pod moves away: a pod
|
|
# rescheduled from k3s-prod-1 to k3s-prod-2 left the lease on k3s-prod-1, still renewing three
|
|
# minutes later, with LAN resolution dead the whole time. Only deleting the Lease forced a
|
|
# re-election — and that re-election is random, so it may pick a backend-less node again.
|
|
#
|
|
# A resolver that works one time in three and stays broken after any reschedule is not a resolver.
|
|
# `Cluster` makes every node able to serve the VIP by forwarding to wherever the pod is, which is
|
|
# what makes this survive a reschedule and a rebuild. VERIFIED in exactly the broken arrangement
|
|
# above — leases on k3s-prod-2 and k3s-prod-3, pod on k3s-prod-1 — udp/53, tcp/53 and recursion all
|
|
# answering.
|
|
#
|
|
# THE COST, STATED PLAINLY: Cilium runs in SNAT mode here (no `loadBalancer.mode: dsr` in
|
|
# platform/cilium/helmrelease.yaml), so a query forwarded between nodes reaches PiHole with the
|
|
# INGRESS NODE's address as its source, not the client's. PiHole's per-client query log, per-client
|
|
# groups and per-client rate limiting therefore see one apparent client instead of the LAN. Two
|
|
# consequences are handled: the rate limit is disabled in catalog/pihole.yaml (otherwise the whole
|
|
# LAN counts as one client against PiHole's 1000-queries-per-60s default and PiHole starts
|
|
# REFUSING), and the LAN restriction is enforced in the datapath rather than by source identity —
|
|
# see loadBalancerSourceRanges below. The fix that would restore client IPs is `loadBalancer.mode:
|
|
# dsr` on the Cilium HelmRelease, which is a platform-wide change with an Ansible [parity] contract
|
|
# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
|
|
#
|
|
# -------------------------------------------------------------------------------------------------
|
|
# loadBalancerSourceRanges — THE LAN + VPN RESTRICTION, ENFORCED IN THE DATAPATH.
|
|
# -------------------------------------------------------------------------------------------------
|
|
# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
|
|
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} and ${WG_CLIENT_CIDR} is
|
|
# dropped by the datapath — it never reaches a backend and never reaches a network policy. This is what makes
|
|
# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
|
|
# range to a /24 that excludes the test client, after which the same query that had been answering
|
|
# stopped being answered at all, and restoring it, after which it answered again.
|
|
#
|
|
# ${WG_CLIENT_CIDR} IS LISTED BECAUSE VPN CLIENTS REACH THIS CHECK UNTRANSLATED. wg0 is one of
|
|
# Cilium's kube-proxy-replacement devices (measured 2026-11-08: `cilium-dbg status` lists eno1, wg0,
|
|
# tun0), so a tunnel client's query is load-balanced as it enters from wg0, ahead of the iptables
|
|
# MASQUERADE in platform/vpn-ingress/daemonset.yaml, and the source tested here is still 10.90.0.x
|
|
# rather than the node's address. Without this entry a full-tunnel client's DNS is predicted to die
|
|
# with DROP_NOT_IN_SRC_RANGE (derived from Cilium's source; NOT observed live, because no peer has
|
|
# ever completed a handshake). The range is private and reachable only by a key in peers.conf, so
|
|
# PiHole stays off the internet; conformance/tests/l0-static/pihole-dns-source-ranges fails the gate
|
|
# if either entry goes missing or any range here stops being RFC1918.
|
|
#
|
|
# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
|
|
# restriction once the source has been rewritten: measured on this cluster, the forwarded query
|
|
# arrives from the ingress node's internal router address and Cilium resolves that source to the
|
|
# reserved `world` identity, which a `fromCIDR` clause does not match even when the CIDR covers the
|
|
# address (tried: fromCIDR ${CLUSTER_POD_CIDR}, still dropped as `world`). So the peer restriction
|
|
# lives here, where it is actually enforceable, and the policy below restricts the PORTS.
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: pihole-dns-udp
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
annotations:
|
|
lbipam.cilium.io/sharing-key: "pihole-dns"
|
|
lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
|
|
spec:
|
|
type: LoadBalancer
|
|
externalTrafficPolicy: Cluster
|
|
loadBalancerSourceRanges:
|
|
- ${LAN_CIDR}
|
|
- ${WG_CLIENT_CIDR}
|
|
selector:
|
|
app.kubernetes.io/name: pihole
|
|
ports:
|
|
- { name: dns-udp, port: 53, targetPort: 53, protocol: UDP }
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: pihole-dns-tcp
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
annotations:
|
|
lbipam.cilium.io/sharing-key: "pihole-dns" # same key + same IP => shares the VIP with -udp
|
|
lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
|
|
spec:
|
|
type: LoadBalancer
|
|
externalTrafficPolicy: Cluster
|
|
loadBalancerSourceRanges:
|
|
- ${LAN_CIDR}
|
|
- ${WG_CLIENT_CIDR}
|
|
selector:
|
|
app.kubernetes.io/name: pihole
|
|
ports:
|
|
- { name: dns-tcp, port: 53, targetPort: 53, protocol: TCP }
|
|
---
|
|
# =================================================================================================
|
|
# NETWORK POLICY — a NEW ingress path from off-cluster, restricted to the two DNS ports.
|
|
# =================================================================================================
|
|
# The pihole namespace is default-deny on ingress (the baseline NetworkPolicy in the rendered
|
|
# apps/wave-0/pihole.yaml: `ingress: []`). Cilium enforces the UNION of that NetworkPolicy and the
|
|
# CiliumNetworkPolicies in the namespace, so this file ADDS one path and nothing else. Everything
|
|
# undeclared stays denied, and the existing `pihole-allow-gateway-ingress` (Gateway -> :4180, the
|
|
# oauth2-proxy gate in front of the admin UI) is untouched. PiHole's own :80 is NOT in this rule:
|
|
# the admin UI stays reachable only through the Gateway and oauth2-proxy, and a LAN client that
|
|
# talks to ${PIHOLE_DNS_VIP} on any port other than 53 gets nothing — there is no Service port for
|
|
# it and no policy for it.
|
|
#
|
|
# PORTS ARE THE NARROWING HERE; THE LAN RESTRICTION IS THE SOURCE-RANGE FILTER ABOVE. The peer is
|
|
# `world` for a mechanical reason, not a lazy one: with externalTrafficPolicy Cluster in Cilium's
|
|
# SNAT mode a forwarded query arrives from the ingress node's internal router address, which Cilium
|
|
# resolves to the reserved `world` identity — a `fromCIDR` clause does not match it, which was
|
|
# measured (a fromCIDR of ${CLUSTER_POD_CIDR}, which contains that address, still produced
|
|
# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
|
|
# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
|
|
# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
|
|
# policy is consulted. Together: ${LAN_CIDR} + ${WG_CLIENT_CIDR} in the datapath, udp/53 + tcp/53
|
|
# here. A VPN client delivered to a backend on its own ingress node keeps its 10.90.0.x source,
|
|
# which Cilium also classifies as `world`, so this rule needs no new peer for it.
|
|
#
|
|
# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
|
|
# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
|
|
# this rule admits fewer entities (no `cluster` — in-cluster pods resolve through CoreDNS and none
|
|
# needs PiHole on :53) on fewer ports, behind a filter those two do not have.
|
|
#
|
|
# WHAT REMAINS REACHABLE WITHOUT THE VIP, ACKNOWLEDGED: Cilium applies source ranges to the
|
|
# LoadBalancer address, not to the NodePorts the same Services allocate, so a host that can route to
|
|
# a node IP can still reach udp/53 and tcp/53 through the NodePort. On this LAN that is the same
|
|
# population the source range admits, and nothing at the edge forwards 53 inward, so PiHole is not
|
|
# an internet-facing open resolver. Say so out loud rather than implying the filter is total.
|
|
#
|
|
# `host` and `remote-node` are named explicitly because node-sourced traffic carries a reserved
|
|
# Cilium identity that no CIDR or `world` selector matches: the kubelet's probes and the
|
|
# node-to-node forwarding that IS the Cluster-policy datapath both need them.
|
|
apiVersion: cilium.io/v2
|
|
kind: CiliumNetworkPolicy
|
|
metadata:
|
|
name: pihole-allow-lan-dns-ingress
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
endpointSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: pihole
|
|
ingress:
|
|
- fromEntities:
|
|
- world # LAN + VPN clients, post-LoadBalancer: see the block above for why not fromCIDR
|
|
- host # this node's kubelet probes
|
|
- remote-node # the node-to-node hop that forwards a query to wherever the pod runs
|
|
toPorts:
|
|
- ports:
|
|
- { port: "53", protocol: UDP }
|
|
- { port: "53", protocol: TCP }
|