Files
home-network-k3s-impl/apps/wave-0/pihole.yaml
T
jakman85andClaude Opus 5 fda5c31a94
gate / gate (pull_request) Successful in 1m56s
gate / conformance (pull_request) Has been skipped
fix(pihole): make the split-horizon answer cacheable, or one blip latches the WAN IP for 5 minutes
Operator reported DNS flapping between the cluster ingress and the router's
public IP. PiHole is not at fault -- both replicas answer 192.168.2.20
consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in
CACHEABILITY that we created. Measured:

  pihole  192.168.2.23 -> 192.168.2.20               TTL=0
  public  1.1.1.1      -> wan.home.keithjackson.dev  TTL=300
  router  192.168.2.1  -> wan....  TTL=153 -> 148 -> 143   (decrementing = caching)

dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router
forwards LAN queries to PiHole and caches the result, so the CORRECT answer is
uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while
the WRONG answer caches for 300s. The instant the router falls back to upstream
(standard forwarder behaviour when the target does not answer) it latches the
WAN address for the whole LAN for five minutes, then expires, re-queries PiHole,
gets TTL 0, and is immediately re-exposed.

So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured
reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That
asymmetry, not the gap, is why it reads as "flapping".

Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies
local-ttl to replies made from configuration, so one directive covers all five
address= names. 60s rides out a pod restart and a lease re-election while a real
VIP change still propagates inside a minute; these VIPs are pinned and static.

apps/wave-0/pihole.yaml regenerated with the canonical render command
(--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves);
exactly two files changed and the rendered diff is the single local-ttl prefix.

UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected
L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If
the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough
line -- see the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 02:56:49 -05:00

326 lines
11 KiB
YAML

---
# Namespace (trust zone; default-deny + policies apply)
apiVersion: v1
kind: Namespace
metadata:
name: pihole
labels:
homelab.keithjackson.dev/app-namespace: 'true'
gateway-access: shared
annotations:
homelab.keithjackson.dev/backup-policy: daily-offcluster
---
# Default-deny baseline (R-NET-01): deny all, allow only DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: pihole
labels:
homelab.keithjackson.dev/baseline: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
# StatefulSet (R-PLAT-02)
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: pihole
namespace: pihole
labels:
app.kubernetes.io/name: pihole
spec:
replicas: 2
serviceName: pihole
podManagementPolicy: Parallel
selector:
matchLabels:
app.kubernetes.io/name: pihole
template:
metadata:
labels:
app.kubernetes.io/name: pihole
homelab.keithjackson.dev/egress-class: internet
homelab.keithjackson.dev/auth-mode: forward-auth
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: platform-admins
spec:
containers:
- name: pihole
image: pihole/pihole
ports:
- containerPort: 80
env:
- name: FTLCONF_dns_listeningMode
value: ALL
- name: FTLCONF_dns_upstreams
value: 8.8.8.8;8.8.4.4
- name: FTLCONF_dns_rateLimit_count
value: '0'
- name: FTLCONF_misc_dnsmasq_lines
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
- name: FTLCONF_webserver_api_password
valueFrom:
secretKeyRef:
name: pihole-bootstrap
key: webui-password
volumeMounts:
- name: config
mountPath: /etc/pihole
lifecycle:
postStart:
exec:
command:
- /bin/bash
- -c
- "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n [ -n \"$url\" ] || continue\n printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n [ -n \"$url\" ] || continue\n # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n have=$(sql \"select count(*) from adlist where address='$url';\")\n if [ \"$have\" = \"0\" ]; then\n echo \"pihole-adlists: adding $url\"\n sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n CHANGED=1\n else\n en=$(sql \"select enabled from adlist where address='$url';\")\n if [ \"$en\" != \"1\" ]; then\n echo \"pihole-adlists: re-enabling $url\"\n sql \"update adlist set enabled=1 where address='$url';\"\n CHANGED=1\n fi\n fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n [ -n \"$addr\" ] || continue\n if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n echo \"pihole-adlists: disabling undeclared $addr\"\n sql \"update adlist set enabled=0 where address='$addr';\"\n echo changed >> /tmp/adlist-changed\n fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n echo \"pihole-adlists: declared set changed - running gravity update\"\n pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
imagePullPolicy: IfNotPresent
args:
- --http-address=0.0.0.0:4180
- --reverse-proxy=true
- --provider=oidc
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
- --upstream=http://127.0.0.1:80
- --email-domain=*
- --scope=openid email profile
- --oidc-groups-claim=groups
- --allowed-group=platform-admins
- --skip-provider-button=true
- --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
- --cookie-name=_oauth2_proxy_pihole
- --cookie-domain=pihole.${BASE_DOMAIN}
- --whitelist-domain=.${BASE_DOMAIN}
- --cookie-secure=true
- --cookie-refresh=2m30s
- --pass-access-token=true
- --set-xauthrequest=true
- --metrics-address=0.0.0.0:44180
env:
- name: OAUTH2_PROXY_CLIENT_ID
valueFrom:
secretKeyRef:
name: pihole-oauth2-proxy
key: client-id
- name: OAUTH2_PROXY_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: pihole-oauth2-proxy
key: client-secret
- name: OAUTH2_PROXY_COOKIE_SECRET
valueFrom:
secretKeyRef:
name: pihole-oauth2-proxy
key: cookie-secret
ports:
- name: http
containerPort: 4180
protocol: TCP
- name: metrics
containerPort: 44180
protocol: TCP
resources:
requests:
cpu: 25m
memory: 64Mi
limits:
memory: 128Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
livenessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 5
periodSeconds: 15
readinessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 5
periodSeconds: 15
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: pihole
topologyKey: kubernetes.io/hostname
volumeClaimTemplates:
- metadata:
name: config
labels:
app.kubernetes.io/name: pihole
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 2Gi
---
# PodDisruptionBudget — keep one replica through a drain (availability.replicas)
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: pihole
namespace: pihole
labels:
app.kubernetes.io/name: pihole
spec:
minAvailable: 1
selector:
matchLabels:
app.kubernetes.io/name: pihole
---
# Service
apiVersion: v1
kind: Service
metadata:
name: pihole
namespace: pihole
labels:
app.kubernetes.io/name: pihole
spec:
selector:
app.kubernetes.io/name: pihole
ports:
- name: port-80
port: 80
targetPort: 80
- name: authproxy
port: 4180
targetPort: 4180
---
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): pihole-allow-gateway-ingress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: pihole-allow-gateway-ingress
namespace: pihole
labels:
app.kubernetes.io/name: pihole
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: pihole
ingress:
- fromEntities:
- ingress
toPorts:
- ports:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: pihole-allow-monitoring-scrape
namespace: pihole
labels:
app.kubernetes.io/name: pihole
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: pihole
ingress:
- fromEndpoints:
- matchLabels:
k8s:io.kubernetes.pod.namespace: monitoring
toPorts:
- ports:
- port: '44180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: pihole-allow-internet-egress
namespace: pihole
labels:
app.kubernetes.io/name: pihole
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: pihole
egress:
- toEntities:
- world
- toEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: keycloak
app: keycloak
---
# HTTPRoute pihole-80 (R-TLS/R-DNS)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: pihole-80
namespace: pihole
labels:
app.kubernetes.io/name: pihole
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: platform-admins
external-dns.alpha.kubernetes.io/controller: none
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https
hostnames:
- pihole.${BASE_DOMAIN}
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: pihole
port: 4180
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: pihole-slis
namespace: pihole
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: pihole
spec:
groups:
- name: pihole.slis
rules:
- alert: pihole-down
expr: kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"})
for: 5m
labels:
severity: critical
service: pihole
annotations:
summary: pihole is down
description: 'pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'