Operator reported DNS flapping between the cluster ingress and the router's public IP. PiHole is not at fault -- both replicas answer 192.168.2.20 consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in CACHEABILITY that we created. Measured: pihole 192.168.2.23 -> 192.168.2.20 TTL=0 public 1.1.1.1 -> wan.home.keithjackson.dev TTL=300 router 192.168.2.1 -> wan.... TTL=153 -> 148 -> 143 (decrementing = caching) dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router forwards LAN queries to PiHole and caches the result, so the CORRECT answer is uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while the WRONG answer caches for 300s. The instant the router falls back to upstream (standard forwarder behaviour when the target does not answer) it latches the WAN address for the whole LAN for five minutes, then expires, re-queries PiHole, gets TTL 0, and is immediately re-exposed. So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That asymmetry, not the gap, is why it reads as "flapping". Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies local-ttl to replies made from configuration, so one directive covers all five address= names. 60s rides out a pod restart and a lease re-election while a real VIP change still propagates inside a minute; these VIPs are pinned and static. apps/wave-0/pihole.yaml regenerated with the canonical render command (--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves); exactly two files changed and the rendered diff is the single local-ttl prefix. UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough line -- see the handoff. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
326 lines
11 KiB
YAML
326 lines
11 KiB
YAML
---
|
|
# Namespace (trust zone; default-deny + policies apply)
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: pihole
|
|
labels:
|
|
homelab.keithjackson.dev/app-namespace: 'true'
|
|
gateway-access: shared
|
|
annotations:
|
|
homelab.keithjackson.dev/backup-policy: daily-offcluster
|
|
---
|
|
# Default-deny baseline (R-NET-01): deny all, allow only DNS
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: default-deny
|
|
namespace: pihole
|
|
labels:
|
|
homelab.keithjackson.dev/baseline: default-deny
|
|
spec:
|
|
podSelector: {}
|
|
policyTypes:
|
|
- Ingress
|
|
- Egress
|
|
ingress: []
|
|
egress:
|
|
- to:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: kube-system
|
|
ports:
|
|
- protocol: UDP
|
|
port: 53
|
|
- protocol: TCP
|
|
port: 53
|
|
---
|
|
# StatefulSet (R-PLAT-02)
|
|
apiVersion: apps/v1
|
|
kind: StatefulSet
|
|
metadata:
|
|
name: pihole
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
replicas: 2
|
|
serviceName: pihole
|
|
podManagementPolicy: Parallel
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: pihole
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
homelab.keithjackson.dev/egress-class: internet
|
|
homelab.keithjackson.dev/auth-mode: forward-auth
|
|
annotations:
|
|
homelab.keithjackson.dev/auth-mode: forward-auth
|
|
homelab.keithjackson.dev/auth-groups: platform-admins
|
|
spec:
|
|
containers:
|
|
- name: pihole
|
|
image: pihole/pihole
|
|
ports:
|
|
- containerPort: 80
|
|
env:
|
|
- name: FTLCONF_dns_listeningMode
|
|
value: ALL
|
|
- name: FTLCONF_dns_upstreams
|
|
value: 8.8.8.8;8.8.4.4
|
|
- name: FTLCONF_dns_rateLimit_count
|
|
value: '0'
|
|
- name: FTLCONF_misc_dnsmasq_lines
|
|
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
|
- name: FTLCONF_webserver_api_password
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: pihole-bootstrap
|
|
key: webui-password
|
|
volumeMounts:
|
|
- name: config
|
|
mountPath: /etc/pihole
|
|
lifecycle:
|
|
postStart:
|
|
exec:
|
|
command:
|
|
- /bin/bash
|
|
- -c
|
|
- "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n [ -n \"$url\" ] || continue\n printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n [ -n \"$url\" ] || continue\n # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n have=$(sql \"select count(*) from adlist where address='$url';\")\n if [ \"$have\" = \"0\" ]; then\n echo \"pihole-adlists: adding $url\"\n sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n CHANGED=1\n else\n en=$(sql \"select enabled from adlist where address='$url';\")\n if [ \"$en\" != \"1\" ]; then\n echo \"pihole-adlists: re-enabling $url\"\n sql \"update adlist set enabled=1 where address='$url';\"\n CHANGED=1\n fi\n fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n [ -n \"$addr\" ] || continue\n if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n echo \"pihole-adlists: disabling undeclared $addr\"\n sql \"update adlist set enabled=0 where address='$addr';\"\n echo changed >> /tmp/adlist-changed\n fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n echo \"pihole-adlists: declared set changed - running gravity update\"\n pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
|
|
- name: oauth2-proxy
|
|
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
|
imagePullPolicy: IfNotPresent
|
|
args:
|
|
- --http-address=0.0.0.0:4180
|
|
- --reverse-proxy=true
|
|
- --provider=oidc
|
|
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
|
- --upstream=http://127.0.0.1:80
|
|
- --email-domain=*
|
|
- --scope=openid email profile
|
|
- --oidc-groups-claim=groups
|
|
- --allowed-group=platform-admins
|
|
- --skip-provider-button=true
|
|
- --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
|
|
- --cookie-name=_oauth2_proxy_pihole
|
|
- --cookie-domain=pihole.${BASE_DOMAIN}
|
|
- --whitelist-domain=.${BASE_DOMAIN}
|
|
- --cookie-secure=true
|
|
- --cookie-refresh=2m30s
|
|
- --pass-access-token=true
|
|
- --set-xauthrequest=true
|
|
- --metrics-address=0.0.0.0:44180
|
|
env:
|
|
- name: OAUTH2_PROXY_CLIENT_ID
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: pihole-oauth2-proxy
|
|
key: client-id
|
|
- name: OAUTH2_PROXY_CLIENT_SECRET
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: pihole-oauth2-proxy
|
|
key: client-secret
|
|
- name: OAUTH2_PROXY_COOKIE_SECRET
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: pihole-oauth2-proxy
|
|
key: cookie-secret
|
|
ports:
|
|
- name: http
|
|
containerPort: 4180
|
|
protocol: TCP
|
|
- name: metrics
|
|
containerPort: 44180
|
|
protocol: TCP
|
|
resources:
|
|
requests:
|
|
cpu: 25m
|
|
memory: 64Mi
|
|
limits:
|
|
memory: 128Mi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /ping
|
|
port: http
|
|
initialDelaySeconds: 5
|
|
periodSeconds: 15
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /ping
|
|
port: http
|
|
initialDelaySeconds: 5
|
|
periodSeconds: 15
|
|
affinity:
|
|
podAntiAffinity:
|
|
requiredDuringSchedulingIgnoredDuringExecution:
|
|
- labelSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: pihole
|
|
topologyKey: kubernetes.io/hostname
|
|
volumeClaimTemplates:
|
|
- metadata:
|
|
name: config
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
storageClassName: longhorn
|
|
resources:
|
|
requests:
|
|
storage: 2Gi
|
|
---
|
|
# PodDisruptionBudget — keep one replica through a drain (availability.replicas)
|
|
apiVersion: policy/v1
|
|
kind: PodDisruptionBudget
|
|
metadata:
|
|
name: pihole
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
minAvailable: 1
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: pihole
|
|
---
|
|
# Service
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: pihole
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
selector:
|
|
app.kubernetes.io/name: pihole
|
|
ports:
|
|
- name: port-80
|
|
port: 80
|
|
targetPort: 80
|
|
- name: authproxy
|
|
port: 4180
|
|
targetPort: 4180
|
|
---
|
|
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): pihole-allow-gateway-ingress
|
|
apiVersion: cilium.io/v2
|
|
kind: CiliumNetworkPolicy
|
|
metadata:
|
|
name: pihole-allow-gateway-ingress
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
endpointSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: pihole
|
|
ingress:
|
|
- fromEntities:
|
|
- ingress
|
|
toPorts:
|
|
- ports:
|
|
- port: '4180'
|
|
protocol: TCP
|
|
---
|
|
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape
|
|
apiVersion: cilium.io/v2
|
|
kind: CiliumNetworkPolicy
|
|
metadata:
|
|
name: pihole-allow-monitoring-scrape
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
endpointSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: pihole
|
|
ingress:
|
|
- fromEndpoints:
|
|
- matchLabels:
|
|
k8s:io.kubernetes.pod.namespace: monitoring
|
|
toPorts:
|
|
- ports:
|
|
- port: '44180'
|
|
protocol: TCP
|
|
---
|
|
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
|
|
apiVersion: cilium.io/v2
|
|
kind: CiliumNetworkPolicy
|
|
metadata:
|
|
name: pihole-allow-internet-egress
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
endpointSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: pihole
|
|
egress:
|
|
- toEntities:
|
|
- world
|
|
- toEndpoints:
|
|
- matchLabels:
|
|
io.kubernetes.pod.namespace: keycloak
|
|
app: keycloak
|
|
---
|
|
# HTTPRoute pihole-80 (R-TLS/R-DNS)
|
|
apiVersion: gateway.networking.k8s.io/v1
|
|
kind: HTTPRoute
|
|
metadata:
|
|
name: pihole-80
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
annotations:
|
|
homelab.keithjackson.dev/auth-mode: forward-auth
|
|
homelab.keithjackson.dev/auth-groups: platform-admins
|
|
external-dns.alpha.kubernetes.io/controller: none
|
|
spec:
|
|
parentRefs:
|
|
- name: shared
|
|
namespace: gateway
|
|
sectionName: https
|
|
hostnames:
|
|
- pihole.${BASE_DOMAIN}
|
|
rules:
|
|
- matches:
|
|
- path:
|
|
type: PathPrefix
|
|
value: /
|
|
backendRefs:
|
|
- name: pihole
|
|
port: 4180
|
|
---
|
|
# PrometheusRule SLI alerts (R-OBS-01)
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: pihole-slis
|
|
namespace: pihole
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
groups:
|
|
- name: pihole.slis
|
|
rules:
|
|
- alert: pihole-down
|
|
expr: kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"})
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
service: pihole
|
|
annotations:
|
|
summary: pihole is down
|
|
description: 'pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'
|