platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that have never once scraped. Measured on prod 2026-10-04: all six read health=down with lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the configured scrapeTimeout, i.e. packets DROPPED rather than refused. Cause: every app namespace ships a default-deny NetworkPolicy with policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180. Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr, qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object. render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this catalog declares one. It was wired in at the call site and had never emitted a single object on any branch: correct, and entirely latent. It now also fires for the forward-auth gate sidecar's own metrics port, keyed on _needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the sidecar exists. Keying on observability.metrics instead would let metrics: false produce a discovered-but-blocked target: the same "two halves authored apart" failure this function exists to prevent. Rendered here rather than in platform/oauth2-proxy because that tier would have to name app namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering per-app dissolves it. Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring. apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved. Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an object necessarily drifts the goldens that pin the renderer's output. The first gate run caught container-workload-render and egress-class-render as FAIL with exactly this CNP as the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the evidence that the regen is faithful and not a rubber stamp. That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author approving review. The local gate cannot see that — offline it reports approval-policy green (N/A). Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements. Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
326 lines
11 KiB
YAML
326 lines
11 KiB
YAML
---
|
|
# Namespace (trust zone; default-deny + policies apply)
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: pihole
|
|
labels:
|
|
homelab.keithjackson.dev/app-namespace: 'true'
|
|
gateway-access: shared
|
|
annotations:
|
|
homelab.keithjackson.dev/backup-policy: daily-offcluster
|
|
---
|
|
# Default-deny baseline (R-NET-01): deny all, allow only DNS
|
|
apiVersion: networking.k8s.io/v1
|
|
kind: NetworkPolicy
|
|
metadata:
|
|
name: default-deny
|
|
namespace: pihole
|
|
labels:
|
|
homelab.keithjackson.dev/baseline: default-deny
|
|
spec:
|
|
podSelector: {}
|
|
policyTypes:
|
|
- Ingress
|
|
- Egress
|
|
ingress: []
|
|
egress:
|
|
- to:
|
|
- namespaceSelector:
|
|
matchLabels:
|
|
kubernetes.io/metadata.name: kube-system
|
|
ports:
|
|
- protocol: UDP
|
|
port: 53
|
|
- protocol: TCP
|
|
port: 53
|
|
---
|
|
# StatefulSet (R-PLAT-02)
|
|
apiVersion: apps/v1
|
|
kind: StatefulSet
|
|
metadata:
|
|
name: pihole
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
replicas: 2
|
|
serviceName: pihole
|
|
podManagementPolicy: Parallel
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: pihole
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
homelab.keithjackson.dev/egress-class: internet
|
|
homelab.keithjackson.dev/auth-mode: forward-auth
|
|
annotations:
|
|
homelab.keithjackson.dev/auth-mode: forward-auth
|
|
homelab.keithjackson.dev/auth-groups: platform-admins
|
|
spec:
|
|
containers:
|
|
- name: pihole
|
|
image: pihole/pihole
|
|
ports:
|
|
- containerPort: 80
|
|
env:
|
|
- name: FTLCONF_dns_listeningMode
|
|
value: ALL
|
|
- name: FTLCONF_dns_upstreams
|
|
value: 8.8.8.8;8.8.4.4
|
|
- name: FTLCONF_dns_rateLimit_count
|
|
value: '0'
|
|
- name: FTLCONF_misc_dnsmasq_lines
|
|
value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
|
|
- name: FTLCONF_webserver_api_password
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: pihole-bootstrap
|
|
key: webui-password
|
|
volumeMounts:
|
|
- name: config
|
|
mountPath: /etc/pihole
|
|
lifecycle:
|
|
postStart:
|
|
exec:
|
|
command:
|
|
- /bin/bash
|
|
- -c
|
|
- "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n [ -n \"$url\" ] || continue\n printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n [ -n \"$url\" ] || continue\n # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n have=$(sql \"select count(*) from adlist where address='$url';\")\n if [ \"$have\" = \"0\" ]; then\n echo \"pihole-adlists: adding $url\"\n sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n CHANGED=1\n else\n en=$(sql \"select enabled from adlist where address='$url';\")\n if [ \"$en\" != \"1\" ]; then\n echo \"pihole-adlists: re-enabling $url\"\n sql \"update adlist set enabled=1 where address='$url';\"\n CHANGED=1\n fi\n fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n [ -n \"$addr\" ] || continue\n if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n echo \"pihole-adlists: disabling undeclared $addr\"\n sql \"update adlist set enabled=0 where address='$addr';\"\n echo changed >> /tmp/adlist-changed\n fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n echo \"pihole-adlists: declared set changed - running gravity update\"\n pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
|
|
- name: oauth2-proxy
|
|
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
|
imagePullPolicy: IfNotPresent
|
|
args:
|
|
- --http-address=0.0.0.0:4180
|
|
- --reverse-proxy=true
|
|
- --provider=oidc
|
|
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
|
- --upstream=http://127.0.0.1:80
|
|
- --email-domain=*
|
|
- --scope=openid email profile
|
|
- --oidc-groups-claim=groups
|
|
- --allowed-group=platform-admins
|
|
- --skip-provider-button=true
|
|
- --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
|
|
- --cookie-name=_oauth2_proxy_pihole
|
|
- --cookie-domain=pihole.${BASE_DOMAIN}
|
|
- --whitelist-domain=.${BASE_DOMAIN}
|
|
- --cookie-secure=true
|
|
- --cookie-refresh=2m30s
|
|
- --pass-access-token=true
|
|
- --set-xauthrequest=true
|
|
- --metrics-address=0.0.0.0:44180
|
|
env:
|
|
- name: OAUTH2_PROXY_CLIENT_ID
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: pihole-oauth2-proxy
|
|
key: client-id
|
|
- name: OAUTH2_PROXY_CLIENT_SECRET
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: pihole-oauth2-proxy
|
|
key: client-secret
|
|
- name: OAUTH2_PROXY_COOKIE_SECRET
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: pihole-oauth2-proxy
|
|
key: cookie-secret
|
|
ports:
|
|
- name: http
|
|
containerPort: 4180
|
|
protocol: TCP
|
|
- name: metrics
|
|
containerPort: 44180
|
|
protocol: TCP
|
|
resources:
|
|
requests:
|
|
cpu: 25m
|
|
memory: 64Mi
|
|
limits:
|
|
memory: 128Mi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /ping
|
|
port: http
|
|
initialDelaySeconds: 5
|
|
periodSeconds: 15
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /ping
|
|
port: http
|
|
initialDelaySeconds: 5
|
|
periodSeconds: 15
|
|
affinity:
|
|
podAntiAffinity:
|
|
requiredDuringSchedulingIgnoredDuringExecution:
|
|
- labelSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: pihole
|
|
topologyKey: kubernetes.io/hostname
|
|
volumeClaimTemplates:
|
|
- metadata:
|
|
name: config
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
storageClassName: longhorn
|
|
resources:
|
|
requests:
|
|
storage: 2Gi
|
|
---
|
|
# PodDisruptionBudget — keep one replica through a drain (availability.replicas)
|
|
apiVersion: policy/v1
|
|
kind: PodDisruptionBudget
|
|
metadata:
|
|
name: pihole
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
minAvailable: 1
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: pihole
|
|
---
|
|
# Service
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: pihole
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
selector:
|
|
app.kubernetes.io/name: pihole
|
|
ports:
|
|
- name: port-80
|
|
port: 80
|
|
targetPort: 80
|
|
- name: authproxy
|
|
port: 4180
|
|
targetPort: 4180
|
|
---
|
|
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): pihole-allow-gateway-ingress
|
|
apiVersion: cilium.io/v2
|
|
kind: CiliumNetworkPolicy
|
|
metadata:
|
|
name: pihole-allow-gateway-ingress
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
endpointSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: pihole
|
|
ingress:
|
|
- fromEntities:
|
|
- ingress
|
|
toPorts:
|
|
- ports:
|
|
- port: '4180'
|
|
protocol: TCP
|
|
---
|
|
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape
|
|
apiVersion: cilium.io/v2
|
|
kind: CiliumNetworkPolicy
|
|
metadata:
|
|
name: pihole-allow-monitoring-scrape
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
endpointSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: pihole
|
|
ingress:
|
|
- fromEndpoints:
|
|
- matchLabels:
|
|
k8s:io.kubernetes.pod.namespace: monitoring
|
|
toPorts:
|
|
- ports:
|
|
- port: '44180'
|
|
protocol: TCP
|
|
---
|
|
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
|
|
apiVersion: cilium.io/v2
|
|
kind: CiliumNetworkPolicy
|
|
metadata:
|
|
name: pihole-allow-internet-egress
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
endpointSelector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: pihole
|
|
egress:
|
|
- toEntities:
|
|
- world
|
|
- toEndpoints:
|
|
- matchLabels:
|
|
io.kubernetes.pod.namespace: keycloak
|
|
app: keycloak
|
|
---
|
|
# HTTPRoute pihole-80 (R-TLS/R-DNS)
|
|
apiVersion: gateway.networking.k8s.io/v1
|
|
kind: HTTPRoute
|
|
metadata:
|
|
name: pihole-80
|
|
namespace: pihole
|
|
labels:
|
|
app.kubernetes.io/name: pihole
|
|
annotations:
|
|
homelab.keithjackson.dev/auth-mode: forward-auth
|
|
homelab.keithjackson.dev/auth-groups: platform-admins
|
|
external-dns.alpha.kubernetes.io/controller: none
|
|
spec:
|
|
parentRefs:
|
|
- name: shared
|
|
namespace: gateway
|
|
sectionName: https
|
|
hostnames:
|
|
- pihole.${BASE_DOMAIN}
|
|
rules:
|
|
- matches:
|
|
- path:
|
|
type: PathPrefix
|
|
value: /
|
|
backendRefs:
|
|
- name: pihole
|
|
port: 4180
|
|
---
|
|
# PrometheusRule SLI alerts (R-OBS-01)
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: pihole-slis
|
|
namespace: pihole
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
app.kubernetes.io/name: pihole
|
|
spec:
|
|
groups:
|
|
- name: pihole.slis
|
|
rules:
|
|
- alert: pihole-down
|
|
expr: kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"})
|
|
for: 5m
|
|
labels:
|
|
severity: critical
|
|
service: pihole
|
|
annotations:
|
|
summary: pihole is down
|
|
description: 'pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'
|