feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes
gate / gate (pull_request) Successful in 1m47s
gate / conformance (pull_request) Has been skipped

Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one
node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no
LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO
force-detach), while losing the node holding the pihole-dns L2 lease -- a
DIFFERENT node -- meant tens of seconds. Two of three nodes were single points
of failure with wildly different blast radii.

Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent
only; the mechanism is render.py's, documented in the renderer contract. 1 or
absent renders a Deployment exactly as before -- verified byte-identical across
the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a
multi-replica Deployment never schedules the second pod), per-replica
volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy
Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the
rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever).

Pinned by ~20 new --selftest assertions rather than a conformance change.

ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default
/data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M --
gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn
volume held lost+found and 24K of nothing. So every restart discarded all state
and re-downloaded 72k gravity domains, and backup.policy has been backing up an
empty volume while reporting success. Fixed here because otherwise the new
per-replica volumes would hold nothing while state stayed ephemeral -- and
because persisting gravity.db is what makes a restart cheap.

The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as
uid 0 and chowns the directory recursively (no fsGroup needed), and it calls
migrate_gravity which creates the database when absent.

The gate caught a real regression on the first run: the wave-0 alert unit test
fed pihole a Deployment-shaped series. Updated; promtool SUCCESS.

Touches no sensitive paths, so this one needs no override.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-04 01:34:52 -05:00
1 parent 17f0c11403
commit 028853c87c
7 files changed
+597 -45

No files matched your search

+8 -6
View File
@@ -1,6 +1,8 @@
# promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
# Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
# qbittorrent (container/Deployment) and vm-demo (KubeVirt VMI). The harness discovers this file by
# qbittorrent (container/Deployment), pihole (container/StatefulSet — `availability.replicas: 2`, so
# its SLI reads kube_statefulset_status_replicas_ready; see schemas/renderer-contract.md
# `### availability`) and vm-demo (KubeVirt VMI). The harness discovers this file by
# the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
# tests run against the exact expressions Flux applies, never a copy.
#
@@ -41,7 +43,7 @@ tests:
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
- series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
values: '1+0x20'
@@ -81,7 +83,7 @@ tests:
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
- series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
values: '1+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
values: '0+0x20'
@@ -132,7 +134,7 @@ tests:
values: '0+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
values: '0+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
- series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
values: '0+0x20'
- series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
values: '1+0x20'
@@ -170,10 +172,10 @@ tests:
severity: critical
service: pihole
namespace: pihole
deployment: pihole
statefulset: pihole
exp_annotations:
summary: "pihole is down"
description: "pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
description: "pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
- eval_time: 10m
alertname: qbittorrent-down
exp_alerts: