feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes
gate / gate (pull_request) Successful in 1m47s
gate / conformance (pull_request) Has been skipped

Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one
node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no
LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO
force-detach), while losing the node holding the pihole-dns L2 lease -- a
DIFFERENT node -- meant tens of seconds. Two of three nodes were single points
of failure with wildly different blast radii.

Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent
only; the mechanism is render.py's, documented in the renderer contract. 1 or
absent renders a Deployment exactly as before -- verified byte-identical across
the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a
multi-replica Deployment never schedules the second pod), per-replica
volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy
Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the
rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever).

Pinned by ~20 new --selftest assertions rather than a conformance change.

ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default
/data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M --
gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn
volume held lost+found and 24K of nothing. So every restart discarded all state
and re-downloaded 72k gravity domains, and backup.policy has been backing up an
empty volume while reporting success. Fixed here because otherwise the new
per-replica volumes would hold nothing while state stayed ephemeral -- and
because persisting gravity.db is what makes a restart cheap.

The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as
uid 0 and chowns the directory recursively (no fsGroup needed), and it calls
migrate_gravity which creates the database when absent.

The gate caught a real regression on the first run: the wave-0 alert unit test
fed pihole a Deployment-shaped series. Updated; promtool SUCCESS.

Touches no sensitive paths, so this one needs no override.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-04 01:34:52 -05:00
1 parent 17f0c11403
commit 028853c87c
7 files changed
+597 -45

No files matched your search

+69
View File
@@ -12,6 +12,42 @@ storage:
- name: config
size: 2Gi
class: longhorn # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
# THE PVC WAS MOUNTED SOMEWHERE PIHOLE NEVER WRITES, AND SO HELD NOTHING AT ALL. Without an
# explicit mountPath the renderer defaults to /data/<name>, i.e. /data/config — but PiHole keeps
# every piece of its state in /etc/pihole. MEASURED on live prod 2026-10-04: `/etc/pihole` (7.1 M
# — gravity.db, pihole.toml, the query DB) was on the **overlay** filesystem, i.e. the
# container's EPHEMERAL layer, while `/dev/longhorn/pvc-849edb86...` sat mounted at
# /data/config holding `lost+found` and 24 K of nothing. Three consequences, all live until now:
# * every restart silently discarded PiHole's whole state, re-downloading 72 k gravity domains
# and regenerating the admin password hash;
# * `backup.policy: daily-offcluster` below has been faithfully backing up an EMPTY volume,
# which is worse than no backup because it reports success;
# * it is also why phase 1's password/adlist declaration mattered more than it first appeared —
# that state was not merely undeclared, it was ephemeral.
# This is the same class of bug catalog/plex.yaml's `config` entry documents ("Without this the
# PVC sat unused and Plex wrote to the container's EPHEMERAL layer").
#
# /etc/pihole IS THE IMAGE'S INTENDED MOUNT POINT, and an EMPTY volume there self-initialises.
# Verified in the running image, not assumed:
# * `/usr/bin/start.sh` (PID 1) calls `migrate_gravity`, whose `migrate_to_database()` creates
# the gravity database when `[ ! -e "$gravityDBfile" ]` — the same branch the postStart hook
# below relies on, and the branch that produced the current ephemeral copy.
# * FTL writes `pihole.toml` itself from the `FTLCONF_*` env above (that is what the
# `### CHANGED (env)` markers in it are).
# * NO `podSecurityContext.fsGroup` is needed and adding one would be inert: `start.sh` line 5
# is `chown -R pihole:pihole /etc/pihole/ /var/log/pihole/` and PID 1 runs as **uid 0**, so
# the image fixes ownership of a fresh root-owned volume before anything else starts. Same
# arrangement as plex's image managing its own /config ownership.
# * the image's own default for `files.macvendor` was moved OUT of this directory (to
# `/macvendor.db`), which is what a maintainer does to a path they expect to be a volume.
#
# WHY IT IS FIXED IN THIS BRANCH rather than left for its own: `availability.replicas: 2` renders
# per-replica volumeClaimTemplates, and with the wrong mountPath those would be two 2 Gi Longhorn
# volumes (times three Longhorn replicas each) holding nothing but `lost+found` while all state
# stayed ephemeral — paying the cost of persistence and getting none of it. It also REMOVES this
# change's main operational risk: with gravity.db persisted, a pod restart is cheap, instead of
# every restart re-downloading 72 k domains inside a wave that has a 5 m timeout.
mountPath: /etc/pihole
dependencies: [] # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
# ------------------------------------------------------------------------------------------------
# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
@@ -114,6 +150,39 @@ env:
secretKeyRef:
name: pihole-bootstrap
key: webui-password
availability:
# TWO REPLICAS, ON DIFFERENT NODES. LAN DNS is a critical network service and it was ONE pod on
# ONE node. Measured 2026-10-04, that cost more than "until it reschedules":
# * the node running the pod dies -> ~40s to mark it NotReady, then `tolerationSeconds: 300`
# before the pod is even evicted, then reschedule, then Longhorn must force-detach the RWO
# volume from a dead node and reattach it. Roughly 6-10 MINUTES with no LAN resolution.
# * the node holding the `pihole-dns-*` L2 lease dies -> Cilium re-elects an announcer, tens of
# seconds. (These were DIFFERENT nodes: lease on k3s-prod-2, pod on k3s-prod-1. Two of three
# nodes were single points of failure for LAN DNS, with wildly different blast radii.)
# The Services already tolerate this: `externalTrafficPolicy: Cluster` on the DNS LoadBalancers
# (apps/wave-0/pihole-dns.yaml) means the L2 lease holder forwards to ANY ready backend, so a
# second replica is picked up with no change there.
#
# WHAT THIS DOES NOT FIX, and it is the bigger risk: LAN DNS still depends on the CLUSTER. etcd
# quorum loss, a Cilium regression, a Longhorn problem or a bad reconcile takes DNS down
# everywhere regardless of replica count. The only thing that removes that dependency is a
# resolver outside the cluster carrying the same `address=` lines. Considered and not chosen
# (operator decision, 2026-10-04); see decisions/2026-10-04-pihole-multi-replica.md.
#
# THE PREREQUISITE WAS DONE FIRST, and this field would have been actively harmful without it. A
# replica gets a FRESH volume, and until 2026-10-04 PiHole's admin password and adlists lived only
# on the PVC — so replica 1 would have come up SERVING DNS AND BLOCKING NOTHING, with a different
# password, and answers would have varied by which pod replied. Both are now declared (the
# password as a SOPS Secret -> FTLCONF_webserver_api_password; the adlists reconciled into
# gravity.db by the postStart hook below), which is what makes a fresh replica identical to its
# sibling. See decisions/2026-10-04-pihole-declare-state.md.
#
# WHY 2 AND NOT 3. Two tolerates the loss of any single node, which is the stated goal, and leaves
# a spare node for the evicted replica to land on under the HARD anti-affinity the renderer emits.
# Three would pin one replica per node with no spare, so a single node failure would leave a
# replica permanently Pending and — `apps-wave-0` being `wait: true` — stall the app tier for as
# long as the node was down. Two is the count that degrades gracefully here.
replicas: 2
ingressRoutes:
- host: pihole.${BASE_DOMAIN}
port: 80 # admin web UI, protected at the Gateway by forward-auth (docs/09)
+283 -10
View File
@@ -272,7 +272,15 @@ def apply_defaults(sd: dict) -> dict:
# object that actually runs on the cluster is not the one this SD renders (keycloak: the
# Keycloak operator runs a StatefulSet, so a Deployment-shaped rule would `absent()` forever).
lw = dict(obs.get("livenessWorkload") or {})
lw.setdefault("kind", "VirtualMachineInstance" if s["kind"] == "vm" else "Deployment")
# The default must track what is ACTUALLY rendered, or the `up` SLI silently dies: a
# Deployment-shaped rule against a StatefulSet `absent()`s forever, which is an alert that can
# never fire — the exact failure mode the keycloak override was added for. Deriving it from
# `availability.replicas` means bumping the replica count cannot leave a stale rule behind.
lw.setdefault(
"kind",
"VirtualMachineInstance" if s["kind"] == "vm"
else ("StatefulSet" if _is_statefulset(s) else "Deployment"),
)
lw.setdefault("name", s["name"])
obs["livenessWorkload"] = lw
s["observability"] = obs
@@ -475,6 +483,37 @@ def r_bulk_pvs(s: dict) -> list:
return out
def _replicas(s: dict) -> int:
"""Declared replica count (schema 1.10.0 `availability.replicas`). Default 1."""
return int((s.get("availability") or {}).get("replicas", 1) or 1)
def _is_statefulset(s: dict) -> bool:
"""True when this container workload must render a StatefulSet rather than a Deployment.
THE DECIDING CONSTRAINT IS THE VOLUME, not a preference. A longhorn storage entry renders
ReadWriteOnce (r_pvcs), and one RWO PVC cannot be attached by pods on two different nodes --
which is exactly what `availability.replicas > 1` plus the node anti-affinity below asks for.
A 2-replica Deployment over a shared RWO claim does not degrade, it simply never schedules the
second pod. A StatefulSet's volumeClaimTemplates give each replica its OWN claim, which is the
only shape that satisfies both at once.
A replicas-1 service keeps rendering a Deployment so every pre-existing golden is byte-identical.
"""
return s["kind"] == "container" and _replicas(s) > 1
def _sts_claim_volumes(s: dict) -> list:
"""Storage entries that become per-replica volumeClaimTemplates on a StatefulSet.
Only the dynamically-provisioned (longhorn) entries. A `bulk` entry is a STATIC, pre-bound NFS
PV over data that already exists outside the cluster and is shared read-only by every consumer
(r_bulk_pvs) -- per-replica copies of it are neither possible nor wanted, so it stays a
standalone PVC and an ordinary pod volume.
"""
return [v for v in s["storage"] if v.get("class", "longhorn") != "bulk"]
def r_pvcs(s: dict) -> list:
"""One PVC per storage entry (R-STATE-01/02/04).
@@ -482,8 +521,15 @@ def r_pvcs(s: dict) -> list:
bulk -> the STATIC external NFS PV rendered by r_bulk_pvs, pre-bound by volumeName.
"""
out = []
# On a StatefulSet the longhorn claims are NOT standalone objects — they are
# volumeClaimTemplates, so each replica gets its own (see _is_statefulset). Emitting both would
# leave an orphan PVC that nothing mounts and that `prune` would later delete out from under
# nobody. `bulk` is unaffected and still rendered here.
sts = _is_statefulset(s)
for vol in s["storage"]:
cls = vol.get("class", "longhorn")
if sts and cls != "bulk":
continue
if cls == "bulk":
# Mirror image of the PV: the same explicitly-empty class (so nothing dynamically
# provisions), the same access mode, and volumeName completing the two-sided binding.
@@ -564,6 +610,12 @@ def r_deployment(s: dict) -> dict:
if vol.get("readOnly"):
mount["readOnly"] = True
volume_mounts.append(mount)
# On a StatefulSet a longhorn volume is supplied by its volumeClaimTemplate, whose
# metadata.name IS the pod volume name. Adding it to pod.volumes as well is a conflict
# (duplicate volume name), so it is skipped here and emitted below. The MOUNT is unchanged —
# the container cannot tell the difference, which is what keeps this transparent to the app.
if _is_statefulset(s) and vol.get("class", "longhorn") != "bulk":
continue
volumes.append(
{
"name": vol["name"],
@@ -661,20 +713,80 @@ def r_deployment(s: dict) -> dict:
if ann:
tmpl_meta["annotations"] = ann
spec = {"replicas": 1}
# An app with persistent storage mounts a ReadWriteOnce Longhorn PVC. The default
# RollingUpdate strategy tries to start the NEW pod before terminating the old one, but
# the old pod still holds the RWO volume -> the new pod is stuck ContainerCreating and
# the rollout deadlocks (single-replica app can never hand the volume over). Recreate
# terminates the old pod first, freeing the PVC for the new one (R-STATE-01/02).
if s["storage"]:
sts = _is_statefulset(s)
# NODE ANTI-AFFINITY, and it is REQUIRED rather than preferred. `availability.replicas > 1`
# exists to survive the loss of one node; `preferred` would let both replicas land on the same
# node and quietly deliver none of that, with every object reporting healthy. The cost is real
# and is accepted: with N replicas the workload needs N schedulable nodes, so on a 3-node
# cluster losing TWO nodes leaves a replica Pending and — because `apps-wave-0` is `wait: true`
# with `apps-wave-1` dependsOn it — STALLS THE APP TIER. That coincides with having already lost
# etcd quorum on a 3-node control plane, i.e. the cluster is not reconciling anything at that
# point anyway, so the stall is a symptom rather than a new failure.
if sts:
pod_spec["affinity"] = {
"podAntiAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": [
{
"labelSelector": {
"matchLabels": {"app.kubernetes.io/name": s["name"]}
},
"topologyKey": "kubernetes.io/hostname",
}
]
}
}
spec = {"replicas": _replicas(s)}
if sts:
# serviceName is a required StatefulSet field. It points at the service rendered by
# r_service; that Service is a normal ClusterIP rather than headless, so per-pod DNS
# (<pod>.<svc>) does NOT resolve — and nothing here needs it. These replicas are
# interchangeable resolvers behind one VIP, not peers that address each other.
spec["serviceName"] = s["name"]
# Parallel, not the OrderedReady default. OrderedReady starts replica N+1 only after N is
# Ready, so a cold start costs the sum of every replica's first-boot work instead of the
# max — and `apps-wave-0` has `timeout: 5m`. There is no ordering requirement between
# replicas to protect (no leader, no migration, no shared volume), so serialising buys
# nothing and risks the wave timing out on a rebuild.
spec["podManagementPolicy"] = "Parallel"
elif s["storage"]:
# An app with persistent storage mounts a ReadWriteOnce Longhorn PVC. The default
# RollingUpdate strategy tries to start the NEW pod before terminating the old one, but
# the old pod still holds the RWO volume -> the new pod is stuck ContainerCreating and
# the rollout deadlocks (single-replica app can never hand the volume over). Recreate
# terminates the old pod first, freeing the PVC for the new one (R-STATE-01/02).
#
# A StatefulSet needs none of this and cannot express it: each replica owns its own claim,
# so a rolling update hands nothing over, and `strategy` is not a StatefulSet field at all.
spec["strategy"] = {"type": "Recreate"}
spec["selector"] = {"matchLabels": {"app.kubernetes.io/name": s["name"]}}
spec["template"] = {"metadata": tmpl_meta, "spec": pod_spec}
if sts:
# Per-replica claims. metadata.name MUST equal the pod volume name the container mounts
# (see the volumes loop above); the resulting PVCs are <name>-<sts>-<ordinal>, e.g.
# config-pihole-0. Same tier and access mode a standalone longhorn PVC gets from r_pvcs —
# this changes WHO owns the claim, never what it is.
tmpls = []
for vol in _sts_claim_volumes(s):
vspec = {"accessModes": ["ReadWriteOnce"], "storageClassName": "longhorn"}
if vol.get("size"):
vspec["resources"] = {"requests": {"storage": vol["size"]}}
tmpls.append(
{
"metadata": {
"name": vol["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": vspec,
}
)
if tmpls:
spec["volumeClaimTemplates"] = tmpls
return {
"apiVersion": "apps/v1",
"kind": "Deployment",
"kind": "StatefulSet" if sts else "Deployment",
"metadata": {
"name": s["name"],
"namespace": s["name"],
@@ -684,6 +796,35 @@ def r_deployment(s: dict) -> dict:
}
def r_pdb(s: dict):
"""PodDisruptionBudget for a multi-replica workload (availability.replicas > 1).
minAvailable 1, not `maxUnavailable: 1`. The budget's job here is to stop a VOLUNTARY
disruption — a node drain for maintenance, a cluster upgrade — from taking the last replica
with it; `minAvailable: 1` says that whatever else happens, one must keep serving. It does
NOT protect against involuntary disruption (a node dying), which is what the anti-affinity
and the replica count are for.
Rendered only when replicas > 1: a PDB over a single-replica workload is actively harmful, as
`minAvailable: 1` would then block every drain of its node forever.
"""
if not _is_statefulset(s):
return None
return {
"apiVersion": "policy/v1",
"kind": "PodDisruptionBudget",
"metadata": {
"name": s["name"],
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": {
"minAvailable": 1,
"selector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
},
}
def r_service(s: dict):
ports = _service_ports(s)
# A declared metrics endpoint becomes a NAMED Service port, because that name is the only
@@ -1817,7 +1958,13 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
)
for pvc in r_pvcs(s):
docs.append((f"PVC {pvc['metadata']['name']} (R-STATE)", pvc))
docs.append(("Deployment (R-PLAT-02)", r_deployment(s)))
wl = r_deployment(s)
docs.append((f"{wl['kind']} (R-PLAT-02)", wl))
pdb = r_pdb(s)
if pdb:
docs.append(
("PodDisruptionBudget — keep one replica through a drain (availability.replicas)", pdb)
)
svc = r_service(s)
if svc:
docs.append(("Service", svc))
@@ -2266,6 +2413,132 @@ def _selftest() -> int:
check("Namespace" in kinds, "container: missing Namespace")
check("NetworkPolicy" in kinds, "container: missing default-deny NetworkPolicy")
check("Deployment" in kinds, "container: missing Deployment")
check(
"PodDisruptionBudget" not in kinds,
"replicas-1 container must render NO PodDisruptionBudget (minAvailable 1 over a single "
"replica blocks every drain of its node forever)",
)
# 1b) availability.replicas (schema 1.10.0). The DEFAULT path must be untouched, and the >1
# path must change workload kind, claim ownership, spreading, budget AND the SLI metric
# together — each of these has a silent-failure mode if it is the one that gets forgotten.
base_av = {
"name": "av",
"source": {"image": "nginx:1"},
"storage": [{"name": "config", "size": "1Gi"}],
"observability": {"metrics": True, "slis": ["up"]},
}
one = render({**base_av, "availability": {"replicas": 1}})
one_kinds = [d[1]["kind"] for d in one]
check(
"Deployment" in one_kinds and "StatefulSet" not in one_kinds,
"availability.replicas=1 must still render a Deployment (back-compat)",
)
check(
"PersistentVolumeClaim" in one_kinds,
"availability.replicas=1 must still render a standalone PVC",
)
check(
"PodDisruptionBudget" not in one_kinds,
"availability.replicas=1 must render no PodDisruptionBudget",
)
many = render({**base_av, "availability": {"replicas": 2}})
many_kinds = [d[1]["kind"] for d in many]
check(
"StatefulSet" in many_kinds and "Deployment" not in many_kinds,
"availability.replicas>1 must render a StatefulSet, not a Deployment (a shared RWO claim "
"cannot be attached on two nodes, so a multi-replica Deployment never schedules)",
)
check(
"PersistentVolumeClaim" not in many_kinds,
"availability.replicas>1 must NOT also render a standalone longhorn PVC — it would be an "
"orphan that nothing mounts",
)
sts = next(d[1] for d in many if d[1]["kind"] == "StatefulSet")
check(sts["spec"].get("replicas") == 2, "StatefulSet must carry the declared replica count")
check(
"strategy" not in sts["spec"],
"StatefulSet must not carry `strategy` (not a StatefulSet field; Recreate is a "
"Deployment-only workaround for handing over one RWO volume)",
)
check(
sts["spec"].get("podManagementPolicy") == "Parallel",
"StatefulSet must set podManagementPolicy=Parallel — OrderedReady serialises first-boot "
"work across replicas and apps-wave-0 has a 5m timeout",
)
check(sts["spec"].get("serviceName") == "av", "StatefulSet must set serviceName")
tmpls = sts["spec"].get("volumeClaimTemplates") or []
check(
[t["metadata"]["name"] for t in tmpls] == ["config"],
"StatefulSet volumeClaimTemplate name must equal the pod volume name the container mounts, "
"or the pod gets a duplicate/unresolvable volume",
)
check(
tmpls and tmpls[0]["spec"]["accessModes"] == ["ReadWriteOnce"]
and tmpls[0]["spec"]["storageClassName"] == "longhorn",
"StatefulSet claim template must keep the same tier/access mode as the standalone PVC",
)
sts_vols = {v["name"] for v in sts["spec"]["template"]["spec"].get("volumes", [])}
check(
"config" not in sts_vols,
"a claim-template volume must NOT also appear in pod.volumes (duplicate volume name)",
)
mounts = sts["spec"]["template"]["spec"]["containers"][0].get("volumeMounts") or []
check(
any(m["name"] == "config" for m in mounts),
"the container must still mount the volume by name on a StatefulSet (transparent to the app)",
)
aff = (sts["spec"]["template"]["spec"].get("affinity") or {}).get("podAntiAffinity") or {}
req = aff.get("requiredDuringSchedulingIgnoredDuringExecution") or []
check(
req and req[0].get("topologyKey") == "kubernetes.io/hostname",
"availability.replicas>1 must render REQUIRED per-node anti-affinity — `preferred` can put "
"every replica on one node and deliver none of the intended node-failure tolerance",
)
pdbs = [d[1] for d in many if d[1]["kind"] == "PodDisruptionBudget"]
check(len(pdbs) == 1, "availability.replicas>1 must render exactly one PodDisruptionBudget")
check(
pdbs and pdbs[0]["spec"].get("minAvailable") == 1,
"PodDisruptionBudget must be minAvailable=1 (keep one replica serving through a drain)",
)
pr_many = next((d[1] for d in many if d[1]["kind"] == "PrometheusRule"), None)
check(
pr_many is not None
and "kube_statefulset_status_replicas_ready"
in pr_many["spec"]["groups"][0]["rules"][0]["expr"],
"the up SLI must follow the rendered workload kind — a Deployment-shaped rule against a "
"StatefulSet absent()s forever and can never fire",
)
# A bulk volume is shared external NFS: it must stay a standalone PVC even on a StatefulSet.
mixed = render(
{
**base_av,
"availability": {"replicas": 2},
"storage": [
{"name": "config", "size": "1Gi"},
{"name": "media", "class": "bulk", "export": "/mnt/x", "size": "1Ti"},
],
}
)
m_sts = next(d[1] for d in mixed if d[1]["kind"] == "StatefulSet")
check(
[t["metadata"]["name"] for t in m_sts["spec"]["volumeClaimTemplates"]] == ["config"],
"only longhorn volumes become claim templates; a bulk share must not be per-replica",
)
check(
any(
d[1]["kind"] == "PersistentVolumeClaim"
and d[1]["metadata"]["name"] == "av-media"
for d in mixed
),
"a bulk PVC must still be rendered standalone on a StatefulSet",
)
check(
"media" in {v["name"] for v in m_sts["spec"]["template"]["spec"].get("volumes", [])},
"a bulk volume must still appear in pod.volumes on a StatefulSet",
)
ns = next(d[1] for d in docs if d[1]["kind"] == "Namespace")
check(