---
date: 2026-10-04
branch: feat/pihole-multi-replica
title: PiHole runs two replicas on different nodes — and its config PVC was mounted where PiHole never writes
---

## What changed

Phase 2 of making LAN DNS survive a node failure (phase 1:
`decisions/2026-10-04-pihole-declare-state.md`). PiHole was one pod on one node, which for a
whole-house resolver cost more than "until it reschedules" — measured 2026-10-04:

| Failure | Recovery |
|---|---|
| the node running the pod dies | ~40s to mark NotReady + `tolerationSeconds: 300` before eviction + reschedule + a Longhorn RWO force-detach → **roughly 6-10 minutes with no LAN DNS** |
| the node holding the `pihole-dns-*` L2 lease dies | Cilium re-elects an announcer, tens of seconds |

Those were **different** nodes (lease on k3s-prod-2, pod on k3s-prod-1), so two of three nodes were
single points of failure for LAN DNS with wildly different blast radii.

**A new schema field, because this is intent and not a primitive.** `availability.replicas`
(schema **1.10.0**, mechanism in `schemas/renderer-contract.md` `### availability`). `1` or absent
renders a Deployment exactly as before — every pre-existing golden is byte-identical, verified. `>1`
renders, together:

- a **StatefulSet** instead of a Deployment. This is forced, not preferred: a longhorn PVC is
  **ReadWriteOnce**, and one RWO claim cannot attach on two nodes, so a 2-replica Deployment does not
  degrade — the second pod never schedules.
- longhorn storage as **per-replica `volumeClaimTemplates`** (`config-pihole-0`, `config-pihole-1`);
  the standalone PVC is no longer emitted, since it would be an orphan nothing mounts. A `bulk`
  entry stays a standalone PVC — it is a shared static NFS export, not per-replica state.
- **required** `podAntiAffinity` on `kubernetes.io/hostname`. `preferred` would let both replicas
  land on one node and deliver none of the intended tolerance while reporting healthy.
- `podManagementPolicy: Parallel`, so a cold start costs the max of the replicas' first-boot work
  rather than the sum — `apps-wave-0` has `timeout: 5m`.
- a **PodDisruptionBudget**, `minAvailable: 1`, so a node drain cannot take the last replica.
- `observability.livenessWorkload.kind` defaulting to `StatefulSet`, so the `up` SLI reads
  `kube_statefulset_status_replicas_ready`. A Deployment-shaped rule against a StatefulSet
  `absent()`s forever — an alert that can never fire.

`catalog/pihole.yaml` sets `replicas: 2`, not 3: two tolerates any single node loss and leaves a
spare node for the evicted replica under the hard anti-affinity. Three would pin one per node with no
spare, so a single node failure would leave a replica permanently `Pending` and — `apps-wave-0` being
`wait: true` — stall the app tier for as long as the node was down.

### The bug found on the way, and fixed here

**PiHole's config PVC was mounted at a path PiHole never writes, so it held nothing and all of its
state was ephemeral.** With no explicit `mountPath` the renderer defaults to `/data/<name>`; PiHole
keeps everything in `/etc/pihole`. Measured on live prod 2026-10-04: `/etc/pihole` (7.1 M —
`gravity.db`, `pihole.toml`, the query DB) was on the **overlay** filesystem, while the 2 Gi Longhorn
volume sat at `/data/config` holding `lost+found` and 24 K of nothing. Consequences, all live until
now:

- every restart silently discarded PiHole's entire state and re-downloaded 72 k gravity domains;
- `backup.policy: daily-offcluster` has been faithfully backing up an **empty** volume, which is
  worse than no backup because it reports success;
- it also means phase 1 was understated: that state was not merely undeclared, it was **ephemeral**.

Same class of bug `catalog/plex.yaml` documents for its own `config` entry. Fixed in this branch
rather than its own because with the wrong path the new per-replica volumes would be two 2 Gi
Longhorn volumes (×3 Longhorn replicas each) holding only `lost+found` while all state stayed
ephemeral — paying for persistence and getting none. It also removes this change's main operational
risk: with `gravity.db` persisted, a restart is cheap instead of re-downloading 72 k domains inside a
wave with a 5 m timeout.

## What I verified

- **Back-compatibility, the thing most likely to break quietly:** after the renderer change and
  before `catalog/pihole.yaml` declared anything, the canonical
  `render.py --all catalog -o apps --wave-layout --no-image-automation` produced **zero diff** across
  the whole `apps/` tree. `render-golden` passes.
- **The new behaviour is pinned by ~20 new `render.py --selftest` assertions** (so `ci/render-golden.sh`
  hard-fails on a regression, and no `conformance/**` change was needed): `replicas: 1` still renders a
  Deployment + standalone PVC + no PDB; `replicas: 2` renders a StatefulSet and **no** standalone
  longhorn PVC, carries the replica count, omits `strategy`, sets `Parallel` and `serviceName`, names
  the claim template exactly as the mounted volume, keeps RWO/longhorn on it, does **not** duplicate
  that volume in `pod.volumes`, still mounts it on the container, renders required hostname
  anti-affinity, renders exactly one `minAvailable: 1` PDB, and switches the SLI to
  `kube_statefulset_status_replicas_ready`. A mixed longhorn+bulk case asserts the bulk share stays a
  standalone PVC and an ordinary pod volume.
- **Rendered pihole output, read back:** `replicas: 2`, `serviceName: pihole`,
  `podManagementPolicy: Parallel`, no `strategy`, claim template `config` (RWO/longhorn/2Gi),
  anti-affinity `kubernetes.io/hostname`, `pod.volumes: []`, mount `config -> /etc/pihole`,
  PDB `minAvailable: 1`, SLI `kube_statefulset_status_replicas_ready`.
- **The `/etc/pihole` mount is safe, evidenced from the image's own init rather than assumed:**
  `/usr/bin/start.sh` is PID 1 and runs as **uid 0**; its line 5 is
  `chown -R pihole:pihole /etc/pihole/ /var/log/pihole/`, so a fresh root-owned volume has its
  ownership fixed before anything else starts (**no `fsGroup` needed, and one would be inert**); it
  calls `migrate_gravity`, whose `migrate_to_database()` creates the gravity database when
  `[ ! -e "$gravityDBfile" ]`; FTL writes `pihole.toml` itself from the `FTLCONF_*` env; and the
  image's own default for `files.macvendor` was moved **out** of that directory, which is what a
  maintainer does to a path they expect to be a volume.
- **The PVC really is unused:** `df` showed `/etc/pihole` on `overlay` and
  `/dev/longhorn/pvc-849edb86-...` on `/data/config`; `ls -A /data/config` → `lost+found` only.
- **The gate caught a real regression and it is fixed.** The first run came back **rc 1**:
  `alert-rules-unit-tested` failed because `apps/wave-0/tests/app-slis.test.yaml` fed pihole a
  `kube_deployment_status_replicas_available` series and expected a `deployment: pihole` label. Three
  input series, the firing expectation and the file header were updated;
  `platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml` → **SUCCESS** (6 rules).
- **Schema:** `check-jsonschema` passes over all of `catalog/*.yaml`.
- **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
  `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two environmental skips —
  `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3 (`CLUSTER_AVAILABLE`
  unset). **Not green; reported as rc 3.**
- **No sensitive paths touched**, so unlike the previous two branches this one needs no
  `/approve-override` — confirmed against `infra/gitea/approval-policy.json`.

## What I did NOT verify

- **Nothing has been reconciled.** No StatefulSet exists, no second replica has ever run, the
  anti-affinity has never been evaluated by the scheduler, the PDB has never been consulted, and the
  `/etc/pihole` mount has never been exercised on a real pod. Everything above is the rendered
  artifact plus the image's own source.
- **THE ROLLOUT WILL BRIEFLY DROP LAN DNS, and I have not measured for how long.** `postStart` must
  complete before a container is reported Running, so a pod is not Ready — and the Service has no
  endpoints — until phase 1's adlist hook finishes. On this transition both replicas start with an
  empty `/etc/pihole`, so the hook finds a fresh `gravity.db` with zero adlists and runs a real
  `pihole -g` (72 k domains, network download) before either pod becomes Ready. **Do this at a quiet
  time.** Two mitigating facts: every pihole restart has *already* paid this cost (state was
  ephemeral), and after this change `gravity.db` persists so later restarts should not. If it turns
  out slow, backgrounding the gravity rebuild inside the hook would decouple readiness from it, at
  the cost of a window where DNS resolves but does not block — not done here.
- **Whether `pihole -g` finishes inside `apps-wave-0`'s 5 m `timeout`.** `Parallel` was chosen to
  make this the max rather than the sum of two first boots, but the real duration is unmeasured (it
  was stubbed during phase 1's testing). If the wave times out, the Kustomization reports failed even
  though the pods may come up seconds later.
- **The old PVC's deletion.** `apps-wave-0` has `prune: true`, so removing `pihole-config` from the
  rendered output should delete it. That destroys nothing (it holds `lost+found`), but I have not
  observed the prune, and I did not check whether Longhorn retains the backing volume afterwards.
- **The transition overlap.** For a moment the pruned Deployment's pod and the new StatefulSet pods
  may both match the Service selector. Both are configured identically from git so this should be
  benign — unobserved.
- **Longhorn capacity.** Two 2 Gi claims at 3 Longhorn replicas each is ~12 Gi of real storage where
  there was ~6 Gi. Not checked against free space.
- **Whether two independent query databases are acceptable to the operator.** Stats and query history
  now split per replica, and the admin UI shows only the pod that answered. Adlists and password are
  identical (phase 1), so blocking and auth are consistent; the *reporting* is not.
- **The residual risk is unchanged and is the larger one.** Replicas address node failure. LAN DNS
  still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile — and only a resolver
  outside the cluster carrying the same `address=` lines removes that. Considered and not chosen
  (operator decision, 2026-10-04).
- **`tolerationSeconds: 300` is untouched.** With two replicas the surviving one serves immediately, so
  the 5-minute eviction delay no longer gates recovery — but it still delays the *replacement* replica,
  and during that window there is no redundancy left.