home-network-k3s-impl/decisions/2026-10-04-pihole-multi-replica.md

date, branch, title

date	branch	title
2026-10-04	feat/pihole-multi-replica	PiHole runs two replicas on different nodes — and its config PVC was mounted where PiHole never writes

What changed

Phase 2 of making LAN DNS survive a node failure (phase 1: decisions/2026-10-04-pihole-declare-state.md). PiHole was one pod on one node, which for a whole-house resolver cost more than "until it reschedules" — measured 2026-10-04:

Failure	Recovery
the node running the pod dies	~40s to mark NotReady + tolerationSeconds: 300 before eviction + reschedule + a Longhorn RWO force-detach → roughly 6-10 minutes with no LAN DNS
the node holding the pihole-dns-* L2 lease dies	Cilium re-elects an announcer, tens of seconds

Those were different nodes (lease on k3s-prod-2, pod on k3s-prod-1), so two of three nodes were single points of failure for LAN DNS with wildly different blast radii.

A new schema field, because this is intent and not a primitive. availability.replicas (schema 1.10.0, mechanism in schemas/renderer-contract.md ### availability). 1 or absent renders a Deployment exactly as before — every pre-existing golden is byte-identical, verified. >1 renders, together:

• a StatefulSet instead of a Deployment. This is forced, not preferred: a longhorn PVC is ReadWriteOnce, and one RWO claim cannot attach on two nodes, so a 2-replica Deployment does not degrade — the second pod never schedules.
• longhorn storage as per-replica volumeClaimTemplates (config-pihole-0, config-pihole-1); the standalone PVC is no longer emitted, since it would be an orphan nothing mounts. A bulk entry stays a standalone PVC — it is a shared static NFS export, not per-replica state.
• required podAntiAffinity on kubernetes.io/hostname. preferred would let both replicas land on one node and deliver none of the intended tolerance while reporting healthy.
• podManagementPolicy: Parallel, so a cold start costs the max of the replicas' first-boot work rather than the sum — apps-wave-0 has timeout: 5m.
• a PodDisruptionBudget, minAvailable: 1, so a node drain cannot take the last replica.
• observability.livenessWorkload.kind defaulting to StatefulSet, so the up SLI reads kube_statefulset_status_replicas_ready. A Deployment-shaped rule against a StatefulSet absent()s forever — an alert that can never fire.

catalog/pihole.yaml sets replicas: 2, not 3: two tolerates any single node loss and leaves a spare node for the evicted replica under the hard anti-affinity. Three would pin one per node with no spare, so a single node failure would leave a replica permanently Pending and — apps-wave-0 being wait: true — stall the app tier for as long as the node was down.

The bug found on the way, and fixed here

PiHole's config PVC was mounted at a path PiHole never writes, so it held nothing and all of its state was ephemeral. With no explicit mountPath the renderer defaults to /data/<name>; PiHole keeps everything in /etc/pihole. Measured on live prod 2026-10-04: /etc/pihole (7.1 M — gravity.db, pihole.toml, the query DB) was on the overlay filesystem, while the 2 Gi Longhorn volume sat at /data/config holding lost+found and 24 K of nothing. Consequences, all live until now:

• every restart silently discarded PiHole's entire state and re-downloaded 72 k gravity domains;
• backup.policy: daily-offcluster has been faithfully backing up an empty volume, which is worse than no backup because it reports success;
• it also means phase 1 was understated: that state was not merely undeclared, it was ephemeral.

Same class of bug catalog/plex.yaml documents for its own config entry. Fixed in this branch rather than its own because with the wrong path the new per-replica volumes would be two 2 Gi Longhorn volumes (×3 Longhorn replicas each) holding only lost+found while all state stayed ephemeral — paying for persistence and getting none. It also removes this change's main operational risk: with gravity.db persisted, a restart is cheap instead of re-downloading 72 k domains inside a wave with a 5 m timeout.

What I verified

• Back-compatibility, the thing most likely to break quietly: after the renderer change and before catalog/pihole.yaml declared anything, the canonical render.py --all catalog -o apps --wave-layout --no-image-automation produced zero diff across the whole apps/ tree. render-golden passes.
• The new behaviour is pinned by ~20 new render.py --selftest assertions (so ci/render-golden.sh hard-fails on a regression, and no conformance/** change was needed): replicas: 1 still renders a Deployment + standalone PVC + no PDB; replicas: 2 renders a StatefulSet and no standalone longhorn PVC, carries the replica count, omits strategy, sets Parallel and serviceName, names the claim template exactly as the mounted volume, keeps RWO/longhorn on it, does not duplicate that volume in pod.volumes, still mounts it on the container, renders required hostname anti-affinity, renders exactly one minAvailable: 1 PDB, and switches the SLI to kube_statefulset_status_replicas_ready. A mixed longhorn+bulk case asserts the bulk share stays a standalone PVC and an ordinary pod volume.
• Rendered pihole output, read back: replicas: 2, serviceName: pihole, podManagementPolicy: Parallel, no strategy, claim template config (RWO/longhorn/2Gi), anti-affinity kubernetes.io/hostname, pod.volumes: [], mount config -> /etc/pihole, PDB minAvailable: 1, SLI kube_statefulset_status_replicas_ready.
• The /etc/pihole mount is safe, evidenced from the image's own init rather than assumed: /usr/bin/start.sh is PID 1 and runs as uid 0; its line 5 is chown -R pihole:pihole /etc/pihole/ /var/log/pihole/, so a fresh root-owned volume has its ownership fixed before anything else starts (no fsGroup needed, and one would be inert); it calls migrate_gravity, whose migrate_to_database() creates the gravity database when [ ! -e "$gravityDBfile" ]; FTL writes pihole.toml itself from the FTLCONF_* env; and the image's own default for files.macvendor was moved out of that directory, which is what a maintainer does to a path they expect to be a volume.
• The PVC really is unused: df showed /etc/pihole on overlay and /dev/longhorn/pvc-849edb86-... on /data/config; ls -A /data/config → lost+found only.
• The gate caught a real regression and it is fixed. The first run came back rc 1: alert-rules-unit-tested failed because apps/wave-0/tests/app-slis.test.yaml fed pihole a kube_deployment_status_replicas_available series and expected a deployment: pihole label. Three input series, the firing expectation and the file header were updated; platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml → SUCCESS (6 rules).
• Schema: check-jsonschema passes over all of catalog/*.yaml.
• Toolbox gate: rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1, approval-policy all pass. The rc 3 is the two environmental skips — gitea-settings (no local SETTINGS_API_TOKEN) and conformance L2/L3 (CLUSTER_AVAILABLE unset). Not green; reported as rc 3.
• No sensitive paths touched, so unlike the previous two branches this one needs no /approve-override — confirmed against infra/gitea/approval-policy.json.

What I did NOT verify

• Nothing has been reconciled. No StatefulSet exists, no second replica has ever run, the anti-affinity has never been evaluated by the scheduler, the PDB has never been consulted, and the /etc/pihole mount has never been exercised on a real pod. Everything above is the rendered artifact plus the image's own source.
• THE ROLLOUT WILL BRIEFLY DROP LAN DNS, and I have not measured for how long. postStart must complete before a container is reported Running, so a pod is not Ready — and the Service has no endpoints — until phase 1's adlist hook finishes. On this transition both replicas start with an empty /etc/pihole, so the hook finds a fresh gravity.db with zero adlists and runs a real pihole -g (72 k domains, network download) before either pod becomes Ready. Do this at a quiet time. Two mitigating facts: every pihole restart has already paid this cost (state was ephemeral), and after this change gravity.db persists so later restarts should not. If it turns out slow, backgrounding the gravity rebuild inside the hook would decouple readiness from it, at the cost of a window where DNS resolves but does not block — not done here.
• Whether pihole -g finishes inside apps-wave-0's 5 m timeout. Parallel was chosen to make this the max rather than the sum of two first boots, but the real duration is unmeasured (it was stubbed during phase 1's testing). If the wave times out, the Kustomization reports failed even though the pods may come up seconds later.
• The old PVC's deletion. apps-wave-0 has prune: true, so removing pihole-config from the rendered output should delete it. That destroys nothing (it holds lost+found), but I have not observed the prune, and I did not check whether Longhorn retains the backing volume afterwards.
• The transition overlap. For a moment the pruned Deployment's pod and the new StatefulSet pods may both match the Service selector. Both are configured identically from git so this should be benign — unobserved.
• Longhorn capacity. Two 2 Gi claims at 3 Longhorn replicas each is ~12 Gi of real storage where there was ~6 Gi. Not checked against free space.
• Whether two independent query databases are acceptable to the operator. Stats and query history now split per replica, and the admin UI shows only the pod that answered. Adlists and password are identical (phase 1), so blocking and auth are consistent; the reporting is not.
• The residual risk is unchanged and is the larger one. Replicas address node failure. LAN DNS still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile — and only a resolver outside the cluster carrying the same address= lines removes that. Considered and not chosen (operator decision, 2026-10-04).
• tolerationSeconds: 300 is untouched. With two replicas the surviving one serves immediately, so the 5-minute eviction delay no longer gates recovery — but it still delays the replacement replica, and during that window there is no redundancy left.