feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes #45

2026-10-04T06:35:34Z

gitea-admin commented

2026-10-04 06:35:34 +00:00

Phase 2 of making LAN DNS survive a node failure (phase 1). Pi-hole was one pod on one node, which for a whole-house resolver cost more than "until it reschedules" — measured 2026-10-04:

Failure	Recovery
the node running the pod dies	~40s to mark NotReady + `tolerationSeconds: 300` before eviction + reschedule + a Longhorn RWO force-detach → ~6–10 minutes with no LAN DNS
the node holding the `pihole-dns-*` L2 lease dies	Cilium re-elects, tens of seconds

Those were different nodes (lease on prod-2, pod on prod-1) — two of three nodes were single points of failure, with wildly different blast radii.

The new field

availability.replicas (schema 1.10.0; mechanism in schemas/renderer-contract.md ### availability). Intent only — it names no Kubernetes object. 1/absent renders a Deployment exactly as before. >1 renders, together:

a StatefulSet — forced, not preferred: a longhorn PVC is ReadWriteOnce, so a 2-replica Deployment doesn't degrade, it simply never schedules the second pod;
per-replica volumeClaimTemplates (config-pihole-0/1); the standalone PVC is no longer emitted (it would be an orphan). A bulk entry stays standalone — shared static NFS isn't per-replica state;
required hostname podAntiAffinity — preferred would let both replicas land on one node and deliver none of the tolerance while reporting healthy;
podManagementPolicy: Parallel — cold start costs the max, not the sum, of first boots (apps-wave-0 has timeout: 5m);
a PDB, minAvailable: 1, so a drain can't take the last replica;
an SLI that follows the rendered kind — a Deployment-shaped rule against a StatefulSet absent()s forever and can never fire.

replicas: 2, not 3: two tolerates any single node loss and leaves a spare node for the evicted replica under the hard anti-affinity. Three would pin one per node with no spare, so one node down would leave a replica permanently Pending and stall the app tier.

⚠️ A live bug found on the way, and fixed here

Pi-hole's config PVC was mounted where Pi-hole never writes, so it held nothing and all state was ephemeral. With no explicit mountPath the renderer defaults to /data/<name>; Pi-hole writes to /etc/pihole. Measured: /etc/pihole (7.1 MB — gravity.db, pihole.toml, query DB) was on the overlay filesystem, while the 2 GB Longhorn volume sat at /data/config holding lost+found and 24 KB of nothing.

Every restart silently discarded all state and re-downloaded 72k gravity domains.
backup.policy: daily-offcluster has been backing up an empty volume — worse than no backup, because it reports success.
It also means phase 1 was understated: that state wasn't merely undeclared, it was ephemeral.

Same class of bug catalog/plex.yaml documents for its own config. Fixed here because otherwise the new per-replica volumes would hold only lost+found while state stayed ephemeral — paying for persistence and getting none — and because persisting gravity.db is what makes a restart cheap instead of a 72k re-download inside a 5-minute wave.

Safe on the image's own evidence: start.sh is PID 1 as uid 0 and line 5 is chown -R pihole:pihole /etc/pihole/, so a fresh root-owned volume gets fixed before anything starts (no fsGroup needed; one would be inert); it calls migrate_gravity, which creates the DB when absent; FTL writes pihole.toml from FTLCONF_*; and the image moved files.macvendor out of that directory, which is what you do to a path you expect to be a volume.

Verified

Back-compat, the thing most likely to break quietly: after the renderer change and before pihole declared anything, the canonical render produced zero diff across all of apps/.
~20 new render.py --selftest assertions pin the new behaviour (so render-golden.sh hard-fails on regression — and no conformance/** change was needed): replicas:1 → Deployment + standalone PVC + no PDB; replicas:2 → StatefulSet, no standalone longhorn PVC, correct replica count, no strategy, Parallel, serviceName, claim-template name == mounted volume name, RWO/longhorn preserved, volume not duplicated in pod.volumes, still mounted, required hostname anti-affinity, exactly one minAvailable:1 PDB, StatefulSet SLI metric. A mixed longhorn+bulk case asserts the bulk share stays standalone.
The gate caught a real regression. First run came back rc 1: alert-rules-unit-tested failed because the wave-0 test fed pihole a kube_deployment_status_replicas_available series and expected a deployment: pihole label. Fixed; promtool → SUCCESS (6 rules).
Gate: rc 3 — all of spec-immutability, decisions-log, lint, l0, render-golden, l1, approval-policy pass; rc 3 is the two environmental skips (gitea-settings no local token, conformance no CLUSTER_AVAILABLE). Not green; reported as rc 3.
Touches no sensitive paths, so unlike #43 and #44 this needs no /approve-override.

⚠️ Read before merging — the rollout will briefly drop LAN DNS

postStart must finish before a container reports Running, so a pod isn't Ready until phase 1's adlist hook completes. On this transition both replicas start with an empty /etc/pihole, so the hook sees a fresh gravity.db with zero adlists and runs a real pihole -g (72k domains) before either pod is Ready. Do this at a quiet time.

Two mitigating facts: every pihole restart has already paid this cost (state was ephemeral), and after this gravity.db persists so later restarts shouldn't. If it proves slow, backgrounding the rebuild would decouple readiness from it — at the cost of a window where DNS resolves but doesn't block. Not done here.

Other things I did not verify

Nothing is reconciled. No StatefulSet exists; anti-affinity, the PDB and the /etc/pihole mount have never been exercised on a real pod.
Whether pihole -g finishes inside the 5m wave timeout — real duration unmeasured.
The old PVC's prune. prune: true should delete pihole-config; it holds only lost+found so nothing is lost, but I haven't observed it, nor whether Longhorn retains the backing volume.
Longhorn capacity: 2 × 2Gi at 3 replicas each is ~12Gi where there was ~6Gi. Not checked against free space.
Query history now splits per replica — the admin UI shows only the pod that answered. Adlists and password are identical (phase 1), so blocking and auth are consistent; reporting isn't.
The residual risk is unchanged and larger. Replicas fix node failure. LAN DNS still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile. Only an out-of-cluster resolver carrying the same address= lines removes that; considered and not chosen.

🤖 Generated with Claude Code

**Phase 2** of making LAN DNS survive a node failure ([phase 1](../pulls/44)). Pi-hole was one pod on one node, which for a whole-house resolver cost more than "until it reschedules" — measured 2026-10-04: | Failure | Recovery | |---|---| | the node running the pod dies | ~40s to mark NotReady + `tolerationSeconds: 300` before eviction + reschedule + a Longhorn RWO force-detach → **~6–10 minutes with no LAN DNS** | | the node holding the `pihole-dns-*` L2 lease dies | Cilium re-elects, tens of seconds | Those were **different** nodes (lease on prod-2, pod on prod-1) — two of three nodes were single points of failure, with wildly different blast radii. ### The new field `availability.replicas` (schema **1.10.0**; mechanism in `schemas/renderer-contract.md` `### availability`). Intent only — it names no Kubernetes object. `1`/absent renders a Deployment **exactly** as before. `>1` renders, together: - a **StatefulSet** — forced, not preferred: a longhorn PVC is **ReadWriteOnce**, so a 2-replica Deployment doesn't degrade, it simply never schedules the second pod; - per-replica **`volumeClaimTemplates`** (`config-pihole-0/1`); the standalone PVC is no longer emitted (it would be an orphan). A `bulk` entry stays standalone — shared static NFS isn't per-replica state; - **required** hostname `podAntiAffinity` — `preferred` would let both replicas land on one node and deliver none of the tolerance while reporting healthy; - `podManagementPolicy: Parallel` — cold start costs the max, not the sum, of first boots (`apps-wave-0` has `timeout: 5m`); - a **PDB**, `minAvailable: 1`, so a drain can't take the last replica; - an SLI that follows the rendered kind — a Deployment-shaped rule against a StatefulSet `absent()`s forever and **can never fire**. `replicas: 2`, not 3: two tolerates any single node loss *and* leaves a spare node for the evicted replica under the hard anti-affinity. Three would pin one per node with no spare, so one node down would leave a replica permanently `Pending` and stall the app tier. ### ⚠️ A live bug found on the way, and fixed here **Pi-hole's config PVC was mounted where Pi-hole never writes, so it held nothing and all state was ephemeral.** With no explicit `mountPath` the renderer defaults to `/data/<name>`; Pi-hole writes to `/etc/pihole`. Measured: `/etc/pihole` (7.1 MB — `gravity.db`, `pihole.toml`, query DB) was on the **overlay** filesystem, while the 2 GB Longhorn volume sat at `/data/config` holding `lost+found` and 24 KB of nothing. - Every restart silently discarded all state and re-downloaded 72k gravity domains. - **`backup.policy: daily-offcluster` has been backing up an empty volume** — worse than no backup, because it reports success. - It also means phase 1 was understated: that state wasn't merely undeclared, it was *ephemeral*. Same class of bug `catalog/plex.yaml` documents for its own `config`. Fixed here because otherwise the new per-replica volumes would hold only `lost+found` while state stayed ephemeral — paying for persistence and getting none — and because persisting `gravity.db` is what makes a restart cheap instead of a 72k re-download inside a 5-minute wave. Safe on the image's own evidence: `start.sh` is PID 1 as **uid 0** and line 5 is `chown -R pihole:pihole /etc/pihole/`, so a fresh root-owned volume gets fixed before anything starts (**no `fsGroup` needed; one would be inert**); it calls `migrate_gravity`, which creates the DB when absent; FTL writes `pihole.toml` from `FTLCONF_*`; and the image moved `files.macvendor` *out* of that directory, which is what you do to a path you expect to be a volume. ### Verified - **Back-compat, the thing most likely to break quietly:** after the renderer change and before pihole declared anything, the canonical render produced **zero diff** across all of `apps/`. - **~20 new `render.py --selftest` assertions** pin the new behaviour (so `render-golden.sh` hard-fails on regression — and no `conformance/**` change was needed): `replicas:1` → Deployment + standalone PVC + no PDB; `replicas:2` → StatefulSet, **no** standalone longhorn PVC, correct replica count, no `strategy`, `Parallel`, `serviceName`, claim-template name == mounted volume name, RWO/longhorn preserved, volume **not** duplicated in `pod.volumes`, still mounted, required hostname anti-affinity, exactly one `minAvailable:1` PDB, StatefulSet SLI metric. A mixed longhorn+bulk case asserts the bulk share stays standalone. - **The gate caught a real regression.** First run came back **rc 1**: `alert-rules-unit-tested` failed because the wave-0 test fed pihole a `kube_deployment_status_replicas_available` series and expected a `deployment: pihole` label. Fixed; `promtool` → SUCCESS (6 rules). - Gate: rc **3** — all of `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy` **pass**; rc 3 is the two environmental skips (`gitea-settings` no local token, `conformance` no `CLUSTER_AVAILABLE`). Not green; reported as rc 3. - **Touches no sensitive paths**, so unlike #43 and #44 this needs **no `/approve-override`**. ### ⚠️ Read before merging — the rollout will briefly drop LAN DNS `postStart` must finish before a container reports Running, so a pod isn't Ready until phase 1's adlist hook completes. On *this* transition both replicas start with an empty `/etc/pihole`, so the hook sees a fresh `gravity.db` with zero adlists and runs a real `pihole -g` (72k domains) before either pod is Ready. **Do this at a quiet time.** Two mitigating facts: every pihole restart has *already* paid this cost (state was ephemeral), and after this `gravity.db` persists so later restarts shouldn't. If it proves slow, backgrounding the rebuild would decouple readiness from it — at the cost of a window where DNS resolves but doesn't block. Not done here. ### Other things I did not verify - **Nothing is reconciled.** No StatefulSet exists; anti-affinity, the PDB and the `/etc/pihole` mount have never been exercised on a real pod. - **Whether `pihole -g` finishes inside the 5m wave timeout** — real duration unmeasured. - **The old PVC's prune.** `prune: true` should delete `pihole-config`; it holds only `lost+found` so nothing is lost, but I haven't observed it, nor whether Longhorn retains the backing volume. - **Longhorn capacity**: 2 × 2Gi at 3 replicas each is ~12Gi where there was ~6Gi. Not checked against free space. - **Query history now splits per replica** — the admin UI shows only the pod that answered. Adlists and password are identical (phase 1), so blocking and auth are consistent; *reporting* isn't. - **The residual risk is unchanged and larger.** Replicas fix node failure. LAN DNS still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile. Only an out-of-cluster resolver carrying the same `address=` lines removes that; considered and not chosen. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-04 06:35:34 +00:00

feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes

gate / gate (pull_request) Successful in 1m47s

Details

gate / conformance (pull_request) Has been skipped

Details

028853c87c

Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one
node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no
LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO
force-detach), while losing the node holding the pihole-dns L2 lease -- a
DIFFERENT node -- meant tens of seconds. Two of three nodes were single points
of failure with wildly different blast radii.

Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent
only; the mechanism is render.py's, documented in the renderer contract. 1 or
absent renders a Deployment exactly as before -- verified byte-identical across
the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a
multi-replica Deployment never schedules the second pod), per-replica
volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy
Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the
rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever).

Pinned by ~20 new --selftest assertions rather than a conformance change.

ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default
/data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M --
gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn
volume held lost+found and 24K of nothing. So every restart discarded all state
and re-downloaded 72k gravity domains, and backup.policy has been backing up an
empty volume while reporting success. Fixed here because otherwise the new
per-replica volumes would hold nothing while state stayed ephemeral -- and
because persisting gravity.db is what makes a restart cheap.

The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as
uid 0 and chowns the directory recursively (no fsGroup needed), and it calls
migrate_gravity which creates the database when absent.

The gate caught a real regression on the first run: the wave-0 alert unit test
fed pihole a Deployment-shaped series. Updated; promtool SUCCESS.

Touches no sensitive paths, so this one needs no override.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 merged commit dce20b7be6 into main

2026-10-04 06:50:56 +00:00

jakman85 referenced this issue from a commit

2026-10-04 06:50:56 +00:00

Merge pull request 'feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes' (#45) from feat/pihole-multi-replica into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#45