--- date: 2026-10-04 branch: docs/pihole-ha-verified title: PiHole HA is VERIFIED on live prod — including a failover drill with zero dropped DNS queries --- ## What changed **No manifests changed. This entry exists to correct the log.** The three PiHole/Plex entries merged earlier today each say, correctly at the time, that nothing had been reconciled and that the outcome was unverified: - `decisions/2026-10-04-pihole-declare-state.md` — "Nothing here has been reconciled." - `decisions/2026-10-04-pihole-multi-replica.md` — "Nothing has been reconciled. No StatefulSet exists, no second replica has ever run, the anti-affinity has never been evaluated..." All three are now **merged, reconciled and measured**, so those sentences are stale in exactly the way §2b of CLAUDE.md warns about — a reader who found them would conclude the work is unproven. This entry is the current state; it does not edit the earlier files (separate files, nothing to merge). Delivered and live as of 2026-10-04, from PRs #43 (plex tcp/32400), #44 (declare PiHole state) and #45 (PiHole 2 replicas + the mountPath bug): - PiHole is a **2-replica StatefulSet**, one pod per node, with per-replica Longhorn volumes, a PodDisruptionBudget, and an `up` SLI that follows the rendered workload kind. - PiHole's admin password and adlists are **declared in git** and reproduce onto a fresh volume. - PiHole's config PVC is mounted where PiHole actually writes (`/etc/pihole`), which it was not before. ## What I verified All measured against live prod on 2026-10-04, after the merge of #45 (`dce20b7`). **Reconcile.** `GitRepository` artifact `main@sha1:dce20b7`; `apps-wave-0` `lastAppliedRevision=main@sha1:dce20b7`, `Ready=True/ReconciliationSucceeded`. It completed in roughly **90 seconds**, well inside the Kustomization's `timeout: 5m` — the risk flagged in the multi-replica entry ("whether `pihole -g` finishes inside the 5 m timeout") did **not** materialise. A transient `Ready=False/DependencyNotReady: platform-config` seen mid-flight was the dependency cascade, not a failure; `platform-config` was already `Ready=True` at the new revision moments later. **Topology and objects.** | Measured | Value | |---|---| | pods / nodes | `pihole-0` → **k3s-prod-3**, `pihole-1` → **k3s-prod-2** (2/2 Running each) — the required anti-affinity was satisfied | | per-replica claims | `config-pihole-0`, `config-pihole-1` — both `Bound`, 2Gi | | old `pihole-config` PVC | **pruned** (it held only `lost+found`, so nothing was lost) | | old `pihole` Deployment | **pruned** — `No resources found in pihole namespace` | | PodDisruptionBudget | `minAvailable 1`, `disruptionsAllowed 1`, `currentHealthy 2` | **The mountPath bug is fixed.** `df` inside both pods reports `/etc/pihole` on `/dev/longhorn/pvc-...` with **6.9 M used** — previously it was on `overlay` while the Longhorn volume sat unused at `/data/config`. `backup.policy: daily-offcluster` now has real content to back up instead of reporting success over an empty volume. **Phase 1's declarations reproduce onto a FRESH volume — the point of doing it first.** Both replicas booted with an empty `/etc/pihole`: - **72,234 gravity domains** on each, rebuilt from the adlist declared in `catalog/pihole.yaml`. - **The declared SOPS secret authenticates on both pods.** `POST /api/auth` using the container's own `FTLCONF_webserver_api_password` returned `"valid":true` on `pihole-0` **and** `pihole-1` — so Secret → env → FTL's own hashing → working login, identically on both. (The secret's value was never printed; an earlier reading of `pwhash` as "unset" was a quoting bug in the check, disproven by an empty password being rejected with `password incorrect` and then by this positive test.) **User-facing behaviour through the VIP (192.168.2.23), from a LAN host.** plex.home.keithjackson.dev -> 192.168.2.20 split-horizon intact home.keithjackson.dev -> 192.168.2.20 unifi.home.keithjackson.dev -> 192.168.2.21 google.com -> 142.250.115.102 recursion to 8.8.8.8 works doubleclick.net -> 0.0.0.0 blocking LIVE in FTL, not just in the DB googleadservices.com -> 0.0.0.0 12 consecutive queries -> 12/12 answered **THE FAILOVER DRILL — observed by the operator, 2026-10-04.** A single StatefulSet replica was deleted. It was recreated in **~30 seconds**, and **DNS never dropped a request** throughout. This is the first direct evidence that the replica count delivers what it was added for, rather than merely rendering correctly. **The SLI is not inert, and re-deriving it mattered.** Measured in Prometheus: `kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}` returns **1 series, value 2**; `kube_deployment_status_replicas_available{namespace="pihole"}` now returns **0 series**; and Prometheus has `pihole-down` loaded as `state=inactive health=ok` with the StatefulSet expression. Had the rule stayed Deployment-shaped, its `or absent(...)` branch would have matched the now-absent series and **fired permanently** — a standing false page rather than silence. That is the concrete payoff of defaulting `livenessWorkload.kind` from the rendered kind. ## What I did NOT verify - **NODE failure, as opposed to pod failure.** The drill deleted a pod, which does not exercise the `tolerationSeconds: 300` path at all — that timer only applies when a node goes NotReady. The mechanism protecting DNS is the same one just proven (the surviving replica plus `externalTrafficPolicy: Cluster` on the DNS LoadBalancers), so user-visible impact should likewise be nil; what the 300 s still delays is the *replacement* replica, and during that window there is no redundancy left. Untested. Pulling power from a node is the only real test. - **DNS during the cutover itself.** I was not sampling resolution while #45 reconciled, so whether there was a brief gap between the Deployment being pruned and the StatefulSet becoming Ready is **unmeasured**. The drill above covers the steady-state failover case, not that one-off transition. - **Two-node loss.** The hard anti-affinity means `replicas: 2` needs two schedulable nodes; losing two would leave a replica `Pending` and, with `wait: true`, stall the app tier. Reasoned, not exercised — and it coincides with etcd quorum loss on a 3-node control plane. - **Longhorn capacity headroom.** Two 2Gi claims at 3 replicas each is ~12Gi where there was ~6Gi. Both claims are `Bound` and healthy; free space was not checked. - **Query history is now split per replica** and the admin UI shows only whichever pod answered. Adlists and password are identical, so blocking and auth are consistent; the *reporting* is not, and whether that is acceptable long-term is the operator's call. - **The residual risk is unchanged, and it is the larger one.** Replicas address node failure. LAN DNS still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile — and only a resolver outside the cluster carrying the same `address=` lines removes that dependency. Considered and not chosen (operator decision, 2026-10-04). - **Plex remote access (#43) is NOT covered by this entry.** Its two manual steps (the WAN tcp/32400 forward and Plex's "manually specify public port") and the outside-in path remain as `decisions/2026-10-03-plex-remote-access-l4.md` describes them. The operator reported it working; I have not measured it from off-LAN. - **`--emit-waves` drift is still open.** Re-running it still wants to onboard `vm-demo.yaml` and `keycloak.yaml`, both listed as intentionally not onboarded in `ci/render-golden.sh`. Unfixed; see `decisions/2026-10-03-plex-remote-access-l4.md`.