home-network-k3s-impl/decisions/2026-10-04-pihole-ha-verified.md

date, branch, title

date	branch	title
2026-10-04	docs/pihole-ha-verified	PiHole HA is VERIFIED on live prod — including a failover drill with zero dropped DNS queries

What changed

No manifests changed. This entry exists to correct the log. The three PiHole/Plex entries merged earlier today each say, correctly at the time, that nothing had been reconciled and that the outcome was unverified:

• decisions/2026-10-04-pihole-declare-state.md — "Nothing here has been reconciled."
• decisions/2026-10-04-pihole-multi-replica.md — "Nothing has been reconciled. No StatefulSet exists, no second replica has ever run, the anti-affinity has never been evaluated..."

All three are now merged, reconciled and measured, so those sentences are stale in exactly the way §2b of CLAUDE.md warns about — a reader who found them would conclude the work is unproven. This entry is the current state; it does not edit the earlier files (separate files, nothing to merge).

Delivered and live as of 2026-10-04, from PRs #43 (plex tcp/32400), #44 (declare PiHole state) and #45 (PiHole 2 replicas + the mountPath bug):

• PiHole is a 2-replica StatefulSet, one pod per node, with per-replica Longhorn volumes, a PodDisruptionBudget, and an up SLI that follows the rendered workload kind.
• PiHole's admin password and adlists are declared in git and reproduce onto a fresh volume.
• PiHole's config PVC is mounted where PiHole actually writes (/etc/pihole), which it was not before.

What I verified

All measured against live prod on 2026-10-04, after the merge of #45 (dce20b7).

Reconcile. GitRepository artifact main@sha1:dce20b7; apps-wave-0 lastAppliedRevision=main@sha1:dce20b7, Ready=True/ReconciliationSucceeded. It completed in roughly 90 seconds, well inside the Kustomization's timeout: 5m — the risk flagged in the multi-replica entry ("whether pihole -g finishes inside the 5 m timeout") did not materialise. A transient Ready=False/DependencyNotReady: platform-config seen mid-flight was the dependency cascade, not a failure; platform-config was already Ready=True at the new revision moments later.

Topology and objects.

Measured	Value
pods / nodes	pihole-0 → k3s-prod-3, pihole-1 → k3s-prod-2 (2/2 Running each) — the required anti-affinity was satisfied
per-replica claims	config-pihole-0, config-pihole-1 — both Bound, 2Gi
old pihole-config PVC	pruned (it held only lost+found, so nothing was lost)
old pihole Deployment	pruned — No resources found in pihole namespace
PodDisruptionBudget	minAvailable 1, disruptionsAllowed 1, currentHealthy 2

The mountPath bug is fixed. df inside both pods reports /etc/pihole on /dev/longhorn/pvc-... with 6.9 M used — previously it was on overlay while the Longhorn volume sat unused at /data/config. backup.policy: daily-offcluster now has real content to back up instead of reporting success over an empty volume.

Phase 1's declarations reproduce onto a FRESH volume — the point of doing it first. Both replicas booted with an empty /etc/pihole:

• 72,234 gravity domains on each, rebuilt from the adlist declared in catalog/pihole.yaml.
• The declared SOPS secret authenticates on both pods. POST /api/auth using the container's own FTLCONF_webserver_api_password returned "valid":true on pihole-0 and pihole-1 — so Secret → env → FTL's own hashing → working login, identically on both. (The secret's value was never printed; an earlier reading of pwhash as "unset" was a quoting bug in the check, disproven by an empty password being rejected with password incorrect and then by this positive test.)

User-facing behaviour through the VIP (192.168.2.23), from a LAN host.

plex.home.keithjackson.dev   -> 192.168.2.20      split-horizon intact
home.keithjackson.dev        -> 192.168.2.20
unifi.home.keithjackson.dev  -> 192.168.2.21
google.com                   -> 142.250.115.102   recursion to 8.8.8.8 works
doubleclick.net              -> 0.0.0.0           blocking LIVE in FTL, not just in the DB
googleadservices.com         -> 0.0.0.0
12 consecutive queries       -> 12/12 answered

THE FAILOVER DRILL — observed by the operator, 2026-10-04. A single StatefulSet replica was deleted. It was recreated in ~30 seconds, and DNS never dropped a request throughout. This is the first direct evidence that the replica count delivers what it was added for, rather than merely rendering correctly.

The SLI is not inert, and re-deriving it mattered. Measured in Prometheus: kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"} returns 1 series, value 2; kube_deployment_status_replicas_available{namespace="pihole"} now returns 0 series; and Prometheus has pihole-down loaded as state=inactive health=ok with the StatefulSet expression. Had the rule stayed Deployment-shaped, its or absent(...) branch would have matched the now-absent series and fired permanently — a standing false page rather than silence. That is the concrete payoff of defaulting livenessWorkload.kind from the rendered kind.

What I did NOT verify

• NODE failure, as opposed to pod failure. The drill deleted a pod, which does not exercise the tolerationSeconds: 300 path at all — that timer only applies when a node goes NotReady. The mechanism protecting DNS is the same one just proven (the surviving replica plus externalTrafficPolicy: Cluster on the DNS LoadBalancers), so user-visible impact should likewise be nil; what the 300 s still delays is the replacement replica, and during that window there is no redundancy left. Untested. Pulling power from a node is the only real test.
• DNS during the cutover itself. I was not sampling resolution while #45 reconciled, so whether there was a brief gap between the Deployment being pruned and the StatefulSet becoming Ready is unmeasured. The drill above covers the steady-state failover case, not that one-off transition.
• Two-node loss. The hard anti-affinity means replicas: 2 needs two schedulable nodes; losing two would leave a replica Pending and, with wait: true, stall the app tier. Reasoned, not exercised — and it coincides with etcd quorum loss on a 3-node control plane.
• Longhorn capacity headroom. Two 2Gi claims at 3 replicas each is ~12Gi where there was ~6Gi. Both claims are Bound and healthy; free space was not checked.
• Query history is now split per replica and the admin UI shows only whichever pod answered. Adlists and password are identical, so blocking and auth are consistent; the reporting is not, and whether that is acceptable long-term is the operator's call.
• The residual risk is unchanged, and it is the larger one. Replicas address node failure. LAN DNS still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile — and only a resolver outside the cluster carrying the same address= lines removes that dependency. Considered and not chosen (operator decision, 2026-10-04).
• Plex remote access (#43) is NOT covered by this entry. Its two manual steps (the WAN tcp/32400 forward and Plex's "manually specify public port") and the outside-in path remain as decisions/2026-10-03-plex-remote-access-l4.md describes them. The operator reported it working; I have not measured it from off-LAN.
• --emit-waves drift is still open. Re-running it still wants to onboard vm-demo.yaml and keycloak.yaml, both listed as intentionally not onboarded in ci/render-golden.sh. Unfixed; see decisions/2026-10-03-plex-remote-access-l4.md.