From 254985baa6b4c5f62d4642751108f4c114cbd3ea Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 4 Oct 2026 02:03:42 -0500 Subject: [PATCH] docs(decisions): record that PiHole HA is verified live, failover drill included No manifests change. The three entries merged earlier today each say, correctly at the time, that nothing had been reconciled -- which is now stale in exactly the way CLAUDE.md section 2b warns about: a reader would conclude the work is unproven. This entry is the current state and does not edit the earlier files. Measured on live prod after dce20b7: apps-wave-0 reconciled in ~90s (inside the 5m timeout the multi-replica entry flagged as a risk); pihole-0 on k3s-prod-3 and pihole-1 on k3s-prod-2, so the required anti-affinity was satisfied; both per-replica claims Bound; the old Deployment and the old unused PVC both pruned; PDB healthy with one disruption allowed. The mountPath fix is confirmed: /etc/pihole is now on /dev/longhorn with 6.9M used on both pods, where it was previously on the overlay. Phase 1's declarations reproduce onto a FRESH volume, which was the whole reason for sequencing it first: 72,234 gravity domains rebuilt on each replica from the declared adlist, and the declared SOPS secret authenticates on BOTH pods ("valid":true via each container's own env, value never printed). FAILOVER DRILL, observed by the operator: a replica was deleted, recreated in ~30 seconds, and DNS never dropped a request. First direct evidence the replica count delivers what it was added for. Re-deriving the SLI from the rendered workload kind mattered concretely: the StatefulSet series returns 2 while the old Deployment series now returns ZERO series, so a stale Deployment-shaped rule would have matched its own absent() branch and fired permanently -- a standing false page. Still unverified and stated as such: node failure (the drill deleted a pod, so the 300s toleration path is untouched), DNS during the cutover itself, two-node loss, and the cluster-wide dependency that replicas cannot address. Co-Authored-By: Claude Opus 5 (1M context) --- decisions/2026-10-04-pihole-ha-verified.md | 117 +++++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100644 decisions/2026-10-04-pihole-ha-verified.md diff --git a/decisions/2026-10-04-pihole-ha-verified.md b/decisions/2026-10-04-pihole-ha-verified.md new file mode 100644 index 0000000..496d734 --- /dev/null +++ b/decisions/2026-10-04-pihole-ha-verified.md @@ -0,0 +1,117 @@ +--- +date: 2026-10-04 +branch: docs/pihole-ha-verified +title: PiHole HA is VERIFIED on live prod — including a failover drill with zero dropped DNS queries +--- + +## What changed + +**No manifests changed. This entry exists to correct the log.** The three PiHole/Plex entries merged +earlier today each say, correctly at the time, that nothing had been reconciled and that the outcome +was unverified: + +- `decisions/2026-10-04-pihole-declare-state.md` — "Nothing here has been reconciled." +- `decisions/2026-10-04-pihole-multi-replica.md` — "Nothing has been reconciled. No StatefulSet + exists, no second replica has ever run, the anti-affinity has never been evaluated..." + +All three are now **merged, reconciled and measured**, so those sentences are stale in exactly the way +§2b of CLAUDE.md warns about — a reader who found them would conclude the work is unproven. This entry +is the current state; it does not edit the earlier files (separate files, nothing to merge). + +Delivered and live as of 2026-10-04, from PRs #43 (plex tcp/32400), #44 (declare PiHole state) and +#45 (PiHole 2 replicas + the mountPath bug): + +- PiHole is a **2-replica StatefulSet**, one pod per node, with per-replica Longhorn volumes, a + PodDisruptionBudget, and an `up` SLI that follows the rendered workload kind. +- PiHole's admin password and adlists are **declared in git** and reproduce onto a fresh volume. +- PiHole's config PVC is mounted where PiHole actually writes (`/etc/pihole`), which it was not before. + +## What I verified + +All measured against live prod on 2026-10-04, after the merge of #45 (`dce20b7`). + +**Reconcile.** `GitRepository` artifact `main@sha1:dce20b7`; `apps-wave-0` +`lastAppliedRevision=main@sha1:dce20b7`, `Ready=True/ReconciliationSucceeded`. It completed in roughly +**90 seconds**, well inside the Kustomization's `timeout: 5m` — the risk flagged in the multi-replica +entry ("whether `pihole -g` finishes inside the 5 m timeout") did **not** materialise. A transient +`Ready=False/DependencyNotReady: platform-config` seen mid-flight was the dependency cascade, not a +failure; `platform-config` was already `Ready=True` at the new revision moments later. + +**Topology and objects.** + +| Measured | Value | +|---|---| +| pods / nodes | `pihole-0` → **k3s-prod-3**, `pihole-1` → **k3s-prod-2** (2/2 Running each) — the required anti-affinity was satisfied | +| per-replica claims | `config-pihole-0`, `config-pihole-1` — both `Bound`, 2Gi | +| old `pihole-config` PVC | **pruned** (it held only `lost+found`, so nothing was lost) | +| old `pihole` Deployment | **pruned** — `No resources found in pihole namespace` | +| PodDisruptionBudget | `minAvailable 1`, `disruptionsAllowed 1`, `currentHealthy 2` | + +**The mountPath bug is fixed.** `df` inside both pods reports `/etc/pihole` on +`/dev/longhorn/pvc-...` with **6.9 M used** — previously it was on `overlay` while the Longhorn volume +sat unused at `/data/config`. `backup.policy: daily-offcluster` now has real content to back up +instead of reporting success over an empty volume. + +**Phase 1's declarations reproduce onto a FRESH volume — the point of doing it first.** Both replicas +booted with an empty `/etc/pihole`: + +- **72,234 gravity domains** on each, rebuilt from the adlist declared in `catalog/pihole.yaml`. +- **The declared SOPS secret authenticates on both pods.** `POST /api/auth` using the container's own + `FTLCONF_webserver_api_password` returned `"valid":true` on `pihole-0` **and** `pihole-1` — so + Secret → env → FTL's own hashing → working login, identically on both. (The secret's value was never + printed; an earlier reading of `pwhash` as "unset" was a quoting bug in the check, disproven by an + empty password being rejected with `password incorrect` and then by this positive test.) + +**User-facing behaviour through the VIP (192.168.2.23), from a LAN host.** + + plex.home.keithjackson.dev -> 192.168.2.20 split-horizon intact + home.keithjackson.dev -> 192.168.2.20 + unifi.home.keithjackson.dev -> 192.168.2.21 + google.com -> 142.250.115.102 recursion to 8.8.8.8 works + doubleclick.net -> 0.0.0.0 blocking LIVE in FTL, not just in the DB + googleadservices.com -> 0.0.0.0 + 12 consecutive queries -> 12/12 answered + +**THE FAILOVER DRILL — observed by the operator, 2026-10-04.** A single StatefulSet replica was +deleted. It was recreated in **~30 seconds**, and **DNS never dropped a request** throughout. This is +the first direct evidence that the replica count delivers what it was added for, rather than merely +rendering correctly. + +**The SLI is not inert, and re-deriving it mattered.** Measured in Prometheus: +`kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}` returns **1 series, +value 2**; `kube_deployment_status_replicas_available{namespace="pihole"}` now returns **0 series**; +and Prometheus has `pihole-down` loaded as `state=inactive health=ok` with the StatefulSet expression. +Had the rule stayed Deployment-shaped, its `or absent(...)` branch would have matched the now-absent +series and **fired permanently** — a standing false page rather than silence. That is the concrete +payoff of defaulting `livenessWorkload.kind` from the rendered kind. + +## What I did NOT verify + +- **NODE failure, as opposed to pod failure.** The drill deleted a pod, which does not exercise the + `tolerationSeconds: 300` path at all — that timer only applies when a node goes NotReady. The + mechanism protecting DNS is the same one just proven (the surviving replica plus + `externalTrafficPolicy: Cluster` on the DNS LoadBalancers), so user-visible impact should likewise be + nil; what the 300 s still delays is the *replacement* replica, and during that window there is no + redundancy left. Untested. Pulling power from a node is the only real test. +- **DNS during the cutover itself.** I was not sampling resolution while #45 reconciled, so whether + there was a brief gap between the Deployment being pruned and the StatefulSet becoming Ready is + **unmeasured**. The drill above covers the steady-state failover case, not that one-off transition. +- **Two-node loss.** The hard anti-affinity means `replicas: 2` needs two schedulable nodes; losing + two would leave a replica `Pending` and, with `wait: true`, stall the app tier. Reasoned, not + exercised — and it coincides with etcd quorum loss on a 3-node control plane. +- **Longhorn capacity headroom.** Two 2Gi claims at 3 replicas each is ~12Gi where there was ~6Gi. + Both claims are `Bound` and healthy; free space was not checked. +- **Query history is now split per replica** and the admin UI shows only whichever pod answered. + Adlists and password are identical, so blocking and auth are consistent; the *reporting* is not, and + whether that is acceptable long-term is the operator's call. +- **The residual risk is unchanged, and it is the larger one.** Replicas address node failure. LAN DNS + still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile — and only a resolver + outside the cluster carrying the same `address=` lines removes that dependency. Considered and not + chosen (operator decision, 2026-10-04). +- **Plex remote access (#43) is NOT covered by this entry.** Its two manual steps (the WAN tcp/32400 + forward and Plex's "manually specify public port") and the outside-in path remain as + `decisions/2026-10-03-plex-remote-access-l4.md` describes them. The operator reported it working; + I have not measured it from off-LAN. +- **`--emit-waves` drift is still open.** Re-running it still wants to onboard `vm-demo.yaml` and + `keycloak.yaml`, both listed as intentionally not onboarded in `ci/render-golden.sh`. Unfixed; see + `decisions/2026-10-03-plex-remote-access-l4.md`. -- 2.54.0