docs(decisions): record that PiHole HA is verified live, failover drill included #46

2026-10-04T07:04:09Z

gitea-admin commented

2026-10-04 07:04:09 +00:00

No manifests change. This entry exists to correct the log.

The three entries merged earlier today each say, correctly at the time, that nothing had been reconciled — e.g. "No StatefulSet exists, no second replica has ever run, the anti-affinity has never been evaluated." All three are now merged, reconciled and measured, so those sentences are stale in exactly the way CLAUDE.md §2b warns about: a reader who found them would conclude the work is unproven. This entry is the current state, and does not edit the earlier files.

Measured on live prod after `dce20b7`

Check	Result
`apps-wave-0`	`lastAppliedRevision=dce20b7`, `Ready=True`, reconciled in ~90s — inside the `timeout: 5m` the multi-replica entry flagged as a risk
Pods / nodes	`pihole-0` → k3s-prod-3, `pihole-1` → k3s-prod-2 — required anti-affinity satisfied
Per-replica claims	`config-pihole-0`, `config-pihole-1` both `Bound` 2Gi
Old Deployment + old PVC	both pruned (the old PVC held only `lost+found`)
PDB	`minAvailable 1`, `disruptionsAllowed 1`, `currentHealthy 2`

The mountPath bug is fixed: /etc/pihole is now on /dev/longhorn/pvc-... with 6.9 MB used on both pods, where it was previously on overlay while the Longhorn volume sat unused at /data/config. The daily-offcluster backup finally has real content.

Phase 1's declarations reproduce onto a fresh volume — the whole reason for sequencing it first. Both replicas booted with an empty /etc/pihole: 72,234 gravity domains rebuilt on each from the declared adlist, and the declared SOPS secret authenticates on both pods ("valid":true, tested via each container's own env so the value was never printed).

🎯 Failover drill, observed by the operator: a replica was deleted, recreated in ~30 seconds, and DNS never dropped a request. First direct evidence the replica count delivers what it was added for, rather than merely rendering correctly.

Re-deriving the SLI mattered concretely. The StatefulSet series returns 2; the old kube_deployment_status_replicas_available{namespace="pihole"} now returns 0 series. A stale Deployment-shaped rule would have matched its own or absent(...) branch and fired permanently — a standing false page, not silence. Prometheus has pihole-down loaded as state=inactive health=ok.

Still unverified, and stated as such

Node failure, as opposed to pod failure. The drill deleted a pod, which never touches the tolerationSeconds: 300 path (that only applies to a NotReady node). The protecting mechanism is the same one just proven, so impact should be nil — but what 300s still delays is the replacement replica, and in that window there is no redundancy left.
DNS during the cutover itself — I wasn't sampling during the reconcile, so a brief gap between prune and Ready is unmeasured.
Two-node loss (hard anti-affinity ⇒ a replica Pending ⇒ tier stalls); coincides with etcd quorum loss anyway.
Longhorn headroom (~12Gi where there was ~6Gi; both claims Bound, free space unchecked).
Query history now splits per replica — the admin UI shows only the pod that answered.
The cluster-wide dependency replicas cannot fix — only an out-of-cluster resolver removes it; considered and not chosen.
Plex (#43) is not covered here; its manual steps and the outside-in path stand as its own entry describes. Reported working by the operator, not measured by me from off-LAN.
--emit-waves drift remains open (still wants to onboard vm-demo.yaml and keycloak.yaml).

Gate

rc 3 — spec-immutability, decisions-log, lint, l0, render-golden, l1, approval-policy all pass; rc 3 is the two environmental skips. Not green; reported as rc 3. No sensitive paths, so no override needed.

🤖 Generated with Claude Code

**No manifests change.** This entry exists to correct the log. The three entries merged earlier today each say, correctly at the time, that nothing had been reconciled — e.g. *"No StatefulSet exists, no second replica has ever run, the anti-affinity has never been evaluated."* All three are now merged, reconciled and **measured**, so those sentences are stale in exactly the way CLAUDE.md §2b warns about: a reader who found them would conclude the work is unproven. This entry is the current state, and does not edit the earlier files. ### Measured on live prod after `dce20b7` | Check | Result | |---|---| | `apps-wave-0` | `lastAppliedRevision=dce20b7`, `Ready=True`, reconciled in **~90s** — inside the `timeout: 5m` the multi-replica entry flagged as a risk | | Pods / nodes | `pihole-0` → **k3s-prod-3**, `pihole-1` → **k3s-prod-2** — required anti-affinity satisfied | | Per-replica claims | `config-pihole-0`, `config-pihole-1` both `Bound` 2Gi | | Old Deployment + old PVC | both **pruned** (the old PVC held only `lost+found`) | | PDB | `minAvailable 1`, `disruptionsAllowed 1`, `currentHealthy 2` | **The mountPath bug is fixed:** `/etc/pihole` is now on `/dev/longhorn/pvc-...` with **6.9 MB used** on both pods, where it was previously on `overlay` while the Longhorn volume sat unused at `/data/config`. The `daily-offcluster` backup finally has real content. **Phase 1's declarations reproduce onto a fresh volume** — the whole reason for sequencing it first. Both replicas booted with an empty `/etc/pihole`: **72,234** gravity domains rebuilt on each from the declared adlist, and the declared SOPS secret authenticates on **both** pods (`"valid":true`, tested via each container's own env so the value was never printed). **🎯 Failover drill, observed by the operator:** a replica was deleted, recreated in **~30 seconds**, and **DNS never dropped a request**. First direct evidence the replica count delivers what it was added for, rather than merely rendering correctly. **Re-deriving the SLI mattered concretely.** The StatefulSet series returns `2`; the old `kube_deployment_status_replicas_available{namespace="pihole"}` now returns **0 series**. A stale Deployment-shaped rule would have matched its own `or absent(...)` branch and **fired permanently** — a standing false page, not silence. Prometheus has `pihole-down` loaded as `state=inactive health=ok`. ### Still unverified, and stated as such - **Node failure, as opposed to pod failure.** The drill deleted a pod, which never touches the `tolerationSeconds: 300` path (that only applies to a NotReady node). The protecting mechanism is the same one just proven, so impact should be nil — but what 300s still delays is the *replacement* replica, and in that window there is no redundancy left. - **DNS during the cutover itself** — I wasn't sampling during the reconcile, so a brief gap between prune and Ready is unmeasured. - **Two-node loss** (hard anti-affinity ⇒ a replica `Pending` ⇒ tier stalls); coincides with etcd quorum loss anyway. - **Longhorn headroom** (~12Gi where there was ~6Gi; both claims Bound, free space unchecked). - **Query history now splits per replica** — the admin UI shows only the pod that answered. - **The cluster-wide dependency replicas cannot fix** — only an out-of-cluster resolver removes it; considered and not chosen. - **Plex (#43)** is not covered here; its manual steps and the outside-in path stand as its own entry describes. Reported working by the operator, not measured by me from off-LAN. - **`--emit-waves` drift** remains open (still wants to onboard `vm-demo.yaml` and `keycloak.yaml`). ### Gate rc **3** — `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy` all **pass**; rc 3 is the two environmental skips. Not green; reported as rc 3. No sensitive paths, so no override needed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-04 07:04:09 +00:00

docs(decisions): record that PiHole HA is verified live, failover drill included

gate / gate (pull_request) Successful in 1m46s

Details

gate / conformance (pull_request) Has been skipped

Details

254985baa6

No manifests change. The three entries merged earlier today each say, correctly
at the time, that nothing had been reconciled -- which is now stale in exactly
the way CLAUDE.md section 2b warns about: a reader would conclude the work is
unproven. This entry is the current state and does not edit the earlier files.

Measured on live prod after dce20b7: apps-wave-0 reconciled in ~90s (inside the
5m timeout the multi-replica entry flagged as a risk); pihole-0 on k3s-prod-3
and pihole-1 on k3s-prod-2, so the required anti-affinity was satisfied; both
per-replica claims Bound; the old Deployment and the old unused PVC both pruned;
PDB healthy with one disruption allowed.

The mountPath fix is confirmed: /etc/pihole is now on /dev/longhorn with 6.9M
used on both pods, where it was previously on the overlay.

Phase 1's declarations reproduce onto a FRESH volume, which was the whole reason
for sequencing it first: 72,234 gravity domains rebuilt on each replica from the
declared adlist, and the declared SOPS secret authenticates on BOTH pods
("valid":true via each container's own env, value never printed).

FAILOVER DRILL, observed by the operator: a replica was deleted, recreated in
~30 seconds, and DNS never dropped a request. First direct evidence the replica
count delivers what it was added for.

Re-deriving the SLI from the rendered workload kind mattered concretely: the
StatefulSet series returns 2 while the old Deployment series now returns ZERO
series, so a stale Deployment-shaped rule would have matched its own absent()
branch and fired permanently -- a standing false page.

Still unverified and stated as such: node failure (the drill deleted a pod, so
the 300s toleration path is untouched), DNS during the cutover itself, two-node
loss, and the cluster-wide dependency that replicas cannot address.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 merged commit ed6afc3d3f into main

2026-10-04 07:08:19 +00:00

jakman85 referenced this issue from a commit

2026-10-04 07:08:21 +00:00

Merge pull request 'docs(decisions): record that PiHole HA is verified live, failover drill included' (#46) from docs/pihole-ha-verified into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#46