docs(decisions): record that PiHole HA is verified live, failover drill included

Conversation 0 Commits 1 Files Changed 1

+117

1 Commits

Author SHA1 Message Date

Author	SHA1	Message	Date
jakman85andClaude Opus 5	254985baa6	docs(decisions): record that PiHole HA is verified live, failover drill included gate / gate (pull_request) Successful in 1m46s Details gate / conformance (pull_request) Has been skipped Details No manifests change. The three entries merged earlier today each say, correctly at the time, that nothing had been reconciled -- which is now stale in exactly the way CLAUDE.md section 2b warns about: a reader would conclude the work is unproven. This entry is the current state and does not edit the earlier files. Measured on live prod after `dce20b7`: apps-wave-0 reconciled in ~90s (inside the 5m timeout the multi-replica entry flagged as a risk); pihole-0 on k3s-prod-3 and pihole-1 on k3s-prod-2, so the required anti-affinity was satisfied; both per-replica claims Bound; the old Deployment and the old unused PVC both pruned; PDB healthy with one disruption allowed. The mountPath fix is confirmed: /etc/pihole is now on /dev/longhorn with 6.9M used on both pods, where it was previously on the overlay. Phase 1's declarations reproduce onto a FRESH volume, which was the whole reason for sequencing it first: 72,234 gravity domains rebuilt on each replica from the declared adlist, and the declared SOPS secret authenticates on BOTH pods ("valid":true via each container's own env, value never printed). FAILOVER DRILL, observed by the operator: a replica was deleted, recreated in ~30 seconds, and DNS never dropped a request. First direct evidence the replica count delivers what it was added for. Re-deriving the SLI from the rendered workload kind mattered concretely: the StatefulSet series returns 2 while the old Deployment series now returns ZERO series, so a stale Deployment-shaped rule would have matched its own absent() branch and fired permanently -- a standing false page. Still unverified and stated as such: node failure (the drill deleted a pod, so the 300s toleration path is untouched), DNS during the cutover itself, two-node loss, and the cluster-wide dependency that replicas cannot address. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-04 02:03:42 -05:00

jakman85andClaude Opus 5

254985baa6

docs(decisions): record that PiHole HA is verified live, failover drill included

gate / gate (pull_request) Successful in 1m46s

Details

gate / conformance (pull_request) Has been skipped

Details

No manifests change. The three entries merged earlier today each say, correctly
at the time, that nothing had been reconciled -- which is now stale in exactly
the way CLAUDE.md section 2b warns about: a reader would conclude the work is
unproven. This entry is the current state and does not edit the earlier files.

Measured on live prod after dce20b7: apps-wave-0 reconciled in ~90s (inside the
5m timeout the multi-replica entry flagged as a risk); pihole-0 on k3s-prod-3
and pihole-1 on k3s-prod-2, so the required anti-affinity was satisfied; both
per-replica claims Bound; the old Deployment and the old unused PVC both pruned;
PDB healthy with one disruption allowed.

The mountPath fix is confirmed: /etc/pihole is now on /dev/longhorn with 6.9M
used on both pods, where it was previously on the overlay.

Phase 1's declarations reproduce onto a FRESH volume, which was the whole reason
for sequencing it first: 72,234 gravity domains rebuilt on each replica from the
declared adlist, and the declared SOPS secret authenticates on BOTH pods
("valid":true via each container's own env, value never printed).

FAILOVER DRILL, observed by the operator: a replica was deleted, recreated in
~30 seconds, and DNS never dropped a request. First direct evidence the replica
count delivers what it was added for.

Re-deriving the SLI from the rendered workload kind mattered concretely: the
StatefulSet series returns 2 while the old Deployment series now returns ZERO
series, so a stale Deployment-shaped rule would have matched its own absent()
branch and fired permanently -- a standing false page.

Still unverified and stated as such: node failure (the drill deleted a pod, so
the 300s toleration path is untouched), DNS during the cutover itself, two-node
loss, and the cluster-wide dependency that replicas cannot address.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

2026-10-04 02:03:42 -05:00

docs(decisions): record that PiHole HA is verified live, failover drill included #46