home-network-k3s-impl/decisions/2026-10-04-pihole-declare-state.md

date, branch, title

date	branch	title
2026-10-04	feat/pihole-declare-state	PiHole's operator-set state is now declared in git — the prerequisite for running more than one replica

What changed

Phase 1 of making LAN DNS survive a node failure. The operator asked for multi-replica PiHole; measurement showed that cannot be done safely first, because PiHole's important state was not reproducible from git. A second replica with a fresh PVC would have come up serving DNS and blocking nothing, with a different admin password — answers varying by which pod replied. This branch closes that; the replica work is phase 2 and is deliberately NOT here.

The undeclared-state inventory, measured not guessed. pihole.toml marks every non-default setting ### CHANGED, and those that came from the environment ### CHANGED (env). The complete live list on 2026-10-04:

Setting	Status
dns.upstreams, dns.listeningMode, dns.rateLimit.count, misc.dnsmasq_lines	### CHANGED (env) — already declared in catalog/pihole.yaml
files.macvendor	### CHANGED — set by the image itself (container path), not operator state
webserver.api.pwhash	### CHANGED — set out of band. Declared NOWHERE.

So exactly one operator-set value lived only on the PVC. Separately, the adlists were also undeclared: gravity.db held one list (StevenBlack's hosts) and 72,234 gravity domains, with nothing in this repo naming it.

• apps/wave-0/pihole-bootstrap-secret.sops.yaml (new, SOPS/age). Secret pihole-bootstrap, key webui-password. Named to the <svc>-bootstrap-secret.sops.yaml convention so render.py --emit-waves places it with no renderer change. The value is a 32-char random password generated at authoring time and never printed to a transcript.
• catalog/pihole.yaml: FTLCONF_webserver_api_password sourced from that Secret. webserver.api.password takes CLEARTEXT and FTL hashes it itself — verified in the running image, where pihole setpassword is setFTLConfigValue "webserver.api.password", commented upstream "pihole-FTL will automatically hash the password". The PVC's pwhash becomes a derived artefact.
• catalog/pihole.yaml: a lifecycle.postStart hook that reconciles the declared adlist set into gravity.db on every container start — adds missing, re-enables disabled, and disables (never deletes) anything enabled but undeclared, running pihole -g only when something moved. Same level-triggered idiom, and the same reasoning, as catalog/qbittorrent.yaml's postStart.
• apps/wave-0/pihole.yaml, apps/wave-0/kustomization.yaml: re-rendered / secret registered.

Why a postStart hook and not a file. PiHole v6 keeps adlists as DB rows. /etc/pihole/adlists.list still exists in gravity.sh but is migration-only: its import sits inside migrate_to_database() behind if [ ! -e "$gravityDBfile" ], so it fires only while creating a database. A ConfigMap mounted there would be silently ignored on every pod this cluster runs today. There is also no pihole adlist CLI subcommand, so SQL is the actual interface; FTL ships an embedded shell (pihole-FTL sqlite3) so nothing had to be added to the image.

The password is kept rather than set empty, deliberately. The admin UI is already behind the oauth2-proxy sidecar (--allowed-group=platform-admins), so an empty password would have been simpler. But the pihole Service still publishes port 80 (PiHole direct, bypassing the proxy on :4180) and only the CiliumNetworkPolicy closes it — and this repo has a documented default-deny drift history. An unauthenticated admin UI that can rewrite LAN DNS should not rest on one policy staying correct. Narrowing that Service port is follow-up, not done here.

SENSITIVE PATHS — this PR adds a **/*.sops.yaml, so ci/approval-policy.sh will BLOCK it. It needs the operator's /approve-override <reason> as a PR review of type "Comment". Not posted by me: CLAUDE.md section 1 forbids an agent clearing this on its own PR.

What I verified

Live prod, 2026-10-04, read-only except where noted.

• The undeclared-state inventory above — read from pihole.toml's own ### CHANGED markers, with the password value masked. This is the measurement the whole branch rests on.
• A fresh gravity.db seeds NO adlists. The image's own schema template /etc/.pihole/advanced/Templates/gravity.db.sql creates the adlist table and contains no INSERT INTO adlist at all — grepped in the running image. Confirmed empirically below (0 rows). This is why the adlist declaration is load-bearing and not cosmetic.
• The adlists.list path really is migration-only — read the guard in /opt/pihole/gravity.sh (migrate_to_database(), if [ ! -e "$gravityDBfile" ]).
• webserver.api.password is a real settable key — pihole-FTL --config webserver.api.password returns a masked value, and /usr/local/bin/pihole's setpassword sets exactly that key.
• THE RENDERED HOOK ACTUALLY WORKS — all four paths, extracted from apps/wave-0/pihole.yaml and run in the live pod against THROWAWAY COPIES of gravity.db in /tmp (production gravity.db was never written, and pihole -g was stubbed so no 70k-domain download could start):
  1. Steady state → already matches the declared set - nothing to do. No writes, no rebuild.
  2. True fresh DB built from the image's schema template → 0 adlists before; hook added the declared list and triggered the rebuild. This is the fresh-PVC/new-replica case.
  3. An undeclared list enabled → disabled it (enabled=0), left the declared one alone, and did not delete the row.
  4. Declared list present but disabled → re-enabled it.
• Required tools exist in the image: bash, expr, grep, printf, sleep, pihole-FTL.
• The Secret is genuinely encrypted to the repo's age recipient; stringData.webui-password starts ENC[. The gate's no-plaintext-secrets (R-SEC-01) L0 check passes.
• Schema: check-jsonschema against schemas/servicedefinition.schema.json passes.
• Render determinism: the canonical render.py --all catalog -o apps --wave-layout --no-image-automation changed only apps/wave-0/pihole.yaml. render-golden passes, so the hook's text also satisfies assert_substitution_safe (no braced dollar expansion — the Flux envsubst contract).
• Toolbox gate: rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1, approval-policy all pass. The rc 3 is the two usual environmental skips — gitea-settings (no local SETTINGS_API_TOKEN) and conformance L2/L3 (CLUSTER_AVAILABLE unset). Not green; reported as rc 3.

What I did NOT verify

• Nothing here has been reconciled. Flux has not applied any of it, so the password has never been set by this mechanism and the hook has never run during a real container start. Everything above is the rendered artifact tested out-of-band, plus the image's own source.
• THE CURRENT ADMIN PASSWORD WILL STOP WORKING, and that is the intent. FTL re-hashes webserver.api.password on start, overwriting the existing pwhash. Whoever knows the present password (nobody, which is why this started) loses it; the declared one replaces it. I have not observed that replacement happen.
• Cold-start cost is unmeasured and could matter. postStart must complete before the container is reported Running. On a fresh PVC the hook waits for gravity.db (bounded, up to 180s) and then runs a real pihole -g (72k domains, network downloads). apps-wave-0 is wait: true and apps-wave-1 dependsOn it, so a slow cold start delays the app tier. The steady-state path exits in about a second (scenario 1), so this is a rebuild/first-boot cost only — but it is not measured, and the stub in testing means pihole -g's real duration here is unknown.
• Whether pihole -g succeeds from inside the hook's environment. Its failure is caught and the existing gravity.db retained, so a failure degrades rather than breaks — untested live.
• Concurrency. Nothing serialises the hook against FTL's own scheduled gravity update. SQLite handles the locking, and the hook's writes are three small statements, but no interleaving was exercised.
• MULTI-REPLICA — PHASE 2, NOT STARTED. This branch only removes the blocker. The remaining work, and it is substantial: a replicas/placement field on the ServiceDefinition schema (currently additionalProperties: false with no such field), render.py emitting a StatefulSet with volumeClaimTemplates (the config PVC is ReadWriteOnce, so a 2-replica Deployment cannot work), pod anti-affinity, a PodDisruptionBudget (there is none today), re-rendered goldens, and conformance/** updates — the last making phase 2 sensitive-path as well. The PVC rename a StatefulSet forces will discard the existing gravity.db, which is only safe because of this branch, and should be done only after this one is confirmed working in-cluster.
• The real risk the operator named is NOT fixed by phase 2 either. Replicas address node failure. LAN DNS still depends on the cluster as a whole — etcd quorum, Cilium, Longhorn, a bad reconcile. Quantified on 2026-10-04: with tolerationSeconds=300 plus node-NotReady detection plus a Longhorn RWO force-detach, a failure of the node running the pod is roughly 6-10 minutes of no LAN DNS; a failure of the node holding the pihole-dns-* L2 lease (currently a different node — lease on k3s-prod-2, pod on k3s-prod-1) is tens of seconds. An out-of-cluster secondary resolver carrying the same address= lines is the only option that removes the dependency, and it was considered and not chosen.
• Port 80 on the pihole Service is still published and still closed only by policy. Not narrowed.