home-network-k3s-impl/decisions/2026-10-04-pihole-declare-state.md

---
date: 2026-10-04
branch: feat/pihole-declare-state
title: PiHole's operator-set state is now declared in git — the prerequisite for running more than one replica
---

## What changed

Phase 1 of making LAN DNS survive a node failure. The operator asked for multi-replica PiHole;
measurement showed that cannot be done safely first, because PiHole's important state was not
reproducible from git. A second replica with a fresh PVC would have come up **serving DNS and
blocking nothing, with a different admin password** — answers varying by which pod replied. This
branch closes that; the replica work is phase 2 and is deliberately NOT here.

**The undeclared-state inventory, measured not guessed.** `pihole.toml` marks every non-default
setting `### CHANGED`, and those that came from the environment `### CHANGED (env)`. The complete
live list on 2026-10-04:

| Setting | Status |
|---|---|
| `dns.upstreams`, `dns.listeningMode`, `dns.rateLimit.count`, `misc.dnsmasq_lines` | `### CHANGED (env)` — already declared in `catalog/pihole.yaml` |
| `files.macvendor` | `### CHANGED` — set by the image itself (container path), not operator state |
| `webserver.api.pwhash` | **`### CHANGED` — set out of band. Declared NOWHERE.** |

So exactly **one** operator-set value lived only on the PVC. Separately, the **adlists** were also
undeclared: `gravity.db` held one list (StevenBlack's hosts) and 72,234 gravity domains, with nothing
in this repo naming it.

- **`apps/wave-0/pihole-bootstrap-secret.sops.yaml` (new, SOPS/age).** Secret `pihole-bootstrap`,
  key `webui-password`. Named to the `<svc>-bootstrap-secret.sops.yaml` convention so
  `render.py --emit-waves` places it with no renderer change. The value is a 32-char random password
  generated at authoring time and **never printed to a transcript**.
- **`catalog/pihole.yaml`:** `FTLCONF_webserver_api_password` sourced from that Secret.
  `webserver.api.password` takes CLEARTEXT and FTL hashes it itself — verified in the running image,
  where `pihole setpassword` *is* `setFTLConfigValue "webserver.api.password"`, commented upstream
  "pihole-FTL will automatically hash the password". The PVC's `pwhash` becomes a derived artefact.
- **`catalog/pihole.yaml`:** a `lifecycle.postStart` hook that **reconciles** the declared adlist set
  into `gravity.db` on every container start — adds missing, re-enables disabled, and **disables**
  (never deletes) anything enabled but undeclared, running `pihole -g` only when something moved.
  Same level-triggered idiom, and the same reasoning, as `catalog/qbittorrent.yaml`'s postStart.
- **`apps/wave-0/pihole.yaml`, `apps/wave-0/kustomization.yaml`:** re-rendered / secret registered.

**Why a postStart hook and not a file.** PiHole v6 keeps adlists as DB rows. `/etc/pihole/adlists.list`
still exists in `gravity.sh` but is **migration-only**: its import sits inside `migrate_to_database()`
behind `if [ ! -e "$gravityDBfile" ]`, so it fires only while *creating* a database. A ConfigMap
mounted there would be silently ignored on every pod this cluster runs today. There is also no
`pihole adlist` CLI subcommand, so SQL is the actual interface; FTL ships an embedded shell
(`pihole-FTL sqlite3`) so nothing had to be added to the image.

**The password is kept rather than set empty, deliberately.** The admin UI is already behind the
oauth2-proxy sidecar (`--allowed-group=platform-admins`), so an empty password would have been
simpler. But the `pihole` Service still publishes **port 80** (PiHole direct, bypassing the proxy on
`:4180`) and only the CiliumNetworkPolicy closes it — and this repo has a documented default-deny
drift history. An unauthenticated admin UI that can rewrite LAN DNS should not rest on one policy
staying correct. Narrowing that Service port is follow-up, not done here.

**SENSITIVE PATHS — this PR adds a `**/*.sops.yaml`, so `ci/approval-policy.sh` will BLOCK it.** It
needs the operator's `/approve-override <reason>` as a PR **review** of type "Comment". Not posted by
me: CLAUDE.md section 1 forbids an agent clearing this on its own PR.

## What I verified

Live prod, 2026-10-04, read-only except where noted.

- **The undeclared-state inventory above** — read from `pihole.toml`'s own `### CHANGED` markers, with
  the password value masked. This is the measurement the whole branch rests on.
- **A fresh `gravity.db` seeds NO adlists.** The image's own schema template
  `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the `adlist` table and contains no
  `INSERT INTO adlist` at all — grepped in the running image. Confirmed empirically below (0 rows).
  This is why the adlist declaration is load-bearing and not cosmetic.
- **The adlists.list path really is migration-only** — read the guard in `/opt/pihole/gravity.sh`
  (`migrate_to_database()`, `if [ ! -e "$gravityDBfile" ]`).
- **`webserver.api.password` is a real settable key** — `pihole-FTL --config webserver.api.password`
  returns a masked value, and `/usr/local/bin/pihole`'s `setpassword` sets exactly that key.
- **THE RENDERED HOOK ACTUALLY WORKS — all four paths, extracted from `apps/wave-0/pihole.yaml` and
  run in the live pod against THROWAWAY COPIES of `gravity.db` in `/tmp`** (production `gravity.db`
  was never written, and `pihole -g` was stubbed so no 70k-domain download could start):
  1. *Steady state* → `already matches the declared set - nothing to do`. No writes, no rebuild.
  2. *True fresh DB built from the image's schema template* → 0 adlists before; hook added the
     declared list and triggered the rebuild. **This is the fresh-PVC/new-replica case.**
  3. *An undeclared list enabled* → disabled it (`enabled=0`), left the declared one alone, and did
     not delete the row.
  4. *Declared list present but disabled* → re-enabled it.
- **Required tools exist in the image:** `bash`, `expr`, `grep`, `printf`, `sleep`, `pihole-FTL`.
- **The Secret is genuinely encrypted** to the repo's age recipient; `stringData.webui-password`
  starts `ENC[`. The gate's `no-plaintext-secrets (R-SEC-01)` L0 check passes.
- **Schema:** `check-jsonschema` against `schemas/servicedefinition.schema.json` passes.
- **Render determinism:** the canonical
  `render.py --all catalog -o apps --wave-layout --no-image-automation` changed **only**
  `apps/wave-0/pihole.yaml`. `render-golden` passes, so the hook's text also satisfies
  `assert_substitution_safe` (no braced dollar expansion — the Flux envsubst contract).
- **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
  `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two usual environmental
  skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3
  (`CLUSTER_AVAILABLE` unset). **Not green; reported as rc 3.**

## What I did NOT verify

- **Nothing here has been reconciled.** Flux has not applied any of it, so the password has never
  been set by this mechanism and the hook has never run during a real container start. Everything
  above is the rendered artifact tested out-of-band, plus the image's own source.
- **THE CURRENT ADMIN PASSWORD WILL STOP WORKING, and that is the intent.** FTL re-hashes
  `webserver.api.password` on start, overwriting the existing `pwhash`. Whoever knows the present
  password (nobody, which is why this started) loses it; the declared one replaces it. I have not
  observed that replacement happen.
- **Cold-start cost is unmeasured and could matter.** `postStart` must complete before the container
  is reported Running. On a fresh PVC the hook waits for `gravity.db` (bounded, up to 180s) and then
  runs a real `pihole -g` (72k domains, network downloads). `apps-wave-0` is `wait: true` and
  `apps-wave-1` `dependsOn` it, so a slow cold start **delays the app tier**. The steady-state path
  exits in about a second (scenario 1), so this is a rebuild/first-boot cost only — but it is not
  measured, and the stub in testing means `pihole -g`'s real duration here is unknown.
- **Whether `pihole -g` succeeds from inside the hook's environment.** Its failure is caught and the
  existing `gravity.db` retained, so a failure degrades rather than breaks — untested live.
- **Concurrency.** Nothing serialises the hook against FTL's own scheduled gravity update. SQLite
  handles the locking, and the hook's writes are three small statements, but no interleaving was
  exercised.
- **MULTI-REPLICA — PHASE 2, NOT STARTED.** This branch only removes the blocker. The remaining work,
  and it is substantial: a `replicas`/placement field on the ServiceDefinition schema (currently
  `additionalProperties: false` with no such field), `render.py` emitting a StatefulSet with
  `volumeClaimTemplates` (the config PVC is **ReadWriteOnce**, so a 2-replica Deployment cannot
  work), pod anti-affinity, a PodDisruptionBudget (there is none today), re-rendered goldens, and
  `conformance/**` updates — the last making phase 2 sensitive-path as well. The PVC rename a
  StatefulSet forces will discard the existing `gravity.db`, which is only safe **because** of this
  branch, and should be done only after this one is confirmed working in-cluster.
- **The real risk the operator named is NOT fixed by phase 2 either.** Replicas address *node*
  failure. LAN DNS still depends on the cluster as a whole — etcd quorum, Cilium, Longhorn, a bad
  reconcile. Quantified on 2026-10-04: with `tolerationSeconds=300` plus node-NotReady detection
  plus a Longhorn RWO force-detach, a failure of the node running the pod is roughly **6-10 minutes
  of no LAN DNS**; a failure of the node holding the `pihole-dns-*` L2 lease (currently a
  *different* node — lease on k3s-prod-2, pod on k3s-prod-1) is tens of seconds. An out-of-cluster
  secondary resolver carrying the same `address=` lines is the only option that removes the
  dependency, and it was considered and not chosen.
- **Port 80 on the `pihole` Service is still published** and still closed only by policy. Not narrowed.