From 3b1d6ae11531c7ac5939e35ba7e3911f324c347b Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 4 Oct 2026 01:07:30 -0500 Subject: [PATCH] feat(pihole): declare the admin password and adlists in git Prerequisite for running more than one PiHole replica. A second replica with a fresh PVC would have come up serving DNS and blocking nothing, with a different admin password -- answers varying by which pod replied. pihole.toml's own ### CHANGED markers gave the exact inventory: four settings already came from env (declared), one came from the image, and exactly ONE operator-set value lived only on the PVC -- webserver.api.pwhash, a one-way hash nobody could recover. Separately the adlists were undeclared: one list and 72,234 gravity domains that nothing in this repo named. Declares both: - A SOPS secret feeding FTLCONF_webserver_api_password. webserver.api.password takes cleartext and FTL hashes it itself (verified: pihole setpassword IS setFTLConfigValue on that key), so the PVC's pwhash becomes derived state. - A level-triggered postStart hook that reconciles the declared adlist set into gravity.db -- adds, re-enables, and disables (never deletes) undeclared lists, running pihole -g only when something moved. A file would not work: v6's adlists.list import is migration-only, guarded on the DB not existing. The rendered hook was run in the live pod against throwaway copies of gravity.db, exercising all four paths. A fresh DB built from the image's own schema template has ZERO adlists, confirming this is load-bearing rather than cosmetic. Password kept rather than set empty: the pihole Service still publishes port 80 past the oauth2-proxy, closed only by policy, and default-deny has drifted here before. Multi-replica is phase 2 and deliberately not in this branch -- it needs a schema field, a StatefulSet (the config PVC is RWO), anti-affinity and a PDB. Adds a *.sops.yaml -- a sensitive path -- so approval-policy will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) --- apps/wave-0/kustomization.yaml | 1 + apps/wave-0/pihole-bootstrap-secret.sops.yaml | 72 ++++++++++ apps/wave-0/pihole.yaml | 12 ++ catalog/pihole.yaml | 118 ++++++++++++++++ decisions/2026-10-04-pihole-declare-state.md | 133 ++++++++++++++++++ 5 files changed, 336 insertions(+) create mode 100644 apps/wave-0/pihole-bootstrap-secret.sops.yaml create mode 100644 decisions/2026-10-04-pihole-declare-state.md diff --git a/apps/wave-0/kustomization.yaml b/apps/wave-0/kustomization.yaml index 49a173a..394f76d 100644 --- a/apps/wave-0/kustomization.yaml +++ b/apps/wave-0/kustomization.yaml @@ -14,6 +14,7 @@ resources: - nextcloud-db-backup-credentials-prod.sops.yaml - nextcloud-db-backup-credentials.sops.yaml - nextcloud-db.yaml +- pihole-bootstrap-secret.sops.yaml - pihole-dns.yaml - pihole-oauth2-proxy-credentials.sops.yaml - pihole.yaml diff --git a/apps/wave-0/pihole-bootstrap-secret.sops.yaml b/apps/wave-0/pihole-bootstrap-secret.sops.yaml new file mode 100644 index 0000000..9d90759 --- /dev/null +++ b/apps/wave-0/pihole-bootstrap-secret.sops.yaml @@ -0,0 +1,72 @@ +# ================================================================================================= +# PiHole WEB/API ADMIN PASSWORD — the one piece of PiHole state that was NOT declared anywhere. +# ================================================================================================= +# WHY THIS FILE EXISTS. Measured on live prod 2026-10-04, `pihole.toml` marks every non-default +# setting with `### CHANGED`, and names the ones that came from the environment `### CHANGED (env)`. +# The complete list was: +# +# dns.upstreams ### CHANGED (env) <- declared, catalog/pihole.yaml +# dns.listeningMode ### CHANGED (env) <- declared, catalog/pihole.yaml +# dns.rateLimit.count ### CHANGED (env) <- declared, catalog/pihole.yaml +# misc.dnsmasq_lines ### CHANGED (env) <- declared, catalog/pihole.yaml +# files.macvendor ### CHANGED <- set by the image itself, container path +# webserver.api.pwhash ### CHANGED <- SET OUT OF BAND. Declared NOWHERE. This file. +# +# So exactly ONE operator-set value lived only on the Longhorn PVC: the admin password, as a +# one-way BALLOON-SHA256 hash. Nobody could recover it (a hash does not reverse), nothing in git +# produced it, and a rebuild from git would have come up with a DIFFERENT password — or none. +# +# THE REASON THIS BLOCKS HA, which is why it is fixed first. The next step is running more than one +# PiHole replica, and a replica with a fresh PVC would come up with its own unrelated password. The +# admin UI would then accept or reject the same credential depending on which pod answered. Declaring +# it here makes every replica, and every rebuild, identical by construction. +# +# MECHANISM: `webserver.api.password` is a real, settable FTL config key — verified in the running +# image, where `pihole setpassword` is itself just `setFTLConfigValue "webserver.api.password"` with +# the comment "pihole-FTL will automatically hash the password". So FTL takes CLEARTEXT on this key +# and writes the hash into `webserver.api.pwhash` itself. catalog/pihole.yaml therefore passes this +# value as `FTLCONF_webserver_api_password`, and the hash on the PVC becomes a derived artefact +# rather than the source of truth. +# +# DEFENCE IN DEPTH, NOT THE ONLY GATE. The admin UI is already behind the oauth2-proxy sidecar +# (`--allowed-group=platform-admins`), so this password is a SECOND gate. It is kept rather than set +# empty because the `pihole` Service still publishes port 80 (PiHole direct, bypassing the proxy on +# :4180) and only the CiliumNetworkPolicy closes it — and the default-deny policy in this repo has a +# documented drift history. An unauthenticated admin UI that can rewrite LAN DNS is not a good thing +# to leave resting on one policy staying correct. Narrowing that Service port is tracked follow-up. +# +# ROTATION: `sops` this file in your own terminal, change the value, commit. Flux re-applies the +# Secret, the pod restarts, FTL re-hashes. Do NOT `sops -d` it into a shared terminal or a log +# (CLAUDE.md section 6); read it in an editor. The value below was generated randomly at authoring +# time and has never been printed to a transcript. +apiVersion: v1 +kind: Secret +metadata: + name: pihole-bootstrap + namespace: pihole + labels: + app.kubernetes.io/name: pihole +type: Opaque +stringData: + #ENC[AES256_GCM,data:TMOuRyJeb5IVn3lZPXY7lgh1KGxu7GY9+WIOLHpMB50JqzmM2tHIaWUWKTt1JtZoX/E5/Y7WCUTnfkJLhRBN2zV2MwWRBSLYVy5CMg7xV8PTLRZJYAVUYS5Trg==,iv:q+DpfT6PQfY5ZuNsmP7Ddl+PDLjSm89fJICAhYoW64k=,tag:axijaO3DaxIZgZ3LeK2hXA==,type:comment] + webui-password: ENC[AES256_GCM,data:7FcK/bUW79M46f/MlouAmATKbMnoUTNuMUjtqJ5vpU4=,iv:I8DBFB39uXZV42Ie2IPwSefmMgdKEtiDknfc9Tu7h5E=,tag:L423784UrDU3TvylpHc6KQ==,type:str] +sops: + kms: [] + gcp_kms: [] + azure_kv: [] + hc_vault: [] + age: + - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg + enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0MkRSNjZLZlpXYkcvaVJL + cjlqb2wrcWREZnZLdGpsTHNPWEpseHlRZnpjCnJVM1FDQ2M2dXlaS1JBZ2k2V1BP + Yi9QUDZSMzZsL09pTi93RTdjdGxtY2MKLS0tIDIrQTZIVWdhMGIxSCtlUFN2Mlho + SWVybld4VjJyaHRSKzZZU0xsUUV2SEEKbDYHV/qBZVspK7DpBW3vYjRMJ/HmAN+9 + G9bOlNEp0jAtcNZEn6lpQ9ORgvmIrhJHoBQRLabdMWzNWSZsNn56tA== + -----END AGE ENCRYPTED FILE----- + lastmodified: "2026-10-04T06:01:27Z" + mac: ENC[AES256_GCM,data:rhTMTOwLOtHXHgqI5M2ZW/6w+UpLpxBlTk7NoN0wmGSqOcE+OGrXA1DhkZrJweyQT0RSNSSBm/6QMztLP81eepOzdp2rEkwvbQPK/l6K9g16Ww9c1IatS3NQsKEB8AoMDx7jBQ6exXYgggPx3MJsNYbE+WyloQ96U9KPdrecAM8=,iv:n34v2zX9D5xsaR1PFCH94mNMiuRfhUyI2lseUCoqiTA=,tag:9YmN9nIVLXfsGYIb0bUl2w==,type:str] + pgp: [] + encrypted_regex: ^(data|stringData)$ + version: 3.9.4 diff --git a/apps/wave-0/pihole.yaml b/apps/wave-0/pihole.yaml index 2541c87..d609790 100644 --- a/apps/wave-0/pihole.yaml +++ b/apps/wave-0/pihole.yaml @@ -90,9 +90,21 @@ spec: value: '0' - name: FTLCONF_misc_dnsmasq_lines value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP} + - name: FTLCONF_webserver_api_password + valueFrom: + secretKeyRef: + name: pihole-bootstrap + key: webui-password volumeMounts: - name: config mountPath: /data/config + lifecycle: + postStart: + exec: + command: + - /bin/bash + - -c + - "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n [ -n \"$url\" ] || continue\n printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n [ -n \"$url\" ] || continue\n # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n have=$(sql \"select count(*) from adlist where address='$url';\")\n if [ \"$have\" = \"0\" ]; then\n echo \"pihole-adlists: adding $url\"\n sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n CHANGED=1\n else\n en=$(sql \"select enabled from adlist where address='$url';\")\n if [ \"$en\" != \"1\" ]; then\n echo \"pihole-adlists: re-enabling $url\"\n sql \"update adlist set enabled=1 where address='$url';\"\n CHANGED=1\n fi\n fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n [ -n \"$addr\" ] || continue\n if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n echo \"pihole-adlists: disabling undeclared $addr\"\n sql \"update adlist set enabled=0 where address='$addr';\"\n echo changed >> /tmp/adlist-changed\n fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n echo \"pihole-adlists: declared set changed - running gravity update\"\n pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n" - name: oauth2-proxy image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 imagePullPolicy: IfNotPresent diff --git a/catalog/pihole.yaml b/catalog/pihole.yaml index 3047cf0..6052687 100644 --- a/catalog/pihole.yaml +++ b/catalog/pihole.yaml @@ -100,6 +100,20 @@ env: # while resolution for the zone does not work. Confirmed both ways against the live image. - name: FTLCONF_misc_dnsmasq_lines value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}" + # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from + # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which + # carries the full rationale and the measurement that found it. + # + # `webserver.api.password` takes CLEARTEXT and FTL hashes it itself into `webserver.api.pwhash`. + # Verified in the running image, not assumed: `pihole setpassword` is itself + # `setFTLConfigValue "webserver.api.password" "$PASSWORD"`, commented upstream as "pihole-FTL will + # automatically hash the password". So the BALLOON-SHA256 hash on the PVC is now a DERIVED artefact + # and this is its source of truth. Rotating the Secret rotates the password. + - name: FTLCONF_webserver_api_password + valueFrom: + secretKeyRef: + name: pihole-bootstrap + key: webui-password ingressRoutes: - host: pihole.${BASE_DOMAIN} port: 80 # admin web UI, protected at the Gateway by forward-auth (docs/09) @@ -114,6 +128,110 @@ auth: # rather than a renderer constant: the measurement that justifies the flag on two apps is # evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across # the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on. +lifecycle: + # LEVEL-TRIGGERED ADLIST ASSERTION, run on EVERY container start. Same pattern, and the same + # reasoning, as catalog/qbittorrent.yaml's postStart: desired state lives in git and is + # RE-ASSERTED unconditionally rather than set once by hand. + # + # WHY THIS IS NEEDED AT ALL — the blocklists were NOT reproducible from git. Measured on live prod + # 2026-10-04: `gravity.db` held exactly ONE adlist (StevenBlack's hosts, enabled) and 72,234 + # gravity domains, and NOTHING in this repo named it. Worse, a FRESH database does not seed it: + # the image's own schema template `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the + # `adlist` table and inserts NO rows into it (grepped in the running image — there is no + # `INSERT INTO adlist` at all). So a rebuild from git, or a replica with a new PVC, would come up + # SERVING DNS AND BLOCKING NOTHING, silently and with every object reporting healthy. + # + # WHY A postStart HOOK AND NOT A FILE. PiHole v6 keeps adlists as DB rows, not config. The + # `/etc/pihole/adlists.list` file still exists in `gravity.sh` but is MIGRATION-ONLY: its import + # sits inside `migrate_to_database()`, guarded by `if [ ! -e "$gravityDBfile" ]`, so it fires only + # while CREATING a database and never again on an existing one. A ConfigMap mounted there would + # therefore be silently ignored on every pod this cluster currently runs. There is also no + # `pihole adlist` CLI subcommand (only allow/deny for domains), so SQL against the DB is the + # actual interface. FTL ships an embedded shell (`pihole-FTL sqlite3`, SQLite 3.53.1) so no extra + # tooling is needed in the image. + # + # IT RECONCILES, IT DOES NOT JUST ADD: adlists present in the DB but absent from DECLARED below are + # DISABLED (not deleted — deleting cascades to `gravity` rows and would force a full re-download of + # everything on a typo). So this list is the source of truth in both directions, and `pihole -g` + # runs ONLY when something actually changed, which keeps the steady-state start free of a 70k-domain + # rebuild. + # + # NO BRACED DOLLAR EXPANSION ANYWHERE IN THIS SCRIPT, DELIBERATELY — the same contract + # catalog/qbittorrent.yaml documents at length. apps-wave-0 declares postBuild.substituteFrom + # cluster-vars, so Flux runs envsubst over this whole manifest at reconcile time; a `${VAR}` form + # would behave differently on the cluster than under a plain `kubectl apply` of the golden. Bare + # `$VAR` and `$(cmd)` are untouched by envsubst and so behave identically however this is run. + # `render.py`'s assert_substitution_safe refuses to emit a manifest that breaks this. + postStart: + exec: + command: + - /bin/bash + - -c + - | + # Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and + # does NOT run a gravity rebuild. + set -u + GRAVITY=/etc/pihole/gravity.db + + # THE DECLARED SET — one URL per line, the source of truth for what PiHole blocks. + # To add a list: add a line here, commit, let Flux reconcile. Nothing else to do. + DECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts' + + # On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded, + # and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN. + i=0 + while [ ! -s "$GRAVITY" ] && [ "$i" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done + if [ ! -s "$GRAVITY" ]; then + echo "pihole-adlists: gravity.db absent after 180s - leaving adlists untouched" >&2 + exit 0 + fi + + sql() { pihole-FTL sqlite3 "$GRAVITY" "$1"; } + CHANGED=0 + + # 1) Every declared URL must be present AND enabled. + echo "$DECLARED" | while IFS= read -r url; do + [ -n "$url" ] || continue + printf '%s\n' "$url" + done > /tmp/declared.txt + while IFS= read -r url; do + [ -n "$url" ] || continue + # Single quotes inside a URL would break this SQL; adlist URLs do not contain them. + have=$(sql "select count(*) from adlist where address='$url';") + if [ "$have" = "0" ]; then + echo "pihole-adlists: adding $url" + sql "insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');" + CHANGED=1 + else + en=$(sql "select enabled from adlist where address='$url';") + if [ "$en" != "1" ]; then + echo "pihole-adlists: re-enabling $url" + sql "update adlist set enabled=1 where address='$url';" + CHANGED=1 + fi + fi + done < /tmp/declared.txt + + # 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above). + sql "select address from adlist where enabled=1;" | while IFS= read -r addr; do + [ -n "$addr" ] || continue + if ! grep -qxF "$addr" /tmp/declared.txt; then + echo "pihole-adlists: disabling undeclared $addr" + sql "update adlist set enabled=0 where address='$addr';" + echo changed >> /tmp/adlist-changed + fi + done + [ -s /tmp/adlist-changed ] && CHANGED=1 + rm -f /tmp/adlist-changed /tmp/declared.txt + + # 3) Rebuild gravity ONLY if the declared set actually moved. + if [ "$CHANGED" = "1" ]; then + echo "pihole-adlists: declared set changed - running gravity update" + pihole -g || echo "pihole-adlists: gravity update failed; existing gravity.db retained" >&2 + else + echo "pihole-adlists: already matches the declared set - nothing to do" + fi + exit 0 update: channel: stable # R-UPD-01 range: ">=2024.0.0" diff --git a/decisions/2026-10-04-pihole-declare-state.md b/decisions/2026-10-04-pihole-declare-state.md new file mode 100644 index 0000000..7aac3f6 --- /dev/null +++ b/decisions/2026-10-04-pihole-declare-state.md @@ -0,0 +1,133 @@ +--- +date: 2026-10-04 +branch: feat/pihole-declare-state +title: PiHole's operator-set state is now declared in git — the prerequisite for running more than one replica +--- + +## What changed + +Phase 1 of making LAN DNS survive a node failure. The operator asked for multi-replica PiHole; +measurement showed that cannot be done safely first, because PiHole's important state was not +reproducible from git. A second replica with a fresh PVC would have come up **serving DNS and +blocking nothing, with a different admin password** — answers varying by which pod replied. This +branch closes that; the replica work is phase 2 and is deliberately NOT here. + +**The undeclared-state inventory, measured not guessed.** `pihole.toml` marks every non-default +setting `### CHANGED`, and those that came from the environment `### CHANGED (env)`. The complete +live list on 2026-10-04: + +| Setting | Status | +|---|---| +| `dns.upstreams`, `dns.listeningMode`, `dns.rateLimit.count`, `misc.dnsmasq_lines` | `### CHANGED (env)` — already declared in `catalog/pihole.yaml` | +| `files.macvendor` | `### CHANGED` — set by the image itself (container path), not operator state | +| `webserver.api.pwhash` | **`### CHANGED` — set out of band. Declared NOWHERE.** | + +So exactly **one** operator-set value lived only on the PVC. Separately, the **adlists** were also +undeclared: `gravity.db` held one list (StevenBlack's hosts) and 72,234 gravity domains, with nothing +in this repo naming it. + +- **`apps/wave-0/pihole-bootstrap-secret.sops.yaml` (new, SOPS/age).** Secret `pihole-bootstrap`, + key `webui-password`. Named to the `-bootstrap-secret.sops.yaml` convention so + `render.py --emit-waves` places it with no renderer change. The value is a 32-char random password + generated at authoring time and **never printed to a transcript**. +- **`catalog/pihole.yaml`:** `FTLCONF_webserver_api_password` sourced from that Secret. + `webserver.api.password` takes CLEARTEXT and FTL hashes it itself — verified in the running image, + where `pihole setpassword` *is* `setFTLConfigValue "webserver.api.password"`, commented upstream + "pihole-FTL will automatically hash the password". The PVC's `pwhash` becomes a derived artefact. +- **`catalog/pihole.yaml`:** a `lifecycle.postStart` hook that **reconciles** the declared adlist set + into `gravity.db` on every container start — adds missing, re-enables disabled, and **disables** + (never deletes) anything enabled but undeclared, running `pihole -g` only when something moved. + Same level-triggered idiom, and the same reasoning, as `catalog/qbittorrent.yaml`'s postStart. +- **`apps/wave-0/pihole.yaml`, `apps/wave-0/kustomization.yaml`:** re-rendered / secret registered. + +**Why a postStart hook and not a file.** PiHole v6 keeps adlists as DB rows. `/etc/pihole/adlists.list` +still exists in `gravity.sh` but is **migration-only**: its import sits inside `migrate_to_database()` +behind `if [ ! -e "$gravityDBfile" ]`, so it fires only while *creating* a database. A ConfigMap +mounted there would be silently ignored on every pod this cluster runs today. There is also no +`pihole adlist` CLI subcommand, so SQL is the actual interface; FTL ships an embedded shell +(`pihole-FTL sqlite3`) so nothing had to be added to the image. + +**The password is kept rather than set empty, deliberately.** The admin UI is already behind the +oauth2-proxy sidecar (`--allowed-group=platform-admins`), so an empty password would have been +simpler. But the `pihole` Service still publishes **port 80** (PiHole direct, bypassing the proxy on +`:4180`) and only the CiliumNetworkPolicy closes it — and this repo has a documented default-deny +drift history. An unauthenticated admin UI that can rewrite LAN DNS should not rest on one policy +staying correct. Narrowing that Service port is follow-up, not done here. + +**SENSITIVE PATHS — this PR adds a `**/*.sops.yaml`, so `ci/approval-policy.sh` will BLOCK it.** It +needs the operator's `/approve-override ` as a PR **review** of type "Comment". Not posted by +me: CLAUDE.md section 1 forbids an agent clearing this on its own PR. + +## What I verified + +Live prod, 2026-10-04, read-only except where noted. + +- **The undeclared-state inventory above** — read from `pihole.toml`'s own `### CHANGED` markers, with + the password value masked. This is the measurement the whole branch rests on. +- **A fresh `gravity.db` seeds NO adlists.** The image's own schema template + `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the `adlist` table and contains no + `INSERT INTO adlist` at all — grepped in the running image. Confirmed empirically below (0 rows). + This is why the adlist declaration is load-bearing and not cosmetic. +- **The adlists.list path really is migration-only** — read the guard in `/opt/pihole/gravity.sh` + (`migrate_to_database()`, `if [ ! -e "$gravityDBfile" ]`). +- **`webserver.api.password` is a real settable key** — `pihole-FTL --config webserver.api.password` + returns a masked value, and `/usr/local/bin/pihole`'s `setpassword` sets exactly that key. +- **THE RENDERED HOOK ACTUALLY WORKS — all four paths, extracted from `apps/wave-0/pihole.yaml` and + run in the live pod against THROWAWAY COPIES of `gravity.db` in `/tmp`** (production `gravity.db` + was never written, and `pihole -g` was stubbed so no 70k-domain download could start): + 1. *Steady state* → `already matches the declared set - nothing to do`. No writes, no rebuild. + 2. *True fresh DB built from the image's schema template* → 0 adlists before; hook added the + declared list and triggered the rebuild. **This is the fresh-PVC/new-replica case.** + 3. *An undeclared list enabled* → disabled it (`enabled=0`), left the declared one alone, and did + not delete the row. + 4. *Declared list present but disabled* → re-enabled it. +- **Required tools exist in the image:** `bash`, `expr`, `grep`, `printf`, `sleep`, `pihole-FTL`. +- **The Secret is genuinely encrypted** to the repo's age recipient; `stringData.webui-password` + starts `ENC[`. The gate's `no-plaintext-secrets (R-SEC-01)` L0 check passes. +- **Schema:** `check-jsonschema` against `schemas/servicedefinition.schema.json` passes. +- **Render determinism:** the canonical + `render.py --all catalog -o apps --wave-layout --no-image-automation` changed **only** + `apps/wave-0/pihole.yaml`. `render-golden` passes, so the hook's text also satisfies + `assert_substitution_safe` (no braced dollar expansion — the Flux envsubst contract). +- **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`, + `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two usual environmental + skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3 + (`CLUSTER_AVAILABLE` unset). **Not green; reported as rc 3.** + +## What I did NOT verify + +- **Nothing here has been reconciled.** Flux has not applied any of it, so the password has never + been set by this mechanism and the hook has never run during a real container start. Everything + above is the rendered artifact tested out-of-band, plus the image's own source. +- **THE CURRENT ADMIN PASSWORD WILL STOP WORKING, and that is the intent.** FTL re-hashes + `webserver.api.password` on start, overwriting the existing `pwhash`. Whoever knows the present + password (nobody, which is why this started) loses it; the declared one replaces it. I have not + observed that replacement happen. +- **Cold-start cost is unmeasured and could matter.** `postStart` must complete before the container + is reported Running. On a fresh PVC the hook waits for `gravity.db` (bounded, up to 180s) and then + runs a real `pihole -g` (72k domains, network downloads). `apps-wave-0` is `wait: true` and + `apps-wave-1` `dependsOn` it, so a slow cold start **delays the app tier**. The steady-state path + exits in about a second (scenario 1), so this is a rebuild/first-boot cost only — but it is not + measured, and the stub in testing means `pihole -g`'s real duration here is unknown. +- **Whether `pihole -g` succeeds from inside the hook's environment.** Its failure is caught and the + existing `gravity.db` retained, so a failure degrades rather than breaks — untested live. +- **Concurrency.** Nothing serialises the hook against FTL's own scheduled gravity update. SQLite + handles the locking, and the hook's writes are three small statements, but no interleaving was + exercised. +- **MULTI-REPLICA — PHASE 2, NOT STARTED.** This branch only removes the blocker. The remaining work, + and it is substantial: a `replicas`/placement field on the ServiceDefinition schema (currently + `additionalProperties: false` with no such field), `render.py` emitting a StatefulSet with + `volumeClaimTemplates` (the config PVC is **ReadWriteOnce**, so a 2-replica Deployment cannot + work), pod anti-affinity, a PodDisruptionBudget (there is none today), re-rendered goldens, and + `conformance/**` updates — the last making phase 2 sensitive-path as well. The PVC rename a + StatefulSet forces will discard the existing `gravity.db`, which is only safe **because** of this + branch, and should be done only after this one is confirmed working in-cluster. +- **The real risk the operator named is NOT fixed by phase 2 either.** Replicas address *node* + failure. LAN DNS still depends on the cluster as a whole — etcd quorum, Cilium, Longhorn, a bad + reconcile. Quantified on 2026-10-04: with `tolerationSeconds=300` plus node-NotReady detection + plus a Longhorn RWO force-detach, a failure of the node running the pod is roughly **6-10 minutes + of no LAN DNS**; a failure of the node holding the `pihole-dns-*` L2 lease (currently a + *different* node — lease on k3s-prod-2, pod on k3s-prod-1) is tens of seconds. An out-of-cluster + secondary resolver carrying the same `address=` lines is the only option that removes the + dependency, and it was considered and not chosen. +- **Port 80 on the `pihole` Service is still published** and still closed only by policy. Not narrowed. -- 2.54.0