feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes
Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO force-detach), while losing the node holding the pihole-dns L2 lease -- a DIFFERENT node -- meant tens of seconds. Two of three nodes were single points of failure with wildly different blast radii. Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent only; the mechanism is render.py's, documented in the renderer contract. 1 or absent renders a Deployment exactly as before -- verified byte-identical across the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a multi-replica Deployment never schedules the second pod), per-replica volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever). Pinned by ~20 new --selftest assertions rather than a conformance change. ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default /data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M -- gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn volume held lost+found and 24K of nothing. So every restart discarded all state and re-downloaded 72k gravity domains, and backup.policy has been backing up an empty volume while reporting success. Fixed here because otherwise the new per-replica volumes would hold nothing while state stayed ephemeral -- and because persisting gravity.db is what makes a restart cheap. The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as uid 0 and chowns the directory recursively (no fsGroup needed), and it calls migrate_gravity which creates the database when absent. The gate caught a real regression on the first run: the wave-0 alert unit test fed pihole a Deployment-shaped series. Updated; promtool SUCCESS. Touches no sensitive paths, so this one needs no override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
17f0c11403
commit
028853c87c
7 files changed
+597
-45
No files matched your search
@@ -161,6 +161,41 @@ platform must render, and cites the requirement each mapping satisfies. Changing
|
||||
- **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication),
|
||||
**R-STATE-04** (bulk tier). (docs/04)
|
||||
|
||||
### `availability` (`replicas`) — workload kind, per-replica volumes, spreading, budget
|
||||
|
||||
**Schema 1.10.0.** Availability INTENT. It names no Kubernetes object, no topology key and no
|
||||
budget — exactly like `hardware.gpu`, and for the same reason: the mechanism below should be
|
||||
replaceable without a catalog edit.
|
||||
|
||||
- **Absent, or `replicas: 1`** → a **Deployment**, standalone PVC(s), `strategy: Recreate` when there
|
||||
is storage, no PodDisruptionBudget. Byte-identical to pre-1.10.0 output, so every existing app is
|
||||
untouched.
|
||||
- **`replicas: N > 1`** → all of the following, together:
|
||||
| Rendered | Why |
|
||||
|---|---|
|
||||
| **StatefulSet** instead of Deployment | A longhorn PVC is **ReadWriteOnce**. One RWO claim cannot attach on two nodes, so a multi-replica Deployment does not degrade — the second pod never schedules. |
|
||||
| longhorn storage becomes **`volumeClaimTemplates`** | Gives each replica its own claim (`<vol>-<svc>-<ordinal>`, e.g. `config-pihole-0`). The template's `metadata.name` **must** equal the pod volume name the container mounts. The standalone PVC is then **not** emitted — it would be an orphan nothing mounts. |
|
||||
| `bulk` storage stays a **standalone PVC** | A bulk entry is a static, pre-bound NFS PV over data that already exists outside the cluster and is shared read-only. Per-replica copies are neither possible nor wanted. |
|
||||
| **required** `podAntiAffinity` on `kubernetes.io/hostname` | `preferred` would let every replica land on one node and deliver none of the intended node-failure tolerance, while every object still reported healthy. |
|
||||
| `podManagementPolicy: Parallel` | `OrderedReady` serialises first-boot work across replicas; `apps-wave-0` has `timeout: 5m`. There is no ordering requirement between replicas to protect. |
|
||||
| **PodDisruptionBudget**, `minAvailable: 1` | Stops a *voluntary* disruption (node drain, upgrade) taking the last replica. It does nothing about a node dying — that is what the replica count and anti-affinity are for. Never rendered at `replicas: 1`, where `minAvailable: 1` would block every drain of that node forever. |
|
||||
| `observability.livenessWorkload.kind` defaults to **`StatefulSet`** | The `up` SLI reads kube-state-metrics by workload kind. A Deployment-shaped rule against a StatefulSet `absent()`s forever — an alert that can never fire. Deriving the default means bumping the replica count cannot leave a stale rule behind. |
|
||||
|
||||
**Two costs, both deliberate, neither hidden:**
|
||||
|
||||
1. **N replicas need N schedulable nodes.** On a 3-node cluster, losing two nodes leaves a replica
|
||||
`Pending`; since `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, that **stalls the
|
||||
app tier**. That coincides with having lost etcd quorum on a 3-node control plane, so nothing is
|
||||
reconciling at that point anyway — the stall is a symptom, not a new failure mode.
|
||||
2. **Only declare `> 1` for a service whose on-disk state is reproducible from git.** A replica with
|
||||
a fresh volume comes up serving whatever it can bootstrap itself and nothing else. For pihole that
|
||||
prerequisite was done first — the admin password and the adlists were declared in
|
||||
`decisions/2026-10-04-pihole-declare-state.md` — because otherwise a second replica would have
|
||||
served DNS while blocking nothing, with a different password.
|
||||
|
||||
**Requirements:** R-PLAT-02 (the workload), R-STATE-01/02 (per-replica replicated volumes),
|
||||
R-OBS-01 (the SLI must follow the rendered kind).
|
||||
|
||||
### `hardware` (`gpu`) — host device capability
|
||||
- **The field is INTENT, never MECHANISM.** `hardware.gpu` is an **enum of capability classes**
|
||||
(today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
|
||||
|
||||
Reference in new issue
Block a user