Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions 1 Packages Projects Releases Wiki Activity

feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes
gate / gate (pull_request) Successful in 1m47s
Details
gate / conformance (pull_request) Has been skipped
Details

Browse Source 
Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one
node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no
LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO
force-detach), while losing the node holding the pihole-dns L2 lease -- a
DIFFERENT node -- meant tens of seconds. Two of three nodes were single points
of failure with wildly different blast radii.

Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent
only; the mechanism is render.py's, documented in the renderer contract. 1 or
absent renders a Deployment exactly as before -- verified byte-identical across
the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a
multi-replica Deployment never schedules the second pod), per-replica
volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy
Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the
rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever).

Pinned by ~20 new --selftest assertions rather than a conformance change.

ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default
/data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M --
gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn
volume held lost+found and 24K of nothing. So every restart discarded all state
and re-downloaded 72k gravity domains, and backup.policy has been backing up an
empty volume while reporting success. Fixed here because otherwise the new
per-replica volumes would hold nothing while state stayed ephemeral -- and
because persisting gravity.db is what makes a restart cheap.

The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as
uid 0 and chowns the directory recursively (no fsGroup needed), and it calls
migrate_gravity which creates the database when absent.

The gate caught a real regression on the first run: the wave-0 alert unit test
fed pihole a Deployment-shaped series. Updated; promtool SUCCESS.

Touches no sensitive paths, so this one needs no override.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-04 01:34:52 -05:00
1 parent 17f0c11403
commit 028853c87c
7 files changed
+597 -45
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

catalog/render.py
+283 -10
View File
@@ -272,7 +272,15 @@ def apply_defaults(sd: dict) -> dict:
# object that actually runs on the cluster is not the one this SD renders (keycloak: the
# Keycloak operator runs a StatefulSet, so a Deployment-shaped rule would `absent()` forever).
lw = dict(obs.get("livenessWorkload") or {})
lw.setdefault("kind", "VirtualMachineInstance" if s["kind"] == "vm" else "Deployment")
# The default must track what is ACTUALLY rendered, or the `up` SLI silently dies: a
# Deployment-shaped rule against a StatefulSet `absent()`s forever, which is an alert that can
# never fire — the exact failure mode the keycloak override was added for. Deriving it from
# `availability.replicas` means bumping the replica count cannot leave a stale rule behind.
lw.setdefault(
"kind",
"VirtualMachineInstance" if s["kind"] == "vm"
else ("StatefulSet" if _is_statefulset(s) else "Deployment"),
)
lw.setdefault("name", s["name"])
obs["livenessWorkload"] = lw
s["observability"] = obs
@@ -475,6 +483,37 @@ def r_bulk_pvs(s: dict) -> list:
return out
def _replicas(s: dict) -> int:
"""Declared replica count (schema 1.10.0 `availability.replicas`). Default 1."""
return int((s.get("availability") or {}).get("replicas", 1) or 1)
def _is_statefulset(s: dict) -> bool:
"""True when this container workload must render a StatefulSet rather than a Deployment.
THE DECIDING CONSTRAINT IS THE VOLUME, not a preference. A longhorn storage entry renders
ReadWriteOnce (r_pvcs), and one RWO PVC cannot be attached by pods on two different nodes --
which is exactly what `availability.replicas > 1` plus the node anti-affinity below asks for.
A 2-replica Deployment over a shared RWO claim does not degrade, it simply never schedules the
second pod. A StatefulSet's volumeClaimTemplates give each replica its OWN claim, which is the
only shape that satisfies both at once.
A replicas-1 service keeps rendering a Deployment so every pre-existing golden is byte-identical.
"""
return s["kind"] == "container" and _replicas(s) > 1
def _sts_claim_volumes(s: dict) -> list:
"""Storage entries that become per-replica volumeClaimTemplates on a StatefulSet.
Only the dynamically-provisioned (longhorn) entries. A `bulk` entry is a STATIC, pre-bound NFS
PV over data that already exists outside the cluster and is shared read-only by every consumer
(r_bulk_pvs) -- per-replica copies of it are neither possible nor wanted, so it stays a
standalone PVC and an ordinary pod volume.
"""
return [v for v in s["storage"] if v.get("class", "longhorn") != "bulk"]
def r_pvcs(s: dict) -> list:
"""One PVC per storage entry (R-STATE-01/02/04).
@@ -482,8 +521,15 @@ def r_pvcs(s: dict) -> list:
bulk -> the STATIC external NFS PV rendered by r_bulk_pvs, pre-bound by volumeName.
"""
out = []
# On a StatefulSet the longhorn claims are NOT standalone objects — they are
# volumeClaimTemplates, so each replica gets its own (see _is_statefulset). Emitting both would
# leave an orphan PVC that nothing mounts and that `prune` would later delete out from under
# nobody. `bulk` is unaffected and still rendered here.
sts = _is_statefulset(s)
for vol in s["storage"]:
cls = vol.get("class", "longhorn")
if sts and cls != "bulk":
continue
if cls == "bulk":
# Mirror image of the PV: the same explicitly-empty class (so nothing dynamically
# provisions), the same access mode, and volumeName completing the two-sided binding.
@@ -564,6 +610,12 @@ def r_deployment(s: dict) -> dict:
if vol.get("readOnly"):
mount["readOnly"] = True
volume_mounts.append(mount)
# On a StatefulSet a longhorn volume is supplied by its volumeClaimTemplate, whose
# metadata.name IS the pod volume name. Adding it to pod.volumes as well is a conflict
# (duplicate volume name), so it is skipped here and emitted below. The MOUNT is unchanged —
# the container cannot tell the difference, which is what keeps this transparent to the app.
if _is_statefulset(s) and vol.get("class", "longhorn") != "bulk":
continue
volumes.append(
{
"name": vol["name"],
@@ -661,20 +713,80 @@ def r_deployment(s: dict) -> dict:
if ann:
tmpl_meta["annotations"] = ann
spec = {"replicas": 1}
# An app with persistent storage mounts a ReadWriteOnce Longhorn PVC. The default
# RollingUpdate strategy tries to start the NEW pod before terminating the old one, but
# the old pod still holds the RWO volume -> the new pod is stuck ContainerCreating and
# the rollout deadlocks (single-replica app can never hand the volume over). Recreate
# terminates the old pod first, freeing the PVC for the new one (R-STATE-01/02).
if s["storage"]:
sts = _is_statefulset(s)
# NODE ANTI-AFFINITY, and it is REQUIRED rather than preferred. `availability.replicas > 1`
# exists to survive the loss of one node; `preferred` would let both replicas land on the same
# node and quietly deliver none of that, with every object reporting healthy. The cost is real
# and is accepted: with N replicas the workload needs N schedulable nodes, so on a 3-node
# cluster losing TWO nodes leaves a replica Pending and — because `apps-wave-0` is `wait: true`
# with `apps-wave-1` dependsOn it — STALLS THE APP TIER. That coincides with having already lost
# etcd quorum on a 3-node control plane, i.e. the cluster is not reconciling anything at that
# point anyway, so the stall is a symptom rather than a new failure.
if sts:
pod_spec["affinity"] = {
"podAntiAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": [
{
"labelSelector": {
"matchLabels": {"app.kubernetes.io/name": s["name"]}
},
"topologyKey": "kubernetes.io/hostname",
}
]
}
}
spec = {"replicas": _replicas(s)}
if sts:
# serviceName is a required StatefulSet field. It points at the service rendered by
# r_service; that Service is a normal ClusterIP rather than headless, so per-pod DNS
# (<pod>.<svc>) does NOT resolve — and nothing here needs it. These replicas are
# interchangeable resolvers behind one VIP, not peers that address each other.
spec["serviceName"] = s["name"]
# Parallel, not the OrderedReady default. OrderedReady starts replica N+1 only after N is
# Ready, so a cold start costs the sum of every replica's first-boot work instead of the
# max — and `apps-wave-0` has `timeout: 5m`. There is no ordering requirement between
# replicas to protect (no leader, no migration, no shared volume), so serialising buys
# nothing and risks the wave timing out on a rebuild.
spec["podManagementPolicy"] = "Parallel"
elif s["storage"]:
# An app with persistent storage mounts a ReadWriteOnce Longhorn PVC. The default
# RollingUpdate strategy tries to start the NEW pod before terminating the old one, but
# the old pod still holds the RWO volume -> the new pod is stuck ContainerCreating and
# the rollout deadlocks (single-replica app can never hand the volume over). Recreate
# terminates the old pod first, freeing the PVC for the new one (R-STATE-01/02).
#
# A StatefulSet needs none of this and cannot express it: each replica owns its own claim,
# so a rolling update hands nothing over, and `strategy` is not a StatefulSet field at all.
spec["strategy"] = {"type": "Recreate"}
spec["selector"] = {"matchLabels": {"app.kubernetes.io/name": s["name"]}}
spec["template"] = {"metadata": tmpl_meta, "spec": pod_spec}
if sts:
# Per-replica claims. metadata.name MUST equal the pod volume name the container mounts
# (see the volumes loop above); the resulting PVCs are <name>-<sts>-<ordinal>, e.g.
# config-pihole-0. Same tier and access mode a standalone longhorn PVC gets from r_pvcs —
# this changes WHO owns the claim, never what it is.
tmpls = []
for vol in _sts_claim_volumes(s):
vspec = {"accessModes": ["ReadWriteOnce"], "storageClassName": "longhorn"}
if vol.get("size"):
vspec["resources"] = {"requests": {"storage": vol["size"]}}
tmpls.append(
{
"metadata": {
"name": vol["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": vspec,
}
)
if tmpls:
spec["volumeClaimTemplates"] = tmpls
return {
"apiVersion": "apps/v1",
"kind": "Deployment",
"kind": "StatefulSet" if sts else "Deployment",
"metadata": {
"name": s["name"],
"namespace": s["name"],
@@ -684,6 +796,35 @@ def r_deployment(s: dict) -> dict:
}
def r_pdb(s: dict):
"""PodDisruptionBudget for a multi-replica workload (availability.replicas > 1).
minAvailable 1, not `maxUnavailable: 1`. The budget's job here is to stop a VOLUNTARY
disruption — a node drain for maintenance, a cluster upgrade — from taking the last replica
with it; `minAvailable: 1` says that whatever else happens, one must keep serving. It does
NOT protect against involuntary disruption (a node dying), which is what the anti-affinity
and the replica count are for.
Rendered only when replicas > 1: a PDB over a single-replica workload is actively harmful, as
`minAvailable: 1` would then block every drain of its node forever.
"""
if not _is_statefulset(s):
return None
return {
"apiVersion": "policy/v1",
"kind": "PodDisruptionBudget",
"metadata": {
"name": s["name"],
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": {
"minAvailable": 1,
"selector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
},
}
def r_service(s: dict):
ports = _service_ports(s)
# A declared metrics endpoint becomes a NAMED Service port, because that name is the only
@@ -1817,7 +1958,13 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
)
for pvc in r_pvcs(s):
docs.append((f"PVC {pvc['metadata']['name']} (R-STATE)", pvc))
docs.append(("Deployment (R-PLAT-02)", r_deployment(s)))
wl = r_deployment(s)
docs.append((f"{wl['kind']} (R-PLAT-02)", wl))
pdb = r_pdb(s)
if pdb:
docs.append(
("PodDisruptionBudget — keep one replica through a drain (availability.replicas)", pdb)
)
svc = r_service(s)
if svc:
docs.append(("Service", svc))
@@ -2266,6 +2413,132 @@ def _selftest() -> int:
check("Namespace" in kinds, "container: missing Namespace")
check("NetworkPolicy" in kinds, "container: missing default-deny NetworkPolicy")
check("Deployment" in kinds, "container: missing Deployment")
check(
"PodDisruptionBudget" not in kinds,
"replicas-1 container must render NO PodDisruptionBudget (minAvailable 1 over a single "
"replica blocks every drain of its node forever)",
)
# 1b) availability.replicas (schema 1.10.0). The DEFAULT path must be untouched, and the >1
# path must change workload kind, claim ownership, spreading, budget AND the SLI metric
# together — each of these has a silent-failure mode if it is the one that gets forgotten.
base_av = {
"name": "av",
"source": {"image": "nginx:1"},
"storage": [{"name": "config", "size": "1Gi"}],
"observability": {"metrics": True, "slis": ["up"]},
}
one = render({**base_av, "availability": {"replicas": 1}})
one_kinds = [d[1]["kind"] for d in one]
check(
"Deployment" in one_kinds and "StatefulSet" not in one_kinds,
"availability.replicas=1 must still render a Deployment (back-compat)",
)
check(
"PersistentVolumeClaim" in one_kinds,
"availability.replicas=1 must still render a standalone PVC",
)
check(
"PodDisruptionBudget" not in one_kinds,
"availability.replicas=1 must render no PodDisruptionBudget",
)
many = render({**base_av, "availability": {"replicas": 2}})
many_kinds = [d[1]["kind"] for d in many]
check(
"StatefulSet" in many_kinds and "Deployment" not in many_kinds,
"availability.replicas>1 must render a StatefulSet, not a Deployment (a shared RWO claim "
"cannot be attached on two nodes, so a multi-replica Deployment never schedules)",
)
check(
"PersistentVolumeClaim" not in many_kinds,
"availability.replicas>1 must NOT also render a standalone longhorn PVC — it would be an "
"orphan that nothing mounts",
)
sts = next(d[1] for d in many if d[1]["kind"] == "StatefulSet")
check(sts["spec"].get("replicas") == 2, "StatefulSet must carry the declared replica count")
check(
"strategy" not in sts["spec"],
"StatefulSet must not carry `strategy` (not a StatefulSet field; Recreate is a "
"Deployment-only workaround for handing over one RWO volume)",
)
check(
sts["spec"].get("podManagementPolicy") == "Parallel",
"StatefulSet must set podManagementPolicy=Parallel — OrderedReady serialises first-boot "
"work across replicas and apps-wave-0 has a 5m timeout",
)
check(sts["spec"].get("serviceName") == "av", "StatefulSet must set serviceName")
tmpls = sts["spec"].get("volumeClaimTemplates") or []
check(
[t["metadata"]["name"] for t in tmpls] == ["config"],
"StatefulSet volumeClaimTemplate name must equal the pod volume name the container mounts, "
"or the pod gets a duplicate/unresolvable volume",
)
check(
tmpls and tmpls[0]["spec"]["accessModes"] == ["ReadWriteOnce"]
and tmpls[0]["spec"]["storageClassName"] == "longhorn",
"StatefulSet claim template must keep the same tier/access mode as the standalone PVC",
)
sts_vols = {v["name"] for v in sts["spec"]["template"]["spec"].get("volumes", [])}
check(
"config" not in sts_vols,
"a claim-template volume must NOT also appear in pod.volumes (duplicate volume name)",
)
mounts = sts["spec"]["template"]["spec"]["containers"][0].get("volumeMounts") or []
check(
any(m["name"] == "config" for m in mounts),
"the container must still mount the volume by name on a StatefulSet (transparent to the app)",
)
aff = (sts["spec"]["template"]["spec"].get("affinity") or {}).get("podAntiAffinity") or {}
req = aff.get("requiredDuringSchedulingIgnoredDuringExecution") or []
check(
req and req[0].get("topologyKey") == "kubernetes.io/hostname",
"availability.replicas>1 must render REQUIRED per-node anti-affinity — `preferred` can put "
"every replica on one node and deliver none of the intended node-failure tolerance",
)
pdbs = [d[1] for d in many if d[1]["kind"] == "PodDisruptionBudget"]
check(len(pdbs) == 1, "availability.replicas>1 must render exactly one PodDisruptionBudget")
check(
pdbs and pdbs[0]["spec"].get("minAvailable") == 1,
"PodDisruptionBudget must be minAvailable=1 (keep one replica serving through a drain)",
)
pr_many = next((d[1] for d in many if d[1]["kind"] == "PrometheusRule"), None)
check(
pr_many is not None
and "kube_statefulset_status_replicas_ready"
in pr_many["spec"]["groups"][0]["rules"][0]["expr"],
"the up SLI must follow the rendered workload kind — a Deployment-shaped rule against a "
"StatefulSet absent()s forever and can never fire",
)
# A bulk volume is shared external NFS: it must stay a standalone PVC even on a StatefulSet.
mixed = render(
{
**base_av,
"availability": {"replicas": 2},
"storage": [
{"name": "config", "size": "1Gi"},
{"name": "media", "class": "bulk", "export": "/mnt/x", "size": "1Ti"},
],
}
)
m_sts = next(d[1] for d in mixed if d[1]["kind"] == "StatefulSet")
check(
[t["metadata"]["name"] for t in m_sts["spec"]["volumeClaimTemplates"]] == ["config"],
"only longhorn volumes become claim templates; a bulk share must not be per-replica",
)
check(
any(
d[1]["kind"] == "PersistentVolumeClaim"
and d[1]["metadata"]["name"] == "av-media"
for d in mixed
),
"a bulk PVC must still be rendered standalone on a StatefulSet",
)
check(
"media" in {v["name"] for v in m_sts["spec"]["template"]["spec"].get("volumes", [])},
"a bulk volume must still appear in pod.volumes on a StatefulSet",
)
ns = next(d[1] for d in docs if d[1]["kind"] == "Namespace")
check(