platform/coredns/coredns-custom.yaml has always said its split-horizon is for
PODS and deferred LAN clients to "PiHole split-horizon". PiHole served no DNS:
its Service was ClusterIP with ports 80 and 4180 only. The LAN half of R-DNS-03
that docs/09 and the README's Phase 2 row both claim did not exist, so LAN
clients ran on hand-maintained per-client dnsmasq files - one went stale after
the dev decommission and cost the operator sonarr.home.keithjackson.dev. The
SOHO router that owns 192.168.2.0/24 cannot cover for it either: it strips
RFC1918 answers for public domains as rebind protection and returns NOERROR
with no answer.
EXPOSURE (apps/wave-0/pihole-dns.yaml, hand-written, placed into wave 0 by
render.py's EXTRA_UNITS - the schema has no field for a LoadBalancer Service, a
pinned lbipam VIP, a sharing key, source ranges or UDP, the same reason
apps/vms/unifi and platform/crafty hand-write theirs):
- two Services, udp/53 and tcp/53, sharing one pinned VIP
${PIHOLE_DNS_VIP} = 192.168.2.23 on prod, the next free address in the
.20-.30 pool (pool status IPsUsed 3 of 11; .20 Gateway, .21 unifi, .22
crafty; arping from a LAN host got no reply).
- no external-dns hostname annotation: nothing public is created, and
platform/externaldns/ is untouched.
ANSWERS (catalog/pihole.yaml env - declarative, no reconciler): PiHole v6 takes
every FTL setting as an FTLCONF_ environment variable and treats it as
env-forced, applied ahead of pihole.toml on every cold start and read-only in
the UI. That is a real declarative entry point, which the Keycloak and CNPG
CronJob reconcilers exist precisely because their state lacks, so a reconciler
here would be ceremony. misc.dnsmasq_lines carries a wildcard
address=/<zone>/<GATEWAY_VIP> plus one longer-domain override per L4 service
(unifi, crafty-mc, pihole-dns); dnsmasq resolves overlaps by longest matching
domain, so precedence is structural and a newly onboarded Gateway-fronted
service needs no DNS change. dns.upstreams is declared too - the live pod
carried hand-set upstreams that no manifest knew about and that every restart
discarded.
externalTrafficPolicy Cluster, NOT Local, and loadBalancerSourceRanges for the
LAN restriction. Measured on prod, Cilium 1.20.1: the L2 announcer does not
filter lease candidates by which nodes hold a backend - the two Services landed
on k3s-prod-3 and k3s-prod-2 while the pod ran on k3s-prod-1 - and it keeps
renewing after the pod moves away, so with Local a reschedule leaves LAN DNS
dead indefinitely. Cluster serves the VIP from any node; the LAN restriction
moves to loadBalancerSourceRanges, which Cilium enforces in eBPF on the VIP
before load-balancing (verified by narrowing the range to exclude the test
client and back). The cost is SNAT: forwarded queries reach PiHole from the
ingress node, so PiHole's per-client rate limit is disabled in the catalog or
the whole LAN would count as one client against its 1000-per-60s default.
Proven from a LAN host (192.168.2.132) against 192.168.2.23 with the pod and
both leases on three DIFFERENT nodes: sonarr/grafana/nextcloud/keycloak/apex
-> 192.168.2.20, unifi -> .21, crafty-mc -> .22, pihole-dns -> .23, recursion
(cloudflare.com, github.com, debian.org) intact over both udp and tcp, AAAA
NODATA, NXDOMAIN outside the zone, and still answering after a forced
prod-1 -> prod-2 reschedule.
Co-Authored-By: Claude Opus 5 <[email protected]>
catalog — ServiceDefinition entries (the onboarding surface)
One YAML file per service. Each file is a ServiceDefinition conforming to
../schemas/servicedefinition.schema.json. Adding or
editing a service here is the only onboarding step — the platform renders compute, storage, network
policy, egress, TLS, DNS, monitoring, and update policy from it (see
../schemas/renderer-contract.md). Serves R-GITOPS-02.
Source of truth for the initial catalog: spec
docs/13-service-catalog.md (14-service table +
SSO/egress/dependency intent).
Current examples
| File | docs/13 row | exposure | egress | auth | Notes |
|---|---|---|---|---|---|
pihole.yaml |
6 | internal | internet | forward-auth | admin UI behind oauth2-proxy; config on Longhorn (DR-critical); also platform DNS |
keycloak.yaml |
0 | public | internet | native | the IdP itself (own admin login + MFA); PostgreSQL on Longhorn; DR-critical |
The remaining catalog services (plex, sonarr, radarr, jackett, seerr, nextcloud, minecraft, unifi, semaphore, kasm, gitea, qbittorrent) are onboarded in later phases; they follow the same schema and inherit every guarantee automatically.
Authoring rules
- Only
nameandsourceare mandatory; every other field defaults to docs/13's default posture (kind: container,exposure: internal,egress: internet,auth.mode: forward-auth, stable auto-update, DNS-only default-deny). Set a field only to override the default. - Any exposed UI must have an
authblock:forward-auth(default) oroidc, each with a non-emptyauth.groups;native/noneare documented exceptions only. - Irreplaceable state (
class: longhornconfigs/DBs) must setbackup.policyto an off-cluster schedule — neverreacquirable(R-STATE-03). Only bulk media may bereacquirable. - Shell braces must be escaped
$${...}. Anything rendered verbatim onto the workload — alifecyclepostStart script, anenvvalue — lands underapps/wave-<N>/, and everyapps-wave-<N>Flux Kustomization declarespostBuild.substituteFrom: cluster-vars. Flux runs envsubst over the WHOLE manifest at reconcile time, so an unescaped${VAR:-}meant for the shell is replaced with the empty string before the container ever starts (this silently killed gitea's and nextcloud's OIDC registration for days). Bare$VARand$(cmd)are left alone and need no escaping; a genuine environment knob is declared inclusters/base/cluster-vars.yamland written bare${KNOB}. Full contract:../clusters/base/apps-vms.yaml.render.pyREFUSES to emit a manifest that breaks this rule (Flux's strict substitution mode cannot: it deliberately exempts the${V:-}/${V:=x}/${V:?}default-operator forms).
Validate before commit — single command, no cluster (R-TEST-02)
check-jsonschema --schemafile schemas/servicedefinition.schema.json catalog/*.yaml
See ../schemas/README.md for install/pinning and a pure-Python fallback.
This is the L0/L1 schema gate (spec docs/14); CI (.gitea/) runs it on every change.
Rendering — render.py (the ServiceDefinition renderer, T-0.3)
render.py compiles a ServiceDefinition into the concrete manifest set defined
by ../schemas/renderer-contract.md: a Namespace (with the
app-namespace/gateway-access labels), the default-deny baseline (R-NET-01), a
Deployment+Service or a KubeVirt VirtualMachine + CDI DataVolume (R-PLAT-02/03), PVCs
(R-STATE), one CiliumNetworkPolicy egress-allow per dependency (R-NET-02), the
egress-class steering label (R-NET-05), HTTPRoute(s) on the shared Gateway (R-TLS/R-DNS),
ServiceMonitor + SLI alerts (R-OBS-01), the Flux ImageRepository/ImagePolicy update plumbing
(R-UPD), and the auth-mode / backup wiring hints (R-IDP/R-STATE-03).
python3 catalog/render.py catalog/pihole.yaml # render one SD to stdout
python3 catalog/render.py --all catalog -o out/ # render every SD to out/<name>.yaml
python3 catalog/render.py --selftest # deterministic self-test (no cluster)
- Deterministic: identical input → byte-identical output (golden-diffable). The L1 render
conformance tests (
conformance/tests/l1-rendering/*) diff the output against committed goldens — run offline byconformance/run.sh --layer l1andci/l1.sh(no cluster). - How CI/Flux consume it: CI invokes
render.pyin the L1 stage; a GitOps flow renders--all catalog -o <rendered-dir>and Flux reconciles the rendered directory. Design choice (renderer-contract is silent on mechanism): a render-then-consume script is the simplest reproducible option — see the header ofrender.py. - Dependency: PyYAML (the renderer's single dependency; the conformance harness stays stdlib-only and only shells out to this script + diffs text).