Files
home-network-k3s-impl/catalog
jakman85andClaude Opus 5 fd43f332d6 feat(pihole): authoritative LAN DNS for the zone on udp/53 + tcp/53 at a pinned VIP
platform/coredns/coredns-custom.yaml has always said its split-horizon is for
PODS and deferred LAN clients to "PiHole split-horizon". PiHole served no DNS:
its Service was ClusterIP with ports 80 and 4180 only. The LAN half of R-DNS-03
that docs/09 and the README's Phase 2 row both claim did not exist, so LAN
clients ran on hand-maintained per-client dnsmasq files - one went stale after
the dev decommission and cost the operator sonarr.home.keithjackson.dev. The
SOHO router that owns 192.168.2.0/24 cannot cover for it either: it strips
RFC1918 answers for public domains as rebind protection and returns NOERROR
with no answer.

EXPOSURE (apps/wave-0/pihole-dns.yaml, hand-written, placed into wave 0 by
render.py's EXTRA_UNITS - the schema has no field for a LoadBalancer Service, a
pinned lbipam VIP, a sharing key, source ranges or UDP, the same reason
apps/vms/unifi and platform/crafty hand-write theirs):
  - two Services, udp/53 and tcp/53, sharing one pinned VIP
    ${PIHOLE_DNS_VIP} = 192.168.2.23 on prod, the next free address in the
    .20-.30 pool (pool status IPsUsed 3 of 11; .20 Gateway, .21 unifi, .22
    crafty; arping from a LAN host got no reply).
  - no external-dns hostname annotation: nothing public is created, and
    platform/externaldns/ is untouched.

ANSWERS (catalog/pihole.yaml env - declarative, no reconciler): PiHole v6 takes
every FTL setting as an FTLCONF_ environment variable and treats it as
env-forced, applied ahead of pihole.toml on every cold start and read-only in
the UI. That is a real declarative entry point, which the Keycloak and CNPG
CronJob reconcilers exist precisely because their state lacks, so a reconciler
here would be ceremony. misc.dnsmasq_lines carries a wildcard
address=/<zone>/<GATEWAY_VIP> plus one longer-domain override per L4 service
(unifi, crafty-mc, pihole-dns); dnsmasq resolves overlaps by longest matching
domain, so precedence is structural and a newly onboarded Gateway-fronted
service needs no DNS change. dns.upstreams is declared too - the live pod
carried hand-set upstreams that no manifest knew about and that every restart
discarded.

externalTrafficPolicy Cluster, NOT Local, and loadBalancerSourceRanges for the
LAN restriction. Measured on prod, Cilium 1.20.1: the L2 announcer does not
filter lease candidates by which nodes hold a backend - the two Services landed
on k3s-prod-3 and k3s-prod-2 while the pod ran on k3s-prod-1 - and it keeps
renewing after the pod moves away, so with Local a reschedule leaves LAN DNS
dead indefinitely. Cluster serves the VIP from any node; the LAN restriction
moves to loadBalancerSourceRanges, which Cilium enforces in eBPF on the VIP
before load-balancing (verified by narrowing the range to exclude the test
client and back). The cost is SNAT: forwarded queries reach PiHole from the
ingress node, so PiHole's per-client rate limit is disabled in the catalog or
the whole LAN would count as one client against its 1000-per-60s default.

Proven from a LAN host (192.168.2.132) against 192.168.2.23 with the pod and
both leases on three DIFFERENT nodes: sonarr/grafana/nextcloud/keycloak/apex
-> 192.168.2.20, unifi -> .21, crafty-mc -> .22, pihole-dns -> .23, recursion
(cloudflare.com, github.com, debian.org) intact over both udp and tcp, AAAA
NODATA, NXDOMAIN outside the zone, and still answering after a forced
prod-1 -> prod-2 reschedule.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-14 00:57:58 -05:00
..

catalog — ServiceDefinition entries (the onboarding surface)

One YAML file per service. Each file is a ServiceDefinition conforming to ../schemas/servicedefinition.schema.json. Adding or editing a service here is the only onboarding step — the platform renders compute, storage, network policy, egress, TLS, DNS, monitoring, and update policy from it (see ../schemas/renderer-contract.md). Serves R-GITOPS-02.

Source of truth for the initial catalog: spec docs/13-service-catalog.md (14-service table + SSO/egress/dependency intent).

Current examples

File docs/13 row exposure egress auth Notes
pihole.yaml 6 internal internet forward-auth admin UI behind oauth2-proxy; config on Longhorn (DR-critical); also platform DNS
keycloak.yaml 0 public internet native the IdP itself (own admin login + MFA); PostgreSQL on Longhorn; DR-critical

The remaining catalog services (plex, sonarr, radarr, jackett, seerr, nextcloud, minecraft, unifi, semaphore, kasm, gitea, qbittorrent) are onboarded in later phases; they follow the same schema and inherit every guarantee automatically.

Authoring rules

  • Only name and source are mandatory; every other field defaults to docs/13's default posture (kind: container, exposure: internal, egress: internet, auth.mode: forward-auth, stable auto-update, DNS-only default-deny). Set a field only to override the default.
  • Any exposed UI must have an auth block: forward-auth (default) or oidc, each with a non-empty auth.groups; native/none are documented exceptions only.
  • Irreplaceable state (class: longhorn configs/DBs) must set backup.policy to an off-cluster schedule — never reacquirable (R-STATE-03). Only bulk media may be reacquirable.
  • Shell braces must be escaped $${...}. Anything rendered verbatim onto the workload — a lifecycle postStart script, an env value — lands under apps/wave-<N>/, and every apps-wave-<N> Flux Kustomization declares postBuild.substituteFrom: cluster-vars. Flux runs envsubst over the WHOLE manifest at reconcile time, so an unescaped ${VAR:-} meant for the shell is replaced with the empty string before the container ever starts (this silently killed gitea's and nextcloud's OIDC registration for days). Bare $VAR and $(cmd) are left alone and need no escaping; a genuine environment knob is declared in clusters/base/cluster-vars.yaml and written bare ${KNOB}. Full contract: ../clusters/base/apps-vms.yaml. render.py REFUSES to emit a manifest that breaks this rule (Flux's strict substitution mode cannot: it deliberately exempts the ${V:-} / ${V:=x} / ${V:?} default-operator forms).

Validate before commit — single command, no cluster (R-TEST-02)

check-jsonschema --schemafile schemas/servicedefinition.schema.json catalog/*.yaml

See ../schemas/README.md for install/pinning and a pure-Python fallback. This is the L0/L1 schema gate (spec docs/14); CI (.gitea/) runs it on every change.

Rendering — render.py (the ServiceDefinition renderer, T-0.3)

render.py compiles a ServiceDefinition into the concrete manifest set defined by ../schemas/renderer-contract.md: a Namespace (with the app-namespace/gateway-access labels), the default-deny baseline (R-NET-01), a Deployment+Service or a KubeVirt VirtualMachine + CDI DataVolume (R-PLAT-02/03), PVCs (R-STATE), one CiliumNetworkPolicy egress-allow per dependency (R-NET-02), the egress-class steering label (R-NET-05), HTTPRoute(s) on the shared Gateway (R-TLS/R-DNS), ServiceMonitor + SLI alerts (R-OBS-01), the Flux ImageRepository/ImagePolicy update plumbing (R-UPD), and the auth-mode / backup wiring hints (R-IDP/R-STATE-03).

python3 catalog/render.py catalog/pihole.yaml        # render one SD to stdout
python3 catalog/render.py --all catalog -o out/      # render every SD to out/<name>.yaml
python3 catalog/render.py --selftest                 # deterministic self-test (no cluster)
  • Deterministic: identical input → byte-identical output (golden-diffable). The L1 render conformance tests (conformance/tests/l1-rendering/*) diff the output against committed goldens — run offline by conformance/run.sh --layer l1 and ci/l1.sh (no cluster).
  • How CI/Flux consume it: CI invokes render.py in the L1 stage; a GitOps flow renders --all catalog -o <rendered-dir> and Flux reconciles the rendered directory. Design choice (renderer-contract is silent on mechanism): a render-then-consume script is the simplest reproducible option — see the header of render.py.
  • Dependency: PyYAML (the renderer's single dependency; the conformance harness stays stdlib-only and only shells out to this script + diffs text).