Operator reported DNS flapping between the cluster ingress and the router's public IP. PiHole is not at fault -- both replicas answer 192.168.2.20 consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in CACHEABILITY that we created. Measured: pihole 192.168.2.23 -> 192.168.2.20 TTL=0 public 1.1.1.1 -> wan.home.keithjackson.dev TTL=300 router 192.168.2.1 -> wan.... TTL=153 -> 148 -> 143 (decrementing = caching) dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router forwards LAN queries to PiHole and caches the result, so the CORRECT answer is uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while the WRONG answer caches for 300s. The instant the router falls back to upstream (standard forwarder behaviour when the target does not answer) it latches the WAN address for the whole LAN for five minutes, then expires, re-queries PiHole, gets TTL 0, and is immediately re-exposed. So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That asymmetry, not the gap, is why it reads as "flapping". Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies local-ttl to replies made from configuration, so one directive covers all five address= names. 60s rides out a pod restart and a lease re-election while a real VIP change still propagates inside a minute; these VIPs are pinned and static. apps/wave-0/pihole.yaml regenerated with the canonical render command (--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves); exactly two files changed and the rendered diff is the single local-ttl prefix. UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough line -- see the handoff. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
catalog — ServiceDefinition entries (the onboarding surface)
One YAML file per service. Each file is a ServiceDefinition conforming to
../schemas/servicedefinition.schema.json. Adding or
editing a service here is the only onboarding step — the platform renders compute, storage, network
policy, egress, TLS, DNS, monitoring, and update policy from it (see
../schemas/renderer-contract.md). Serves R-GITOPS-02.
Source of truth for the initial catalog: spec
docs/13-service-catalog.md (14-service table +
SSO/egress/dependency intent).
Current examples
| File | docs/13 row | exposure | egress | auth | Notes |
|---|---|---|---|---|---|
pihole.yaml |
6 | internal | internet | forward-auth | admin UI behind oauth2-proxy; config on Longhorn (DR-critical); also platform DNS |
keycloak.yaml |
0 | public | internet | native | the IdP itself (own admin login + MFA); PostgreSQL on Longhorn; DR-critical |
The remaining catalog services (plex, sonarr, radarr, jackett, seerr, nextcloud, minecraft, unifi, semaphore, kasm, gitea, qbittorrent) are onboarded in later phases; they follow the same schema and inherit every guarantee automatically.
Authoring rules
- Only
nameandsourceare mandatory; every other field defaults to docs/13's default posture (kind: container,exposure: internal,egress: internet,auth.mode: forward-auth, stable auto-update, DNS-only default-deny). Set a field only to override the default. - Any exposed UI must have an
authblock:forward-auth(default) oroidc, each with a non-emptyauth.groups;native/noneare documented exceptions only. - Irreplaceable state (
class: longhornconfigs/DBs) must setbackup.policyto an off-cluster schedule — neverreacquirable(R-STATE-03). Only bulk media may bereacquirable. - Shell braces must be escaped
$${...}. Anything rendered verbatim onto the workload — alifecyclepostStart script, anenvvalue — lands underapps/wave-<N>/, and everyapps-wave-<N>Flux Kustomization declarespostBuild.substituteFrom: cluster-vars. Flux runs envsubst over the WHOLE manifest at reconcile time, so an unescaped${VAR:-}meant for the shell is replaced with the empty string before the container ever starts (this silently killed gitea's and nextcloud's OIDC registration for days). Bare$VARand$(cmd)are left alone and need no escaping; a genuine environment knob is declared inclusters/base/cluster-vars.yamland written bare${KNOB}. Full contract:../clusters/base/apps-vms.yaml.render.pyREFUSES to emit a manifest that breaks this rule (Flux's strict substitution mode cannot: it deliberately exempts the${V:-}/${V:=x}/${V:?}default-operator forms).
Validate before commit — single command, no cluster (R-TEST-02)
check-jsonschema --schemafile schemas/servicedefinition.schema.json catalog/*.yaml
See ../schemas/README.md for install/pinning and a pure-Python fallback.
This is the L0/L1 schema gate (spec docs/14); CI (.gitea/) runs it on every change.
Rendering — render.py (the ServiceDefinition renderer, T-0.3)
render.py compiles a ServiceDefinition into the concrete manifest set defined
by ../schemas/renderer-contract.md: a Namespace (with the
app-namespace/gateway-access labels), the default-deny baseline (R-NET-01), a
Deployment+Service or a KubeVirt VirtualMachine + CDI DataVolume (R-PLAT-02/03), PVCs
(R-STATE), one CiliumNetworkPolicy egress-allow per dependency (R-NET-02), the
egress-class steering label (R-NET-05), HTTPRoute(s) on the shared Gateway (R-TLS/R-DNS),
ServiceMonitor + SLI alerts (R-OBS-01), the Flux ImageRepository/ImagePolicy update plumbing
(R-UPD), and the auth-mode / backup wiring hints (R-IDP/R-STATE-03).
python3 catalog/render.py catalog/pihole.yaml # render one SD to stdout
python3 catalog/render.py --all catalog -o out/ # render every SD to out/<name>.yaml
python3 catalog/render.py --selftest # deterministic self-test (no cluster)
- Deterministic: identical input → byte-identical output (golden-diffable). The L1 render
conformance tests (
conformance/tests/l1-rendering/*) diff the output against committed goldens — run offline byconformance/run.sh --layer l1andci/l1.sh(no cluster). - How CI/Flux consume it: CI invokes
render.pyin the L1 stage; a GitOps flow renders--all catalog -o <rendered-dir>and Flux reconciles the rendered directory. Design choice (renderer-contract is silent on mechanism): a render-then-consume script is the simplest reproducible option — see the header ofrender.py. - Dependency: PyYAML (the renderer's single dependency; the conformance harness stays stdlib-only and only shells out to this script + diffs text).