home-network-k3s/docs/13-service-catalog.md

13 — Service Catalog & the ServiceDefinition

Serves: R-GITOPS-02, ties together R-NET-, R-STATE-, R-TLS-, R-DNS-, R-OBS-, R-UPD-. The catalog is the operator-facing onboarding surface.

The ServiceDefinition abstraction

Onboarding a service is adding/editing one ServiceDefinition under catalog/ in the implementation repo. It captures intent; the platform compiles it into concrete resources (Deployment or KubeVirt VM, PVCs, CiliumNetworkPolicies, egress routing, HTTPRoute, Certificate, DNS record, ServiceMonitor + dashboards + alerts, and update policy). Operators declare what; platform provides the how. This keeps every requirement wired from a single reviewable object.

Platform-managed services are the deliberate exception. A component with its own operator, lifecycle or install flow is declared directly in git, under the same controls (default-deny, trusted TLS, SSO, GitOps delivery through the merge gate): keycloak, Crafty, the command-center UIs, and VMs with a custom install flow such as UniFi. kind: vm remains the catalog path for VMs.

Fields (canonical schema — implementers render manifests from these)

name: <string>                     # DNS label → <name>.home.keithjackson.dev
kind: container | vm               # PaaS (default) or IaaS via KubeVirt (R-PLAT-02/03)
source:
  image: <repo/image>              # container: image ref
  # or vmTemplate: <image/source>  # vm: CDI source image
exposure: internal | public        # LAN/VPN-only vs Cloudflare Tunnel (R-DNS-04/05, ADR-0016)
egress: internet | vpn             # egress class (R-NET-05); vpn ⇒ kill-switch (R-NET-06)
storage:
  - name: config
    size: <e.g. 5Gi>
    class: longhorn                # replicated app/config state (R-STATE-02)
  - name: media                    # optional bulk tier (R-STATE-04)
    class: bulk
    export: <path>                 # bulk only: the data's location on the bulk NFS server
dependencies:                      # explicit allow rules; everything else denied (R-NET-01/02)
  - to: <service>
    port: <n>
    protocol: tcp|udp
ingressRoutes:                     # hostnames/paths to expose (rendered to HTTPRoute + TLS + DNS)
  - host: <name>.home.keithjackson.dev
    port: <servicePort>
auth:                              # SSO integration (R-IDP-02; see 15-identity-and-access.md)
  mode: oidc | forward-auth | native | none
  # oidc         → app configured as a Keycloak OIDC client (does its own login)
  # forward-auth → per-app oauth2-proxy sidecar → Keycloak (UIs w/o OIDC; ADR-0022)
  # native       → app manages its own auth (e.g. Plex/Seerr via Plex) — documented exception
  # none         → no auth (internal-only, discouraged)
  groups: [ <keycloak-group>, ... ]  # groups allowed access (R-IDP-06)
update:
  channel: stable                  # never pre-release unless opted in (R-UPD-01)
  range: <semver range | tag regex># acceptable versions
  cadence: <schedule>              # how often to check/apply
  autoRollbackOnFailedHealth: true # R-UPD-03
  pin: <optional version>          # operator freeze escape hatch
observability:
  metrics: true
  slis: [ up, ... ]                # per-service alert intents (R-OBS-01)
backup:
  policy: <ref to backup schedule> # off-cluster (R-STATE-03); or 'reacquirable' for bulk media

The schema is the contract between the operator surface and the implementation. Changing it is an interface change — coordinate per agents/coordination-protocol.md.

The fields above are the core of the contract. The formal schema, schemas/servicedefinition.schema.json in the implementation repo, is canonical and adds workload-shaping fields for what the core cannot express: env, envFrom, lifecycle, podSecurityContext, availability, hardware (a host capability class such as a GPU, never a device path), inClusterClients, auth.apiRoutes, observability.livenessWorkload and storage[].{mountPath,subPath,readOnly}.

Default posture for every service (unless overridden)

• kind: container, exposure: internal, egress: internet.
• Default-deny: no dependencies ⇒ can reach only DNS (03).
• Auth default forward-auth: any exposed UI is protected by Keycloak SSO unless it declares native OIDC (oidc) or a documented exception (native) (15, R-IDP-02).
• Longhorn-backed config PVC; off-cluster backup; trusted TLS; auto-DNS; monitored; auto-updated on stable.

The catalog (13 services)

Egress and dependencies below are normative intent; implementers render the exact policies. "Bulk" = large media tier (04). All get trusted TLS + auto-DNS + monitoring

• auto-update unless noted. Status gives each service's delivery path: catalog (rendered from its ServiceDefinition) or platform-managed (declared directly in git).

#	Service	Status	kind	exposure	egress	Storage	Key dependencies (explicit allows)	Notes
0	keycloak	platform-managed	container	public	internet	db (PostgreSQL, ADR-0019): longhorn (irreplaceable → backed up)	→ its DB; ← all SSO clients + oauth2-proxy	IdP/SSO (15); foundational, onboarded early; DR-critical + break-glass path; hardened public exposure. Run by its operator; its catalog entry is declaration only and deploys nothing
1	plex	catalog	container	internal	internet	config: longhorn (small, replicated); media: bulk	← seerr, ← sonarr, ← radarr; clients reach it	Largest service; media on bulk tier, metadata DB on Longhorn (04); auth: native (Plex); web UI LAN/VPN-only, remote streaming per Cross-cutting notes
2	sonarr	catalog	container	internal	vpn	config: longhorn; media/downloads: bulk	→ qbittorrent, → jackett, → plex:32400; ← seerr	VPN + kill-switch (R-NET-06)
3	radarr	catalog	container	internal	vpn	config: longhorn; media/downloads: bulk	→ qbittorrent, → jackett, → plex:32400; ← seerr	VPN + kill-switch
4	jackett	catalog	container	internal	vpn	config: longhorn	← sonarr, ← radarr	VPN + kill-switch (indexer)
5	seerr	catalog	container	public	internet	config: longhorn	→ sonarr:8989, → radarr:7878, → plex:32400	Request UI; talks to arr + Plex
6	pihole	catalog	container	internal	internet	config: longhorn	(provides DNS to cluster/LAN)	Also a platform DNS dependency (09)
7	nextcloud	catalog	container	public	internet	data: longhorn (irreplaceable → backed up); legacy archive: bulk, read-only	→ its DB	File sync; DR-critical data
8	minecraft (Crafty)	platform-managed	container	internal (game ports); public (web UI)	internet	world: longhorn (backed up)	—	Delivered as the Crafty controller, which runs the game servers; game ports on a LAN/VPN L4 address
9	unifi	platform-managed	vm	internal	internet	VM disk: longhorn (replicated, live-migratable)	—	UniFi OS Server in a KubeVirt VM, declared directly in git (custom install flow); browser and device traffic on a LAN/VPN L4 address, no Gateway route
10	semaphore	catalog	container	internal	internet	config/db: longhorn	→ its DB	Ansible UI
11	gitea	catalog	container	public	internet	repos/db: longhorn (irreplaceable → backed up)	—	Git hosting; bootstrapping-critical — self-hosts the platform's GitOps source + CI/CD; onboarded early (see below); DR-critical (off-cluster backup + mirror)
12	qbittorrent	catalog	container	internal	vpn	config: longhorn; downloads: bulk	← sonarr, ← radarr	VPN + kill-switch; download client

VPN-class set (R-NET-05/06): sonarr, radarr, jackett, qbittorrent — all egress via VPN with an enforced kill-switch. Their inter-dependencies (arr → qbittorrent/jackett) are east-west allows within the deny-by-default fabric; being VPN-class governs egress, not intra-cluster reachability.

Dependency graph (explicit allows only; all else denied):

seerr → sonarr, radarr, plex
sonarr → qbittorrent, jackett, plex
radarr → qbittorrent, jackett, plex
(everything → DNS only, by default)

The arr → plex allows exist because the bulk tier is NFS, which delivers no change notifications: the arr stack tells Plex when new media lands.

Delivery order (waves). Catalog services reconcile in dependency-ordered waves computed from the dependencies graph: a service's wave is its longest dependency path, so a database always lands below its consumer. Each wave health-gates only itself and what depends on it, so a crashloop cannot stall an earlier wave. VMs and leaf platform components sit on their own non-gating tiers: a slow image import or a broken UI never holds up the app tier.

SSO integration (auth.mode, R-IDP-02; see 15):

• oidc (native Keycloak client): gitea, nextcloud, semaphore, grafana*, headlamp* (*platform UIs, not in the numbered app list).
• forward-auth (per-app oauth2-proxy sidecar → Keycloak, ADR-0022): pihole, sonarr, radarr, jackett, qbittorrent, Homepage*, and the Crafty web UI* (which adds its own local login; it has no OIDC).
• native (own auth, documented exceptions): plex, seerr (Plex accounts); minecraft game ports (Mojang/Microsoft); unifi (its own login, on its own L4 address).
• none (documented exception, R-IDP-02): the read-only runbooks at /runbooks, because gating them on Keycloak would lock the operator out of the Keycloak recovery procedures.
• keycloak itself is the IdP — its own admin login is local + MFA (the root of trust).
• Prometheus, Alertmanager and Hubble UI are not exposed; any future exposure MUST be gated.

"…but not limited to": the catalog is open-ended. New services follow the same ServiceDefinition and inherit all guarantees automatically — that is the point of the abstraction.

Contract clarifications (v1.1 — resolved 2026-08-16)

Points the schema left ambiguous, now pinned so the JSON Schema, Kyverno policies, and renderers agree:

• source is exactly-one-of — image for kind: container, vmTemplate for kind: vm (schema-enforced conditional).
• storage[].class: longhorn ⇒ size is required — enforced by an L0 Kyverno policy (a replicated volume with no size is a defect). storage[].class: bulk ⇒ export is required: a bulk volume is statically bound to data that already exists, and its size is advertised capacity only.
• backup is a reference — backup: {policy: <ref>} where policy is either the literal reacquirable (bulk/re-acquirable data) or a named off-cluster backup schedule. Cadence/retention live in the referenced schedule object, not inline in the ServiceDefinition.
• update.channel is a free string defaulting to stable — not a closed enum, so the sanctioned "opt-in to a non-stable channel" (R-UPD-01) is expressible per service.
• auth.groups is required for auth.mode ∈ {oidc, forward-auth} — access must be scoped to at least one Keycloak group (R-IDP-06); optional for native/none.
• exposure: public ⇒ auth.mode ≠ none — a public service may not be unauthenticated (docs/15, R-IDP-02); schema-enforced invariant.
• Irreplaceable ⇒ off-cluster backup — DR-critical services MUST NOT be reacquirable. The JSON Schema can't see criticality, so an L0 Kyverno policy applies the conservative rule: a service with any longhorn volume MUST NOT declare reacquirable.
• update declares policy; it does not run updates. Renovate acts on it through machine-authored changes at the merge gate (11, ADR-0023).

Cross-cutting notes

• Plex media is the R-STATE-04 case: keep the metadata DB on replicated Longhorn; keep the large library on the bulk tier; downloads stage on bulk and are handed off by the arr stack.
• DR-critical data (Nextcloud, Gitea, Minecraft worlds, PiHole config, service DBs) MUST have off-cluster backups (04). The bulk tier is outside platform backup.
• Public exposure is through the Cloudflare Tunnel (09, ADR-0016). A service is public by catalog exposure: public (Keycloak, Seerr, Nextcloud, Gitea) or as a registered platform route (the Homepage apex, /runbooks, Grafana, the Crafty web UI, and the oauth2-proxy host). Everything else stays internal. Plex remote streaming uses plex.tv + an edge forward of tcp/32400 (not the tunnel), and its web UI is LAN-only. Minecraft (crafty-mc) and other L4 services are LAN/VPN-only.
• Gitea is bootstrapping-critical. The platform self-hosts its git and CI/CD, so Gitea is brought online early (the first wave, with PiHole). Gitea + Gitea Actions are the GitOps source and CI (05, ../CONTRIBUTING.md). Because a cluster loss would take its own git host with it, Gitea's repos/DB are backed up off-cluster and mirrored, so the platform can always be rebuilt from a source that outlives the cluster.

Acceptance gates

• Each catalog service is fully wired from its ServiceDefinition alone (compute, storage, policy, egress, TLS, DNS, monitoring, updates) — no manual steps beyond merge. Each platform-managed service meets the same guarantees from its declaration in git.
• Default-deny holds; only listed dependencies work.
• VPN-class services egress via VPN with a proven kill-switch.
• Stateful services survive destroy/rebuild; DR-critical data restores from off-cluster backup.