# Renderer Contract — ServiceDefinition → rendered resources **Status:** authoritative Phase-0 interface contract. **Consumed by** the manifest renderer (WS-GITOPS), the network/egress renderer (WS-NET), storage (WS-STATE), TLS/DNS (WS-EDGE), identity (WS-SEC), observability (WS-OBS), updates (WS-UPD), and the conformance/policy/CI agents (WS-TEST). **Source of truth:** spec [`docs/13-service-catalog.md`](../../home-network-k3s/docs/13-service-catalog.md) (the ServiceDefinition schema and default posture). This document maps each field of [`servicedefinition.schema.json`](./servicedefinition.schema.json) to the concrete resources the platform must render, and cites the requirement each mapping satisfies. Changing this contract is an **interface change** — coordinate per the spec's `agents/coordination-protocol.md` (§2). > The renderer MUST first materialize the schema **defaults** (below), then apply this mapping. A > field absent from a ServiceDefinition renders from its default posture, not "not at all." ## Default posture (applied before mapping) — docs/13 §"Default posture" | Field | Default | Requirement | |---|---|---| | `kind` | `container` | R-PLAT-02/03 | | `exposure` | `internal` | R-DNS/R-NET | | `egress` | `internet` | R-NET-05 | | `storage` | `[]` (config PVC still expected for stateful svc) | R-STATE-02 | | `dependencies` | `[]` → reaches **only DNS** (default-deny) | R-NET-01 | | `inClusterClients` | `[]` → no direct east-west client; only the Gateway may reach an exposed port | R-NET-01/02 | | `auth.mode` | `forward-auth` for any exposed UI | R-IDP-02 | | `auth.apiRoutes` | `[]` → the sidecar renders **no** `--api-route`; every unauthenticated request is 302'd to Keycloak | R-IDP-02 | | `update.channel` | `stable` | R-UPD-01 | | `update.autoRollbackOnFailedHealth` | `true` | R-UPD-03 | | `observability.metrics` | `true` | R-OBS-01 | | `observability.metricsEndpoint` | **absent** → the workload serves no Prometheus endpoint; no ServiceMonitor | R-OBS-01 | | `observability.livenessWorkload` | the object this SD renders (`Deployment` named after the service; `VirtualMachineInstance` for `kind: vm`) | R-OBS-01 | | backup of irreplaceable state | off-cluster (never `reacquirable`) | R-STATE-03 | --- ## Field → resource mapping ### `name` - **Renders:** the service Namespace (trust zone), Deployment/VM name, Service name, and the DNS label `.home.keithjackson.dev`. - **Requirements:** R-DNS-01 (naming model, docs/09), R-NET (namespace = isolation boundary, docs/03). ### `kind` + `source` - `container` + `source.image` → **Deployment** (+ Service) using the image ref. - `vm` + `source.vmTemplate` → **KubeVirt VirtualMachine** with a **CDI**-imported root disk. - The schema enforces exactly-one-of (`image` for container, `vmTemplate` for vm). - **Requirements:** R-PLAT-02/03 (PaaS/IaaS, docs/13); VM disks inherit Longhorn durability (docs/04). ### `exposure` (`internal` | `public`) - Both classes render an **HTTPRoute** on the shared Gateway and are reachable on the LAN via internal split-horizon DNS. - `public` **additionally** signals the documented **manual edge port-forward** of 80/443 (and any service-specific port, e.g. Plex 32400, Minecraft 25565) to the ingress VIP, and a **public** ExternalDNS record for `.home.keithjackson.dev`. - `internal` renders **no public DNS record** and is never port-forwarded (PiHole resolves it to the internal Gateway VIP). - **HOW "no public DNS record" IS MADE TRUE, since it was prose only until 2026-09-14.** Every HTTPRoute rendered for an `internal` service carries ExternalDNS's own controller annotation with a value other than the one ExternalDNS claims: external-dns.alpha.kubernetes.io/controller: none ExternalDNS v0.21.0 (`source/annotations/processors.go`) skips any source object whose `controller` annotation is present and `!= "dns-controller"`, and `source/gateway.go` applies that check **inside `Endpoints()` before any hostname is resolved** — so the route never becomes an endpoint for any name it declares. `source/service.go` applies the identical check, which covers the second configured source. It is derived **from `exposure` and nothing else** (`catalog/render.py` `_externaldns_route_annotations`), so the operator's one declaration decides, a service that OMITS `exposure` inherits the `internal` default and is likewise unpublishable, and there is no per-service list to forget to extend. `ci/render-golden.sh` proves `apps/` reproduces from `catalog/`, so an `internal` service cannot exist in the applied tree without it, and `conformance/tests/l0-static/internal-services-publish-no-public-dns` re-proves both directions (present on internal, **absent** on public) on every gate run. - The route itself, the wildcard TLS and LAN reachability are **unchanged** — only the public record is withdrawn. The internal split-horizon answer is Pi-hole's job (R-DNS-03), which is why withdrawing these records requires Pi-hole to be answering the LAN authoritatively first: the zone has no `*.home.keithjackson.dev` wildcard, and `*.keithjackson.dev` does not match two labels, so a withdrawn name is a public NXDOMAIN with no fallback. - Why not a cluster-wide ExternalDNS opt-in filter (`labelFilter`, which v0.21.0 supports for both configured sources): seven published names come from hand-authored `platform/`/`apps/` manifests that carry no `exposure` field for a renderer to read, and an opt-in filter deletes all seven the moment it is set. Per-resource exclusion has exactly the blast radius of the defect. The filter stays available as hardening once those seven are adjudicated — see `platform/externaldns/helmrelease.yaml` and the register in the L0 check. - `public` renders the record but the renderer does **not** yet set a target: the record still resolves to the shared Gateway VIP, i.e. an RFC1918 address, so a `public` service is not in fact publicly reachable. Fixing that is **not** a renderer change on v0.21.0 — `source/gateway.go` reads `external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and a ListenerSet), never from an HTTPRoute, so a per-service target override is not expressible while the public services share the `shared` Gateway. The open decision is recorded in `platform/externaldns/helmrelease.yaml`. - Exposure does **not** relax east-west default-deny (docs/09 §"Interaction with default-deny"). - **Requirements:** R-DNS-02/03 (docs/09), R-NET-04 (docs/03). ### `egress` (`internet` | `vpn`) — egress steering - `internet` → pod egress allow to the default WAN route (node → SOHO edge → internet). - `vpn` → the pod is **steered through the shared in-cluster Gluetun/PIA (WireGuard) egress gateway** (Cilium Egress Gateway / routing). The pod gets **no independent egress path**; the **kill-switch (fail-closed)** and allow rules are enforced at the gateway + namespace/policy level, **outside** the workload. - **Requirements:** **R-NET-05** (egress class), **R-NET-06** (VPN kill-switch, mandatory & central). VPN-class set is normative: sonarr, radarr, jackett, qbittorrent (docs/03/13). ### `dependencies[]` (`to` / `port` / `protocol`) — east-west policy - Each namespace first gets the **default-deny baseline** (deny all ingress + egress) applied by GitOps at namespace creation, with a standing allow only for **DNS to the in-cluster resolver**. **Observability scraping is NOT in the baseline** — it is an additive allow rendered per declared `observability.metricsEndpoint` (see below), scoped to that one port. This paragraph used to claim the baseline carried a standing monitoring allow; it never did, and an app metrics port would have been dropped by policy even with the Service port and monitor endpoint in agreement. - Each `dependencies[]` entry renders one **CiliumNetworkPolicy** egress allow: this service → `to` service on `port`/`protocol`. Everything undeclared stays denied. - Empty `dependencies` ⇒ the service can reach **only DNS**. - **Requirements:** **R-NET-01** (default-deny baseline), **R-NET-02** (explicit allow / deny undeclared). Auditable via Hubble (R-SEC-03, R-OBS-04). ### `storage[]` (`name` / `size` / `class` / `mountPath` / `export` / `subPath` / `readOnly`) — PVs + PVCs - `class: longhorn` → a **PVC** on the 3-replica Longhorn StorageClass sized by `size` (replicated config/DB/app state), `ReadWriteOnce`, dynamically provisioned. For a `vm`, the disk is a Longhorn volume via CDI. - `class: bulk` → a **static, pre-bound external NFS PersistentVolume _and_ its PVC**. The entry's `export` is the absolute path on the bulk server; the SERVER is never named in a ServiceDefinition — it is the `${BULK_NFS_SERVER}` cluster-vars knob, so one authored definition promotes across environments that reach the same storage by different addresses (**R-ENV-02**). Not 3× replicated: durability is the storage host's ZFS pool, not Longhorn. - `storageClassName: ""` on **both** PV and PVC (explicitly empty ⇒ statically provisioned; no provisioner is ever asked to create anything, and the default-StorageClass annotation is suppressed). Deliberately NOT the `bulk` StorageClass: a StorageClass `provisioner` is **immutable**, so re-pointing that stub would make Flux fail permanently on it. - Binding is pinned from both ends — `claimRef` on the PV, `volumeName` on the PVC — so no other claim of matching size/mode can win a race for the volume. - `persistentVolumeReclaimPolicy: Retain`, always. Bulk data pre-exists the cluster, is far larger than the cluster, and may have consumers outside it; nothing here may reclaim it. - Access mode follows the read/write posture and is always `…Many`, because one export is consumed by several namespaces and several nodes at once: `ReadOnlyMany` when `readOnly`, `ReadWriteMany` otherwise. `ReadWriteOnce` cannot express either. - `mountOptions` are a **platform** decision held in the renderer, not a per-service field: `nfsvers=4.1,hard,noatime,timeo=600,retrans=2`. NFSv4 carries mount and lock in-protocol, so the only port that must be reachable is **tcp/2049** — no rpcbind/111, mountd or statd, which is what keeps the firewall surface to a single rule. `hard` is chosen over `soft` because a soft mount converts a server hiccup into a short read handed to the application. - `size` on a bulk entry is the PV's **advertised capacity and nothing more** — no component enforces it (the ZFS pool is the real limit), so it should state the dataset's actual size so `kubectl get pv` reports something true. - **fsGroup does not apply.** The in-tree NFS volume plugin performs no ownership management, so `podSecurityContext.fsGroup` is a **no-op** on a bulk volume. A bulk mount that must be written therefore requires the container's own uid/gid to be one the export's ownership permits (e.g. Plex's `PLEX_UID`/`PLEX_GID`, LinuxServer images' `PUID`/`PGID`) — or a server-side squash. A mount that succeeds but cannot write is the characteristic failure of this tier. - `mountPath` (any class) overrides the `/data/` default verbatim. - `subPath` (any class) mounts a subtree instead of the volume root. **Two `subPath` mounts of one export are two separate mounts**, so `link()` between them fails `EXDEV`; a workload that needs hardlinks or atomic moves between two directories (Sonarr/Radarr importing from a download path into a library) must mount their common ancestor **once** instead. - `readOnly` (any class) marks the volumeMount read-only, and on a bulk entry also sets the PV's `nfs.readOnly` so the kernel mount really is `ro`. Note this is a **client-side** guarantee: one export serving both a reader and a writer that may land on any node cannot distinguish them server-side. - State separation rule: anything that must survive a rebuild lives on a named volume here, never on ephemeral fs/hostPath — so the workload can be destroyed and re-applied from git with data intact. - **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication), **R-STATE-04** (bulk tier). (docs/04) ### `availability` (`replicas`) — workload kind, per-replica volumes, spreading, budget **Schema 1.10.0.** Availability INTENT. It names no Kubernetes object, no topology key and no budget — exactly like `hardware.gpu`, and for the same reason: the mechanism below should be replaceable without a catalog edit. - **Absent, or `replicas: 1`** → a **Deployment**, standalone PVC(s), `strategy: Recreate` when there is storage, no PodDisruptionBudget. Byte-identical to pre-1.10.0 output, so every existing app is untouched. - **`replicas: N > 1`** → all of the following, together: | Rendered | Why | |---|---| | **StatefulSet** instead of Deployment | A longhorn PVC is **ReadWriteOnce**. One RWO claim cannot attach on two nodes, so a multi-replica Deployment does not degrade — the second pod never schedules. | | longhorn storage becomes **`volumeClaimTemplates`** | Gives each replica its own claim (`--`, e.g. `config-pihole-0`). The template's `metadata.name` **must** equal the pod volume name the container mounts. The standalone PVC is then **not** emitted — it would be an orphan nothing mounts. | | `bulk` storage stays a **standalone PVC** | A bulk entry is a static, pre-bound NFS PV over data that already exists outside the cluster and is shared read-only. Per-replica copies are neither possible nor wanted. | | **required** `podAntiAffinity` on `kubernetes.io/hostname` | `preferred` would let every replica land on one node and deliver none of the intended node-failure tolerance, while every object still reported healthy. | | `podManagementPolicy: Parallel` | `OrderedReady` serialises first-boot work across replicas; `apps-wave-0` has `timeout: 5m`. There is no ordering requirement between replicas to protect. | | **PodDisruptionBudget**, `minAvailable: 1` | Stops a *voluntary* disruption (node drain, upgrade) taking the last replica. It does nothing about a node dying — that is what the replica count and anti-affinity are for. Never rendered at `replicas: 1`, where `minAvailable: 1` would block every drain of that node forever. | | `observability.livenessWorkload.kind` defaults to **`StatefulSet`** | The `up` SLI reads kube-state-metrics by workload kind. A Deployment-shaped rule against a StatefulSet `absent()`s forever — an alert that can never fire. Deriving the default means bumping the replica count cannot leave a stale rule behind. | **Two costs, both deliberate, neither hidden:** 1. **N replicas need N schedulable nodes.** On a 3-node cluster, losing two nodes leaves a replica `Pending`; since `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, that **stalls the app tier**. That coincides with having lost etcd quorum on a 3-node control plane, so nothing is reconciling at that point anyway — the stall is a symptom, not a new failure mode. 2. **Only declare `> 1` for a service whose on-disk state is reproducible from git.** A replica with a fresh volume comes up serving whatever it can bootstrap itself and nothing else. For pihole that prerequisite was done first — the admin password and the adlists were declared in `decisions/2026-10-04-pihole-declare-state.md` — because otherwise a second replica would have served DNS while blocking nothing, with a different password. **Requirements:** R-PLAT-02 (the workload), R-STATE-01/02 (per-replica replicated volumes), R-OBS-01 (the SLI must follow the rendered kind). ### `hardware` (`gpu`) — host device capability - **The field is INTENT, never MECHANISM.** `hardware.gpu` is an **enum of capability classes** (today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The ServiceDefinition says *what class of host device this workload needs*; how the platform hands it over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw `/dev/...` path.** The payoff is no longer hypothetical: on **2026-10-01** the mechanism was swapped outright — from a `/dev/dri` hostPath to the Intel GPU device plugin's extended resource — as a **`catalog/render.py` change plus a re-render, with ZERO catalog edits** and no re-review of per-service device wiring. `catalog/plex.yaml`'s `hardware:` block came through byte-identical. Extend the enum for a genuinely different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one. - `gpu: intel-quicksync` → on the **app container only** (never a sidecar), **exactly one thing**: resources: limits: gpu.intel.com/i915: 1 **That is the whole mapping.** No volume, no volumeMount, no `securityContext`, no `nodeSelector`, no privileged anything. - **`limits`, not `requests`, and that is not a style choice.** An extended resource must be expressed in `limits`; Kubernetes mirrors `requests` from `limits` automatically, and a requests-only extended resource is rejected as invalid. - The resource is advertised by the **`intel-gpu-plugin` DaemonSet** (`platform/intel-gpu-plugin/`, its own Flux Kustomization at `clusters/base/intel-gpu-plugin.yaml`, pinned `intel/intel-gpu-plugin:0.37.0`). **Measured 2026-10-01:** all three prod nodes report `.status.allocatable["gpu.intel.com/i915"] == 1`; DaemonSet 3 desired / 3 ready / 3 available. - This is the **only** app-container `resources` render path in the file, and it is deliberately narrow. **Do not generalise it into a CPU/memory resources feature** while passing through — that is a separate field with a separate justification, and `render.py --selftest` asserts the emitted dict *exactly* so the attempt fails loudly. - **WHY THE hostPath MECHANISM WAS ABANDONED — read this before "simplifying" it back.** The first implementation (merged 2026-09-30) rendered a `dri` hostPath volume at `/dev/dri` plus a read-write mount. It was **measured not to work** in the live Plex pod on k3s-prod-2 on **2026-10-01**. The device node was **visible** in the container — `crw-rw---- root video2 226,128 renderD128` — and **could not be opened**: | attempt | result | |---|---| | `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" | | `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" | `EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions — the two look identical from the outside and are not the same problem. Proven three ways: root on the **host** opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc `config.json` reads `linux.resources.devices: [{allow: False, access: 'rwm'}]` with `linux.devices: None` — **one deny-all rule, no allow entry for major 226**. A hostPath is a *filesystem* operation: it binds the device node in but adds **no cgroup allow rule**, because Kubernetes cannot infer one from a volume. **Note that `uid 0` failed too** — root does not bypass the cgroup device gate, so "run it as root" is not a workaround. Neither is `privileged: true`: Kyverno `restrict-host-namespaces` is **Enforce** on app namespaces and rejects it. The device plugin is what produces the allow rule — confirmed in the v0.37.0 source, `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns into both the device injection and the matching cgroup allow. - **THE FAILURE MODE THIS TRADED INTO IS WORSE IN BLAST RADIUS, AND IT IS UNAVOIDABLE.** Stated plainly rather than sold: | | device/resource unavailable ⇒ | |---|---| | **Before** (hostPath, `DirectoryOrCreate`) | kubelet mounts an empty dir; Plex runs and **silently software-transcodes**. Invisible, but harmless to everything else. | | **Now** (extended resource) | the pod is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. | There is **no graceful-degradation variant**: an extended resource is a hard scheduling constraint, and it is also the only thing that produces the cgroup allow rule. The workload's schedulability is now **coupled to the `intel-gpu-plugin` DaemonSet** — a plugin that is broken, removed, or merely not yet reconciled takes the app tier with it. That is the price of hardware transcoding working at all. - **Escape hatch, one field and no code change:** delete the `hardware:` block from the ServiceDefinition (`catalog/plex.yaml`) and re-render. The workload returns to software transcoding and the tier unblocks. `runbooks/plex-quicksync.md` carries it as a first-class procedure, because it is what someone will need at 2am. - The old mechanism's cost is **gone**, and that is the one improvement: a missing device is no longer silent. There is still no metric and no alert (media apps here render no `metricsEndpoint`, so no ServiceMonitor exists), but "Plex is Unschedulable" is loud, whereas "Plex is quietly software-transcoding" never was. - **`podSecurityContext.supplementalGroups` is deliberately NOT rendered, and adding it would be a regression rather than a hardening.** This survives the mechanism swap unchanged, and it was re-measured against the plugin's behaviour. `renderD128` is `root:render` `0660`, so the obvious move is to push the host `render` GID in — and it does not work. **Measured in the running container on prod-2 (2026-09-30, re-measured 2026-10-01):** the image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` hook `find`s every char device under `/dev/dri`, `stat -c '%g'` each one's GID, `groupadd`s a matching group *inside the container* when none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded. It does exactly the same thing to a **plugin-injected** device as to a bind-mounted one, which is why this swap needed no group work. `cont-init.d` completes before `services.d`, so the account database is already correct when the service execs. The service then starts via `exec s6-setuidgid plex …`, and `s6-setuidgid` **replaces** the process group list from `/etc/group` rather than appending to it — measured live, `Plex Media Server` ran with `Groups: 44 100 992 1000`, where **992 is the host `render` GID**, while root's gid `0` (which PID 1, `s6-svscan`, does hold) was **gone**. A Kubernetes `supplementalGroups` therefore reaches `s6-svscan` and is discarded before the app ever starts. It would be inert *and* it would mislead whoever debugs this next. The renderer asserts its own absence in `render.py --selftest`. - A corollary worth having: because the hook re-`stat`s the device on every start, the host GID **drifting** (a node rebuild renumbering the *systemd dynamic* `render` group) is self-healing. No GID is pinned anywhere in this repo. - **Scope of that measurement, stated so nobody over-reads it.** "The group is the image's job" was established for an image that resolves the device group itself. A FUTURE consumer that runs non-root and has no equivalent init hook would still need the group solved somehow — and the answer for that case is a **renderer** change (per-image handling), not a new ServiceDefinition field. The enum is what keeps that option open, and the 2026-10-01 swap is the proof it works. The first such consumer must re-measure rather than assume this bullet covers it. - **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and it would not have helped anyway — see the `EPERM` table); no `nodeSelector` (placement is a separate concern with its own justification — this field is about the device, not about where the pod lands; the extended resource does constrain placement to nodes the plugin advertises, but that is the **scheduler's** doing, not a selector this renderer writes); no CPU/memory requests or limits (see above — the `resources` block carries the extended resource and nothing else). - **Historical, kept so a reader recognises it rather than reintroduces it:** `type: DirectoryOrCreate` on the retired hostPath was load-bearing *for that mechanism* — it is what made a GPU-less node degrade to software transcoding instead of failing the mount and stalling the app tier. That rationale died with the hostPath; it is **not** an argument for bringing it back, since the mechanism it protected never granted access in the first place. - Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical (no `resources` key, no volume, no mount, no `securityContext`). Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`. - **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need). ### `ingressRoutes[]` (`host` / `port`) — HTTPRoute + Certificate + DNS - Each entry renders an **HTTPRoute** attaching `host` → the service `port` on the shared Gateway. - TLS: the Gateway terminates with the cert-manager **wildcard `*.home.keithjackson.dev`** certificate (Let's Encrypt via **DNS-01**, scoped Cloudflare token), so no per-service cert step — trusted, non-self-signed, auto-renewed, including for `internal` services. Per-service `Certificate` only where isolation warrants. - DNS: **ExternalDNS** watches the HTTPRoute and manages the `` record — but **only for a `public` service**. An `internal` service's route is rendered with ExternalDNS's controller annotation set to `none`, which removes it from ExternalDNS's view entirely (see §`exposure`), so its `` exists **only** in the internal split-horizon answer PiHole serves, never in the public zone. A `public` record's TARGET is still the Gateway VIP, not a routable address — open operator decision, see §`exposure`. - The `host` pattern is enforced to `