Files
home-network-k3s-impl/schemas/renderer-contract.md
T
jakman85andClaude Opus 5 99560c2d22
gate / gate (pull_request) Failing after 1m44s
gate / conformance (pull_request) Has been skipped
feat(auth): declare API paths per service, so --api-route can ship at all
PR #28 fixed the cookie collision, the 5-minute session and the missing metrics
listener in one renderer function, and deliberately declined `--api-route`
because the config model could not express the difference between two apps.
This adds the hook it was missing and sets the flag only where a measurement
says to.

THE DEFECT. oauth2-proxy answers an unauthenticated request with a 302 to
Keycloak. Its AJAX guard would send 401 instead, but it fires only on an EXACT
`Accept: application/json`, which these apps' JavaScript does not consistently
send. So a browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied
silently by the live SSO session, and resolves 200 carrying the app's HTML where
the JavaScript expected JSON. On a write it is worse: a browser following a 302
re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
discarded and the UI reports success.

MEASURED, on two apps and two different paths -- which is the whole argument:
  jackett 2026-09-30: 33 x 302 vs 2 x 401 on /api/ paths, and 4 of 6 POSTs
          answered by the proxy and never reaching Jackett (upstream `-`).
  radarr  2026-10-01, 23h after PR #28 deployed (pod radarr-7d55cbffb9-42hmh,
          0 restarts): 232 unauthenticated requests in the pod's whole life and
          ALL 232 are the same call -- POST /signalr/messages/negotiate from a
          browser, 302, upstream `-`. A left-open tab whose session the cookie
          rename orphaned, polling forever, silently redirected every time and
          NEVER re-prompting. The reported symptom, still in production.

So radarr's broken path is /signalr/ and jackett's is /api/, and PIHOLE SERVES
NO /api/ AT ALL. One function renders all five sidecars, so the cheap fix is a
constant -- and a constant applies one app's measurement to four apps nobody
measured. `auth` was `additionalProperties: false`, so there was no per-app
hook. Adding one named, documented key is the change; the block stays closed.

  schemas/servicedefinition.schema.json  1.8.0 -> 1.9.0: optional
      `auth.apiRoutes`, a list (radarr needs two, and --api-route is natively
      repeatable) of anchored path regexes. Plus the shape invariant that a
      non-empty list requires mode: forward-auth, since no sidecar is rendered
      for oidc/native/none and declaring it there would be a SILENT NO-OP.
  catalog/render.py  emits one --api-route per entry, in order, and RAISES on
      an unanchored entry. The flag is SINGULAR -- verified against the pinned
      v7.8.1 binary's own --help; --api-routes does not exist and the proxy
      refuses to start with it.
  catalog/{jackett,radarr}.yaml  the two measured apps.
  catalog/{pihole,sonarr,qbittorrent}.yaml  comments recording that they get
      NOTHING, and why. Sonarr and qBittorrent were measured (0 unauthenticated
      requests in 23h) and left alone: that is evidence of no TRAFFIC, not of no
      need, and sonarr being nearly radarr's codebase is what makes the guess
      tempting and still a guess.
  apps/wave-0/jackett.yaml, apps/wave-1/radarr.yaml  regenerated. Only TWO
      goldens change; pihole/qbittorrent/sonarr are byte-identical to main,
      which is the per-app property proving itself on the real tree.

BREADTH IS THE DANGEROUS DIRECTION AND IT IS NOT SYMMETRIC. A missing
--api-route is a silent 302. A too-broad one answers a plain PAGE LOAD with
401, so a logged-out operator gets a bare error page and is never offered a
sign-in -- the service looks permanently broken. Each entry must therefore
match `^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$`: anchored at ^/ then literal path
characters. `^/`, `^/.*`, `.*`, `/api/` and `^/api/|^/` are all rejected, all
five provoked. Alternation is refused rather than supported because the flag is
repeatable, so one entry per route costs nothing.

NEW GATE CHECK: conformance/tests/l0-static/oauth2-proxy-api-routes-per-app
(whole-repo scope, so ci/l0.sh's --include-scope union runs it on every
invocation; no bespoke stage, which whole-repo-invariants-wired would fail).
It is a SIBLING of oauth2-proxy-cookie-names-distinct, not an extension of it:
that check's subject is cookie IDENTITY and shares neither inputs nor failure
mode with this, so bundling them would make its name a lie and let a future
api-route decision be expressed only by weakening a COOKIE check.

The check exists for one reason the other gates cannot cover: render-golden
compares the render to the goldens, so a RENDERER CONSTANT plus regenerated
goldens AGREE WITH EACH OTHER while both being wrong. Provoked: the regen exits
0 -- render-golden is happy -- and this check exits 1 with 15 failures naming
pihole, qbittorrent, radarr and sonarr individually. It also asserts the
anchoring pattern is the SAME in the schema and the renderer, because three
surfaces guard this and if they drift the loosest wins silently.

Tested in both directions on throwaway trees outside the repo: compliant tree
rc 0 / 43 assertions / 6 sidecars; pre-change tree rc 1 / 11 failures; and four
targeted mutations (hand-edited golden, widened regex, renderer constant,
deleted schema pattern) each firing their own assertion class and nothing else.

ALSO FOLDED IN (coordinator's request; defect is in the file I own). _selftest()
reached _cookie_name() through a bare `next(genexpr)` with NO default, so
deleting the --cookie-name emission -- the exact mutation that assertion exists
to catch -- raised StopIteration, which escaped _selftest() BEFORE its collected
`SELFTEST FAIL:` lines were printed. Right verdict, zero diagnostics, and it
masked every other assertion the same mutation broke (found by Desmond Oyelaran
while wiring the selftest into the render-golden stage; his PR #33 is not in this
base -- origin/main @ cdf3e14 drives the selftest from no stage, checked). Fixed
with `next(..., None)` plus an explicit check(), so a missing flag reports as a
NAMED assertion. Provoking it exposed three more instances of the same class in
the same block -- `side = fa_containers[1]`, `side["env"][0]`, and the fa_svc
lookup all raise rather than assert when the sidecar is absent -- so all four are
now None-tolerant, and the two fixtures added for --api-route were written that
way from the start. Measured on throwaway copies:
  --cookie-name deleted:  before rc 1 / 0 FAIL lines / StopIteration
                          after  rc 1 / 3 FAIL lines / no traceback
  sidecar not appended:   before rc 1 / 0 FAIL lines / IndexError
                          after  rc 1 / 20 FAIL lines / no traceback
Goldens are unchanged by this part; it touches _selftest() only.

NOT verified, in full in the handoff entry: no browser round-trip by anyone for
this flag -- that a 401 REPAIRS the operator's experience is reasoned from HTTP
semantics, not watched. `^/api/` on radarr is PR #30's measurement of radarr's
pre-fix pod, not mine; the current pod has zero /api/ traffic. Sonarr and
qBittorrent are UNMEASURED, not cleared. --cookie-refresh is still unexercised
(0 AuthSuccess on all five pods). Nothing was applied; the live pods are still
the pre-change spec.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-01 22:07:08 -05:00

379 lines
30 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Renderer Contract — ServiceDefinition → rendered resources
**Status:** authoritative Phase-0 interface contract. **Consumed by** the manifest renderer
(WS-GITOPS), the network/egress renderer (WS-NET), storage (WS-STATE), TLS/DNS (WS-EDGE), identity
(WS-SEC), observability (WS-OBS), updates (WS-UPD), and the conformance/policy/CI agents (WS-TEST).
**Source of truth:** spec [`docs/13-service-catalog.md`](../../home-network-k3s/docs/13-service-catalog.md)
(the ServiceDefinition schema and default posture). This document maps each field of
[`servicedefinition.schema.json`](./servicedefinition.schema.json) to the concrete resources the
platform must render, and cites the requirement each mapping satisfies. Changing this contract is an
**interface change** — coordinate per the spec's `agents/coordination-protocol.md` (§2).
> The renderer MUST first materialize the schema **defaults** (below), then apply this mapping. A
> field absent from a ServiceDefinition renders from its default posture, not "not at all."
## Default posture (applied before mapping) — docs/13 §"Default posture"
| Field | Default | Requirement |
|---|---|---|
| `kind` | `container` | R-PLAT-02/03 |
| `exposure` | `internal` | R-DNS/R-NET |
| `egress` | `internet` | R-NET-05 |
| `storage` | `[]` (config PVC still expected for stateful svc) | R-STATE-02 |
| `dependencies` | `[]` → reaches **only DNS** (default-deny) | R-NET-01 |
| `inClusterClients` | `[]` → no direct east-west client; only the Gateway may reach an exposed port | R-NET-01/02 |
| `auth.mode` | `forward-auth` for any exposed UI | R-IDP-02 |
| `auth.apiRoutes` | `[]` → the sidecar renders **no** `--api-route`; every unauthenticated request is 302'd to Keycloak | R-IDP-02 |
| `update.channel` | `stable` | R-UPD-01 |
| `update.autoRollbackOnFailedHealth` | `true` | R-UPD-03 |
| `observability.metrics` | `true` | R-OBS-01 |
| `observability.metricsEndpoint` | **absent** → the workload serves no Prometheus endpoint; no ServiceMonitor | R-OBS-01 |
| `observability.livenessWorkload` | the object this SD renders (`Deployment` named after the service; `VirtualMachineInstance` for `kind: vm`) | R-OBS-01 |
| backup of irreplaceable state | off-cluster (never `reacquirable`) | R-STATE-03 |
---
## Field → resource mapping
### `name`
- **Renders:** the service Namespace (trust zone), Deployment/VM name, Service name, and the DNS label
`<name>.home.keithjackson.dev`.
- **Requirements:** R-DNS-01 (naming model, docs/09), R-NET (namespace = isolation boundary, docs/03).
### `kind` + `source`
- `container` + `source.image` → **Deployment** (+ Service) using the image ref.
- `vm` + `source.vmTemplate` → **KubeVirt VirtualMachine** with a **CDI**-imported root disk.
- The schema enforces exactly-one-of (`image` for container, `vmTemplate` for vm).
- **Requirements:** R-PLAT-02/03 (PaaS/IaaS, docs/13); VM disks inherit Longhorn durability (docs/04).
### `exposure` (`internal` | `public`)
- Both classes render an **HTTPRoute** on the shared Gateway and are reachable on the LAN via internal
split-horizon DNS.
- `public` **additionally** signals the documented **manual edge port-forward** of 80/443 (and any
service-specific port, e.g. Plex 32400, Minecraft 25565) to the ingress VIP, and a **public**
ExternalDNS record for `<name>.home.keithjackson.dev`.
- `internal` renders **no public DNS record** and is never port-forwarded (PiHole resolves it to the
internal Gateway VIP).
- **HOW "no public DNS record" IS MADE TRUE, since it was prose only until 2026-09-14.** Every
HTTPRoute rendered for an `internal` service carries ExternalDNS's own controller annotation with
a value other than the one ExternalDNS claims:
external-dns.alpha.kubernetes.io/controller: none
ExternalDNS v0.21.0 (`source/annotations/processors.go`) skips any source object whose
`controller` annotation is present and `!= "dns-controller"`, and `source/gateway.go` applies that
check **inside `Endpoints()` before any hostname is resolved** — so the route never becomes an
endpoint for any name it declares. `source/service.go` applies the identical check, which covers
the second configured source. It is derived **from `exposure` and nothing else**
(`catalog/render.py` `_externaldns_route_annotations`), so the operator's one declaration decides,
a service that OMITS `exposure` inherits the `internal` default and is likewise unpublishable, and
there is no per-service list to forget to extend. `ci/render-golden.sh` proves `apps/` reproduces
from `catalog/`, so an `internal` service cannot exist in the applied tree without it, and
`conformance/tests/l0-static/internal-services-publish-no-public-dns` re-proves both directions
(present on internal, **absent** on public) on every gate run.
- The route itself, the wildcard TLS and LAN reachability are **unchanged** — only the public
record is withdrawn. The internal split-horizon answer is Pi-hole's job (R-DNS-03), which is why
withdrawing these records requires Pi-hole to be answering the LAN authoritatively first: the
zone has no `*.home.keithjackson.dev` wildcard, and `*.keithjackson.dev` does not match two
labels, so a withdrawn name is a public NXDOMAIN with no fallback.
- Why not a cluster-wide ExternalDNS opt-in filter (`labelFilter`, which v0.21.0 supports for
both configured sources): seven published names come from hand-authored `platform/`/`apps/`
manifests that carry no `exposure` field for a renderer to read, and an opt-in filter deletes
all seven the moment it is set. Per-resource exclusion has exactly the blast radius of the
defect. The filter stays available as hardening once those seven are adjudicated — see
`platform/externaldns/helmrelease.yaml` and the register in the L0 check.
- `public` renders the record but the renderer does **not** yet set a target: the record still
resolves to the shared Gateway VIP, i.e. an RFC1918 address, so a `public` service is not in fact
publicly reachable. Fixing that is **not** a renderer change on v0.21.0 —
`source/gateway.go` reads `external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and
a ListenerSet), never from an HTTPRoute, so a per-service target override is not expressible while
the public services share the `shared` Gateway. The open decision is recorded in
`platform/externaldns/helmrelease.yaml`.
- Exposure does **not** relax east-west default-deny (docs/09 §"Interaction with default-deny").
- **Requirements:** R-DNS-02/03 (docs/09), R-NET-04 (docs/03).
### `egress` (`internet` | `vpn`) — egress steering
- `internet` → pod egress allow to the default WAN route (node → SOHO edge → internet).
- `vpn` → the pod is **steered through the shared in-cluster Gluetun/PIA (WireGuard) egress gateway**
(Cilium Egress Gateway / routing). The pod gets **no independent egress path**; the
**kill-switch (fail-closed)** and allow rules are enforced at the gateway + namespace/policy level,
**outside** the workload.
- **Requirements:** **R-NET-05** (egress class), **R-NET-06** (VPN kill-switch, mandatory & central).
VPN-class set is normative: sonarr, radarr, jackett, qbittorrent (docs/03/13).
### `dependencies[]` (`to` / `port` / `protocol`) — east-west policy
- Each namespace first gets the **default-deny baseline** (deny all ingress + egress) applied by
GitOps at namespace creation, with a standing allow only for **DNS to the in-cluster resolver**.
**Observability scraping is NOT in the baseline** — it is an additive allow rendered per declared
`observability.metricsEndpoint` (see below), scoped to that one port. This paragraph used to claim
the baseline carried a standing monitoring allow; it never did, and an app metrics port would have
been dropped by policy even with the Service port and monitor endpoint in agreement.
- Each `dependencies[]` entry renders one **CiliumNetworkPolicy** egress allow: this service → `to`
service on `port`/`protocol`. Everything undeclared stays denied.
- Empty `dependencies` ⇒ the service can reach **only DNS**.
- **Requirements:** **R-NET-01** (default-deny baseline), **R-NET-02** (explicit allow / deny
undeclared). Auditable via Hubble (R-SEC-03, R-OBS-04).
### `storage[]` (`name` / `size` / `class` / `mountPath` / `export` / `subPath` / `readOnly`) — PVs + PVCs
- `class: longhorn` → a **PVC** on the 3-replica Longhorn StorageClass sized by `size` (replicated
config/DB/app state), `ReadWriteOnce`, dynamically provisioned. For a `vm`, the disk is a Longhorn
volume via CDI.
- `class: bulk` → a **static, pre-bound external NFS PersistentVolume _and_ its PVC**. The entry's
`export` is the absolute path on the bulk server; the SERVER is never named in a ServiceDefinition
— it is the `${BULK_NFS_SERVER}` cluster-vars knob, so one authored definition promotes across
environments that reach the same storage by different addresses (**R-ENV-02**). Not 3× replicated:
durability is the storage host's ZFS pool, not Longhorn.
- `storageClassName: ""` on **both** PV and PVC (explicitly empty ⇒ statically provisioned; no
provisioner is ever asked to create anything, and the default-StorageClass annotation is
suppressed). Deliberately NOT the `bulk` StorageClass: a StorageClass `provisioner` is
**immutable**, so re-pointing that stub would make Flux fail permanently on it.
- Binding is pinned from both ends — `claimRef` on the PV, `volumeName` on the PVC — so no other
claim of matching size/mode can win a race for the volume.
- `persistentVolumeReclaimPolicy: Retain`, always. Bulk data pre-exists the cluster, is far larger
than the cluster, and may have consumers outside it; nothing here may reclaim it.
- Access mode follows the read/write posture and is always `…Many`, because one export is
consumed by several namespaces and several nodes at once: `ReadOnlyMany` when `readOnly`,
`ReadWriteMany` otherwise. `ReadWriteOnce` cannot express either.
- `mountOptions` are a **platform** decision held in the renderer, not a per-service field:
`nfsvers=4.1,hard,noatime,timeo=600,retrans=2`. NFSv4 carries mount and lock in-protocol, so
the only port that must be reachable is **tcp/2049** — no rpcbind/111, mountd or statd, which is
what keeps the firewall surface to a single rule. `hard` is chosen over `soft` because a soft
mount converts a server hiccup into a short read handed to the application.
- `size` on a bulk entry is the PV's **advertised capacity and nothing more** — no component
enforces it (the ZFS pool is the real limit), so it should state the dataset's actual size so
`kubectl get pv` reports something true.
- **fsGroup does not apply.** The in-tree NFS volume plugin performs no ownership management, so
`podSecurityContext.fsGroup` is a **no-op** on a bulk volume. A bulk mount that must be written
therefore requires the container's own uid/gid to be one the export's ownership permits (e.g.
Plex's `PLEX_UID`/`PLEX_GID`, LinuxServer images' `PUID`/`PGID`) — or a server-side squash. A
mount that succeeds but cannot write is the characteristic failure of this tier.
- `mountPath` (any class) overrides the `/data/<name>` default verbatim.
- `subPath` (any class) mounts a subtree instead of the volume root. **Two `subPath` mounts of one
export are two separate mounts**, so `link()` between them fails `EXDEV`; a workload that needs
hardlinks or atomic moves between two directories (Sonarr/Radarr importing from a download path
into a library) must mount their common ancestor **once** instead.
- `readOnly` (any class) marks the volumeMount read-only, and on a bulk entry also sets the PV's
`nfs.readOnly` so the kernel mount really is `ro`. Note this is a **client-side** guarantee: one
export serving both a reader and a writer that may land on any node cannot distinguish them
server-side.
- State separation rule: anything that must survive a rebuild lives on a named volume here, never on
ephemeral fs/hostPath — so the workload can be destroyed and re-applied from git with data intact.
- **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication),
**R-STATE-04** (bulk tier). (docs/04)
### `hardware` (`gpu`) — host device capability
- **The field is INTENT, never MECHANISM.** `hardware.gpu` is an **enum of capability classes**
(today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
ServiceDefinition says *what class of host device this workload needs*; how the platform hands it
over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw
`/dev/...` path.** The entire payoff is that migrating from the hostPath mechanism below to the
Intel GPU device plugin (`resources.limits: gpu.intel.com/i915`) is a renderer change with **zero**
catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely
different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one.
- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), exactly two things:
1. a **hostPath volume** `dri` → `/dev/dri`, **`type: DirectoryOrCreate`**;
2. a **read-write volumeMount** of `dri` at `/dev/dri` on the app container (VA-API opens the
render node read-write, so it is deliberately not `readOnly`).
**Mounting the device is sufficient. That is the whole mapping.**
- **`podSecurityContext.supplementalGroups` is deliberately NOT rendered, and adding it would be a
regression rather than a hardening.** `/dev/dri/renderD128` is `root:render` `0660`, so the obvious
move is to push the host `render` GID in — and it does not work. **Measured in the running
container on prod-2, 2026-09-30:** the image's own
`/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` hook `find`s every char device under
`/dev/dri`, `stat -c '%g'` each one's GID, `groupadd`s a matching group *inside the container* when
none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded.
`cont-init.d` completes before `services.d`, so the account database is already correct when the
service execs. The service then starts via `exec s6-setuidgid plex …`, and `s6-setuidgid`
**replaces** the process group list from `/etc/group` rather than appending to it:
`Plex Media Server` runs as `Uid: 1000 Gid: 1000 Groups: 100 1000`, with root's gid `0` — which
PID 1 (`s6-svscan`) does hold — **gone**. A Kubernetes `supplementalGroups` therefore reaches
`s6-svscan` and is discarded before the app ever starts. It would be inert *and* it would mislead
whoever debugs this next. The renderer asserts its own absence in `render.py --selftest`.
- A corollary worth having: because the hook re-`stat`s the device on every start, the host GID
**drifting** (a node rebuild renumbering the *systemd dynamic* `render` group) is self-healing.
No GID is pinned anywhere in this repo.
- **Scope of that measurement, stated so nobody over-reads it.** "Mounting the device is
sufficient" was established for an image that resolves the device group itself. A FUTURE
consumer that runs non-root and has no equivalent init hook would still need the group solved
somehow — and the answer for that case is a **renderer** change (per-image handling, or the
device plugin), not a new ServiceDefinition field. The enum is what keeps that option open. The
first such consumer must re-measure rather than assume this bullet covers it.
- **`DirectoryOrCreate` is load-bearing — do not tighten it to `Directory`.** A node that comes back
without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the
workload to **software transcoding**, not fail the mount. `apps-wave-0` is `wait: true` and
`apps-wave-1` `dependsOn` it, so a failed hostPath mount stalls the **whole app tier** for the sake
of a feature that is not load-bearing. `/dev` is tmpfs, so the empty directory the kubelet creates
on such a node leaves nothing durable behind.
- **The cost of that choice, stated plainly:** a missing device is **silent**. The workload
software-transcodes and nothing reports it. There is no metric to alert on — media apps here
render no `metricsEndpoint`, so no ServiceMonitor exists — and an alert that cannot fire is worse
than none (see §`observability`'s 2026-09-13 correction). The only proof of hardware transcode is
an operator observing it in the application's own dashboard.
- **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno
`restrict-host-namespaces` is Enforce on app namespaces and would reject it, and the render node
does not need it); no `nodeSelector` (placement is a separate concern with its own justification —
this field is about the device, not about where the pod lands); no resource requests/limits (there
is no app-container resource render path, and inventing one here would be scope creep).
- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical.
Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`.
- **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need).
### `ingressRoutes[]` (`host` / `port`) — HTTPRoute + Certificate + DNS
- Each entry renders an **HTTPRoute** attaching `host` → the service `port` on the shared Gateway.
- TLS: the Gateway terminates with the cert-manager **wildcard `*.home.keithjackson.dev`** certificate
(Let's Encrypt via **DNS-01**, scoped Cloudflare token), so no per-service cert step — trusted,
non-self-signed, auto-renewed, including for `internal` services. Per-service `Certificate` only
where isolation warrants.
- DNS: **ExternalDNS** watches the HTTPRoute and manages the `<host>` record — but **only for a
`public` service**. An `internal` service's route is rendered with ExternalDNS's controller
annotation set to `none`, which removes it from ExternalDNS's view entirely (see §`exposure`), so
its `<host>` exists **only** in the internal split-horizon answer PiHole serves, never in the
public zone. A `public` record's TARGET is still the Gateway VIP, not a routable address — open
operator decision, see §`exposure`.
- The `host` pattern is enforced to `<label>.home.keithjackson.dev` by the schema.
- **Requirements:** **R-TLS-01/02/03** (docs/08), **R-DNS-01/02/03** (docs/09).
### `inClusterClients[]` (`namespace` / `port` / `protocol` / `name`) — direct east-west clients
- Each entry renders ONE **CiliumNetworkPolicy** ingress allow on this service:
`endpointSelector` = this service, `fromEndpoints` matching **only** the reserved
`k8s:io.kubernetes.pod.namespace: <namespace>` label, `toPorts` carrying **only** the declared
`port`/`protocol`. Name: `<service>-allow-<name>-ingress`, `name` defaulting to `namespace`.
- **Why the field exists.** `ingressRoutes` opens the **north-south** path only (the gateway allow
admits `fromEntities: [ingress]`, the Gateway proxy identity, and nothing else), and
`dependencies` compiles edges **only between catalog services** — both halves come from a catalog
entry's own declaration. A **platform controller** that dials this service's ClusterIP directly
was therefore unauthorable, and under default-deny it is dropped. Prod hit exactly that: its
`GitRepository` points at the in-cluster Gitea Service
(`http://gitea.gitea.svc.cluster.local:3000/...`, clusters/prod/kustomization.yaml) so GitOps
depends on no public DNS/Cloudflare/Gateway/`${BASE_DOMAIN}`, but `flux-system`'s
source-controller was denied — and the HTTPRoute that would have served a public name lives in a
**downstream** tier that cannot apply while the source is down (source down → routes never rename
→ source stays down).
- **Least privilege is part of the contract:** a namespace and a port, never `fromEntities: all`,
never a port-less allow. The client is selected by **namespace** rather than pod labels because an
upstream controller's labels are its chart's to change, not this platform's contract — so name a
platform-controller namespace whose entire membership may hold that reach, never a shared one.
- **Additive, and not a dependency:** the namespace default-deny baseline is unchanged
(`ingress: []`), every undeclared source/port stays denied, and the entry creates **no wave
ordering edge** — the client is a platform component, not a catalog service.
- **Requirements:** **R-NET-01** (default-deny baseline stands), **R-NET-02** (explicit allow / deny
undeclared), **R-GITOPS-02** (the allowance is authored in the one reviewable object rather than
surviving as a live-applied cluster object). Auditable via Hubble (R-SEC-03, R-OBS-04).
### `auth` (`mode` / `groups[]` / `apiRoutes[]`) — identity integration (Keycloak / oauth2-proxy)
- `oidc` → register the app as a **Keycloak OIDC client** (declarative realm/client config, GitOps;
client secret SOPS-encrypted). App does its own login redirect.
- `forward-auth` → protect the service's `ingressRoutes` at the **Gateway** with **oauth2-proxy →
Keycloak** so auth happens before traffic reaches the app (bare UIs without usable OIDC).
- `native` → no platform SSO wiring; app owns its auth (documented exception, e.g. Plex/Seerr; and
Keycloak **itself** as the IdP, admin login local + MFA).
- `none` → no auth wiring (internal-only, discouraged).
- `groups[]` → the Keycloak groups permitted access, enforced by the OIDC client mappers /
oauth2-proxy `allowed_groups`. **Schema requires ≥1 group for `oidc`/`forward-auth`** — access is
"in an authorized group," not merely "logged in."
- `apiRoutes[]` (**`forward-auth` only**) → one repeatable **`--api-route=<regex>`** per entry on
the per-app oauth2-proxy sidecar, making matching paths answer **401** instead of a **302** to
Keycloak. Default `[]` renders **no flag at all**, which is the common case. It exists because
oauth2-proxy's built-in AJAX guard fires only on an exact `Accept: application/json`, which
these apps' JavaScript does not consistently send: the XHR is 302'd, the browser follows it
invisibly, the live SSO session satisfies it silently, and the call resolves 200 carrying the
app's **HTML** where the JavaScript expected JSON — and a POST following a 302 is re-issued as
a GET with its **body dropped**. It is **per-service and must never become a renderer
constant**: the path shape belongs to the app (radarr `/signalr/`, jackett `/api/`, pihole
none), and a wrong constant does not merely fail to help — a path a **browser navigates to**,
answered 401, shows a logged-out operator a bare error page and never offers a sign-in. Each
entry must be **anchored at `^/`** and continue with literal path characters; `^/`, `^/.*`,
`.*`, an unanchored `/api/` and any alternation are **rejected** by the schema and by the
renderer (alternation is refused rather than supported because the flag is repeatable).
- Keycloak, oauth2-proxy, and app→Keycloak / proxy→Keycloak / Keycloak→DB flows are ordinary
workloads under default-deny with explicit allows (ties back to `dependencies`).
- **Requirements:** **R-IDP-02** (protect OIDC-capable and bare UIs), **R-IDP-06** (group-based
authorization). (docs/15, ADR-0014)
### `update` (`channel` / `range` / `cadence` / `autoRollbackOnFailedHealth` / `pin`)
- container `kind` → a Flux **ImagePolicy** (+ ImageRepository) encoding `channel`/`range`; Flux
image-automation advances the running version on its scan interval (`cadence`). `pin` freezes the
service (stops automation). No human/AI edits a tag.
- `vm` `kind` → in-guest unattended package updates on `cadence` (cloud-init), per docs/11 §3.
- `autoRollbackOnFailedHealth: true` → health-gated rollout that reverts on failed health and alerts.
- **Requirements:** **R-UPD-01** (stable only), **R-UPD-02** (no human/AI authoring), **R-UPD-03**
(health-gated auto-rollback). (docs/11, ADR-0009)
### `observability` (`metrics` / `slis[]` / `metricsEndpoint` / `livenessWorkload`) — ServiceMonitor + alerts
- `metrics: true` → the platform **collects signals** for this service: logs to Loki via the node
agent, and kube-state-metrics readiness for the `up` SLI below. Both are platform-wide scrapes;
neither needs a per-service CR. **It does not assert that the workload serves a Prometheus
endpoint.** *(Corrected 2026-09-13. It used to render a ServiceMonitor unconditionally, hardcoded
to a Service port named `metrics`. No rendered Service ever defined such a port, so all twelve app
monitors discovered ZERO targets — verified on prod: 24 jobs in `up`, not one of the twelve — and
every `up{job=...}` SLI built on them was structurally inert. `metrics: true` could not have been
right: Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud serve no
Prometheus endpoint at all without a separate exporter.)*
- `metricsEndpoint: {port, name?, path?, interval?}` → **the workload does serve metrics, here.**
Renders all three halves of a working scrape from the one declaration, because any one of them
missing makes the scrape silently yield nothing:
1. a **named Service port** (`name`, default `metrics`) on that `port`;
2. a **ServiceMonitor** endpoint selecting that port name, on `path` at `interval`;
3. a **CiliumNetworkPolicy** admitting the `monitoring` namespace to that port — the namespace
default-deny baseline is `ingress: []`, so without it the scrape is dropped by policy.
The metrics port is deliberately **not** added to the Gateway ingress allow: it is east-west for
Prometheus, never north-south.
- each entry in `slis[]` → at least one **alert rule** routed by Alertmanager to email; a per-service
Grafana dashboard row. `up` → a `<name>-down` **critical** alert reading the readiness of
`livenessWorkload` from **kube-state-metrics** (`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready`, `kube_daemonset_status_number_available`) or, for a
`VirtualMachineInstance`, the existence of `kubevirt_vmi_info{phase="running"}`. Each rule carries
an `or absent(...)` half, so the signal DISAPPEARING fires the alert rather than silencing it.
- `livenessWorkload: {kind, name}` → which object's readiness **is** this service. Defaults to the
object this SD renders; override it when the running workload is a different one (keycloak's server
is a StatefulSet owned by the Keycloak operator — a Deployment-shaped rule would `absent()` and
fire forever).
- **Requirements:** **R-OBS-01** (per-service SLIs/alerts). Platform-health alerts (R-OBS-04) are
rendered platform-wide, not per ServiceDefinition. (docs/10)
### `backup` (`policy`)
- a schedule ref (e.g. `daily-offcluster`) → **Velero + Longhorn** off-cluster backup schedule for
this service's volumes/namespace.
- literal `reacquirable` → **no** 1:1 backup (bulk media only). **Irreplaceable** state (configs, DBs,
Nextcloud, Gitea, PiHole config, Keycloak DB) MUST NOT be `reacquirable`.
- **Requirements:** **R-STATE-03** (off-cluster DR backup). (docs/04)
---
## Cross-field invariants the renderer / L0-L1 tests rely on
1. **kind ↔ source:** container⇒`image`, vm⇒`vmTemplate` (schema-enforced).
2. **auth ↔ groups:** `oidc`/`forward-auth` ⇒ non-empty `groups` (schema-enforced; R-IDP-06).
2a. **auth ↔ apiRoutes:** a non-empty `auth.apiRoutes` ⇒ `auth.mode: forward-auth`
(schema-enforced). It renders onto the per-app oauth2-proxy sidecar, and no sidecar exists for
`oidc`/`native`/`none`, so declaring it elsewhere would be a **silent no-op** — the operator
believes the app's XHRs now answer 401 and nothing changed. Every entry must be anchored at
`^/` (schema-enforced **and** enforced in `catalog/render.py`, so a breadth mistake is caught by
`render-golden` even on a run where the schema stage did not execute).
3. **default-deny is unconditional:** every namespace gets the deny baseline regardless of
`dependencies`; `dependencies` only *adds* allows (R-NET-01/02). The same holds for
`inClusterClients`: it adds one namespace+port ingress allow and never relaxes the baseline.
4. **egress is enforced outside the workload:** `vpn` renders gateway steering + kill-switch, never a
per-service sidecar the app could bypass (R-NET-05/06).
5. **TLS is never self-signed and needs no per-service step:** the wildcard Gateway cert covers every
`ingressRoutes.host`, `internal` included (R-TLS-01/03).
6. **irreplaceable ⇒ off-cluster backup:** DR-critical services must not carry `backup.policy:
reacquirable` (R-STATE-03). *(Advisory: not enforced by the schema — an L0 policy check should
assert it; see "Ambiguities / escalations" in schemas/README.md.)*
7. **`exposure: internal` ⇒ no public DNS record, structurally.** Every HTTPRoute rendered for an
`internal` service (including one that omits `exposure` and takes the default) carries
`external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it.
Derived from `exposure` alone; enforced on every gate run by
`conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03).
8. **`hardware` is inert until opted into, and its mechanism is renderer-local.** An SD with no
`hardware` key renders no volume and no volumeMount; an SD *with* it gains no `securityContext`
either. Both directions are proven by `render.py --selftest`. The `/dev/dri` hostPath and the
`DirectoryOrCreate` type appear in `catalog/render.py` and nowhere else, so the capability's
implementation can be replaced without touching a single ServiceDefinition.
## Requirement traceability (contract-level)
Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02
(dependencies, inClusterClients), R-NET-05/06 (egress), R-STATE-01/02/04 (storage), R-STATE-03 (backup),
R-TLS-01/02/03 + R-DNS-01/02/03 (ingressRoutes + exposure), R-OBS-01 (observability),
R-UPD-01/02/03 (update), R-IDP-02/06 (auth), R-PLAT-02/03 (kind/source).