PR #28 fixed the cookie collision, the 5-minute session and the missing metrics
listener in one renderer function, and deliberately declined `--api-route`
because the config model could not express the difference between two apps.
This adds the hook it was missing and sets the flag only where a measurement
says to.
THE DEFECT. oauth2-proxy answers an unauthenticated request with a 302 to
Keycloak. Its AJAX guard would send 401 instead, but it fires only on an EXACT
`Accept: application/json`, which these apps' JavaScript does not consistently
send. So a browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied
silently by the live SSO session, and resolves 200 carrying the app's HTML where
the JavaScript expected JSON. On a write it is worse: a browser following a 302
re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
discarded and the UI reports success.
MEASURED, on two apps and two different paths -- which is the whole argument:
jackett 2026-09-30: 33 x 302 vs 2 x 401 on /api/ paths, and 4 of 6 POSTs
answered by the proxy and never reaching Jackett (upstream `-`).
radarr 2026-10-01, 23h after PR #28 deployed (pod radarr-7d55cbffb9-42hmh,
0 restarts): 232 unauthenticated requests in the pod's whole life and
ALL 232 are the same call -- POST /signalr/messages/negotiate from a
browser, 302, upstream `-`. A left-open tab whose session the cookie
rename orphaned, polling forever, silently redirected every time and
NEVER re-prompting. The reported symptom, still in production.
So radarr's broken path is /signalr/ and jackett's is /api/, and PIHOLE SERVES
NO /api/ AT ALL. One function renders all five sidecars, so the cheap fix is a
constant -- and a constant applies one app's measurement to four apps nobody
measured. `auth` was `additionalProperties: false`, so there was no per-app
hook. Adding one named, documented key is the change; the block stays closed.
schemas/servicedefinition.schema.json 1.8.0 -> 1.9.0: optional
`auth.apiRoutes`, a list (radarr needs two, and --api-route is natively
repeatable) of anchored path regexes. Plus the shape invariant that a
non-empty list requires mode: forward-auth, since no sidecar is rendered
for oidc/native/none and declaring it there would be a SILENT NO-OP.
catalog/render.py emits one --api-route per entry, in order, and RAISES on
an unanchored entry. The flag is SINGULAR -- verified against the pinned
v7.8.1 binary's own --help; --api-routes does not exist and the proxy
refuses to start with it.
catalog/{jackett,radarr}.yaml the two measured apps.
catalog/{pihole,sonarr,qbittorrent}.yaml comments recording that they get
NOTHING, and why. Sonarr and qBittorrent were measured (0 unauthenticated
requests in 23h) and left alone: that is evidence of no TRAFFIC, not of no
need, and sonarr being nearly radarr's codebase is what makes the guess
tempting and still a guess.
apps/wave-0/jackett.yaml, apps/wave-1/radarr.yaml regenerated. Only TWO
goldens change; pihole/qbittorrent/sonarr are byte-identical to main,
which is the per-app property proving itself on the real tree.
BREADTH IS THE DANGEROUS DIRECTION AND IT IS NOT SYMMETRIC. A missing
--api-route is a silent 302. A too-broad one answers a plain PAGE LOAD with
401, so a logged-out operator gets a bare error page and is never offered a
sign-in -- the service looks permanently broken. Each entry must therefore
match `^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$`: anchored at ^/ then literal path
characters. `^/`, `^/.*`, `.*`, `/api/` and `^/api/|^/` are all rejected, all
five provoked. Alternation is refused rather than supported because the flag is
repeatable, so one entry per route costs nothing.
NEW GATE CHECK: conformance/tests/l0-static/oauth2-proxy-api-routes-per-app
(whole-repo scope, so ci/l0.sh's --include-scope union runs it on every
invocation; no bespoke stage, which whole-repo-invariants-wired would fail).
It is a SIBLING of oauth2-proxy-cookie-names-distinct, not an extension of it:
that check's subject is cookie IDENTITY and shares neither inputs nor failure
mode with this, so bundling them would make its name a lie and let a future
api-route decision be expressed only by weakening a COOKIE check.
The check exists for one reason the other gates cannot cover: render-golden
compares the render to the goldens, so a RENDERER CONSTANT plus regenerated
goldens AGREE WITH EACH OTHER while both being wrong. Provoked: the regen exits
0 -- render-golden is happy -- and this check exits 1 with 15 failures naming
pihole, qbittorrent, radarr and sonarr individually. It also asserts the
anchoring pattern is the SAME in the schema and the renderer, because three
surfaces guard this and if they drift the loosest wins silently.
Tested in both directions on throwaway trees outside the repo: compliant tree
rc 0 / 43 assertions / 6 sidecars; pre-change tree rc 1 / 11 failures; and four
targeted mutations (hand-edited golden, widened regex, renderer constant,
deleted schema pattern) each firing their own assertion class and nothing else.
ALSO FOLDED IN (coordinator's request; defect is in the file I own). _selftest()
reached _cookie_name() through a bare `next(genexpr)` with NO default, so
deleting the --cookie-name emission -- the exact mutation that assertion exists
to catch -- raised StopIteration, which escaped _selftest() BEFORE its collected
`SELFTEST FAIL:` lines were printed. Right verdict, zero diagnostics, and it
masked every other assertion the same mutation broke (found by Desmond Oyelaran
while wiring the selftest into the render-golden stage; his PR #33 is not in this
base -- origin/main @ cdf3e14 drives the selftest from no stage, checked). Fixed
with `next(..., None)` plus an explicit check(), so a missing flag reports as a
NAMED assertion. Provoking it exposed three more instances of the same class in
the same block -- `side = fa_containers[1]`, `side["env"][0]`, and the fa_svc
lookup all raise rather than assert when the sidecar is absent -- so all four are
now None-tolerant, and the two fixtures added for --api-route were written that
way from the start. Measured on throwaway copies:
--cookie-name deleted: before rc 1 / 0 FAIL lines / StopIteration
after rc 1 / 3 FAIL lines / no traceback
sidecar not appended: before rc 1 / 0 FAIL lines / IndexError
after rc 1 / 20 FAIL lines / no traceback
Goldens are unchanged by this part; it touches _selftest() only.
NOT verified, in full in the handoff entry: no browser round-trip by anyone for
this flag -- that a 401 REPAIRS the operator's experience is reasoned from HTTP
semantics, not watched. `^/api/` on radarr is PR #30's measurement of radarr's
pre-fix pod, not mine; the current pod has zero /api/ traffic. Sonarr and
qBittorrent are UNMEASURED, not cleared. --cookie-refresh is still unexercised
(0 AuthSuccess on all five pods). Nothing was applied; the live pods are still
the pre-change spec.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
30 KiB
Renderer Contract — ServiceDefinition → rendered resources
Status: authoritative Phase-0 interface contract. Consumed by the manifest renderer (WS-GITOPS), the network/egress renderer (WS-NET), storage (WS-STATE), TLS/DNS (WS-EDGE), identity (WS-SEC), observability (WS-OBS), updates (WS-UPD), and the conformance/policy/CI agents (WS-TEST).
Source of truth: spec docs/13-service-catalog.md
(the ServiceDefinition schema and default posture). This document maps each field of
servicedefinition.schema.json to the concrete resources the
platform must render, and cites the requirement each mapping satisfies. Changing this contract is an
interface change — coordinate per the spec's agents/coordination-protocol.md (§2).
The renderer MUST first materialize the schema defaults (below), then apply this mapping. A field absent from a ServiceDefinition renders from its default posture, not "not at all."
Default posture (applied before mapping) — docs/13 §"Default posture"
| Field | Default | Requirement |
|---|---|---|
kind |
container |
R-PLAT-02/03 |
exposure |
internal |
R-DNS/R-NET |
egress |
internet |
R-NET-05 |
storage |
[] (config PVC still expected for stateful svc) |
R-STATE-02 |
dependencies |
[] → reaches only DNS (default-deny) |
R-NET-01 |
inClusterClients |
[] → no direct east-west client; only the Gateway may reach an exposed port |
R-NET-01/02 |
auth.mode |
forward-auth for any exposed UI |
R-IDP-02 |
auth.apiRoutes |
[] → the sidecar renders no --api-route; every unauthenticated request is 302'd to Keycloak |
R-IDP-02 |
update.channel |
stable |
R-UPD-01 |
update.autoRollbackOnFailedHealth |
true |
R-UPD-03 |
observability.metrics |
true |
R-OBS-01 |
observability.metricsEndpoint |
absent → the workload serves no Prometheus endpoint; no ServiceMonitor | R-OBS-01 |
observability.livenessWorkload |
the object this SD renders (Deployment named after the service; VirtualMachineInstance for kind: vm) |
R-OBS-01 |
| backup of irreplaceable state | off-cluster (never reacquirable) |
R-STATE-03 |
Field → resource mapping
name
- Renders: the service Namespace (trust zone), Deployment/VM name, Service name, and the DNS label
<name>.home.keithjackson.dev. - Requirements: R-DNS-01 (naming model, docs/09), R-NET (namespace = isolation boundary, docs/03).
kind + source
container+source.image→ Deployment (+ Service) using the image ref.vm+source.vmTemplate→ KubeVirt VirtualMachine with a CDI-imported root disk.- The schema enforces exactly-one-of (
imagefor container,vmTemplatefor vm). - Requirements: R-PLAT-02/03 (PaaS/IaaS, docs/13); VM disks inherit Longhorn durability (docs/04).
exposure (internal | public)
-
Both classes render an HTTPRoute on the shared Gateway and are reachable on the LAN via internal split-horizon DNS.
-
publicadditionally signals the documented manual edge port-forward of 80/443 (and any service-specific port, e.g. Plex 32400, Minecraft 25565) to the ingress VIP, and a public ExternalDNS record for<name>.home.keithjackson.dev. -
internalrenders no public DNS record and is never port-forwarded (PiHole resolves it to the internal Gateway VIP). -
HOW "no public DNS record" IS MADE TRUE, since it was prose only until 2026-09-14. Every HTTPRoute rendered for an
internalservice carries ExternalDNS's own controller annotation with a value other than the one ExternalDNS claims:external-dns.alpha.kubernetes.io/controller: noneExternalDNS v0.21.0 (
source/annotations/processors.go) skips any source object whosecontrollerannotation is present and!= "dns-controller", andsource/gateway.goapplies that check insideEndpoints()before any hostname is resolved — so the route never becomes an endpoint for any name it declares.source/service.goapplies the identical check, which covers the second configured source. It is derived fromexposureand nothing else (catalog/render.py_externaldns_route_annotations), so the operator's one declaration decides, a service that OMITSexposureinherits theinternaldefault and is likewise unpublishable, and there is no per-service list to forget to extend.ci/render-golden.shprovesapps/reproduces fromcatalog/, so aninternalservice cannot exist in the applied tree without it, andconformance/tests/l0-static/internal-services-publish-no-public-dnsre-proves both directions (present on internal, absent on public) on every gate run.- The route itself, the wildcard TLS and LAN reachability are unchanged — only the public
record is withdrawn. The internal split-horizon answer is Pi-hole's job (R-DNS-03), which is why
withdrawing these records requires Pi-hole to be answering the LAN authoritatively first: the
zone has no
*.home.keithjackson.devwildcard, and*.keithjackson.devdoes not match two labels, so a withdrawn name is a public NXDOMAIN with no fallback. - Why not a cluster-wide ExternalDNS opt-in filter (
labelFilter, which v0.21.0 supports for both configured sources): seven published names come from hand-authoredplatform//apps/manifests that carry noexposurefield for a renderer to read, and an opt-in filter deletes all seven the moment it is set. Per-resource exclusion has exactly the blast radius of the defect. The filter stays available as hardening once those seven are adjudicated — seeplatform/externaldns/helmrelease.yamland the register in the L0 check.
- The route itself, the wildcard TLS and LAN reachability are unchanged — only the public
record is withdrawn. The internal split-horizon answer is Pi-hole's job (R-DNS-03), which is why
withdrawing these records requires Pi-hole to be answering the LAN authoritatively first: the
zone has no
-
publicrenders the record but the renderer does not yet set a target: the record still resolves to the shared Gateway VIP, i.e. an RFC1918 address, so apublicservice is not in fact publicly reachable. Fixing that is not a renderer change on v0.21.0 —source/gateway.goreadsexternal-dns.alpha.kubernetes.io/targetonly from the Gateway (and a ListenerSet), never from an HTTPRoute, so a per-service target override is not expressible while the public services share thesharedGateway. The open decision is recorded inplatform/externaldns/helmrelease.yaml. -
Exposure does not relax east-west default-deny (docs/09 §"Interaction with default-deny").
-
Requirements: R-DNS-02/03 (docs/09), R-NET-04 (docs/03).
egress (internet | vpn) — egress steering
internet→ pod egress allow to the default WAN route (node → SOHO edge → internet).vpn→ the pod is steered through the shared in-cluster Gluetun/PIA (WireGuard) egress gateway (Cilium Egress Gateway / routing). The pod gets no independent egress path; the kill-switch (fail-closed) and allow rules are enforced at the gateway + namespace/policy level, outside the workload.- Requirements: R-NET-05 (egress class), R-NET-06 (VPN kill-switch, mandatory & central). VPN-class set is normative: sonarr, radarr, jackett, qbittorrent (docs/03/13).
dependencies[] (to / port / protocol) — east-west policy
- Each namespace first gets the default-deny baseline (deny all ingress + egress) applied by
GitOps at namespace creation, with a standing allow only for DNS to the in-cluster resolver.
Observability scraping is NOT in the baseline — it is an additive allow rendered per declared
observability.metricsEndpoint(see below), scoped to that one port. This paragraph used to claim the baseline carried a standing monitoring allow; it never did, and an app metrics port would have been dropped by policy even with the Service port and monitor endpoint in agreement. - Each
dependencies[]entry renders one CiliumNetworkPolicy egress allow: this service →toservice onport/protocol. Everything undeclared stays denied. - Empty
dependencies⇒ the service can reach only DNS. - Requirements: R-NET-01 (default-deny baseline), R-NET-02 (explicit allow / deny undeclared). Auditable via Hubble (R-SEC-03, R-OBS-04).
storage[] (name / size / class / mountPath / export / subPath / readOnly) — PVs + PVCs
class: longhorn→ a PVC on the 3-replica Longhorn StorageClass sized bysize(replicated config/DB/app state),ReadWriteOnce, dynamically provisioned. For avm, the disk is a Longhorn volume via CDI.class: bulk→ a static, pre-bound external NFS PersistentVolume and its PVC. The entry'sexportis the absolute path on the bulk server; the SERVER is never named in a ServiceDefinition — it is the${BULK_NFS_SERVER}cluster-vars knob, so one authored definition promotes across environments that reach the same storage by different addresses (R-ENV-02). Not 3× replicated: durability is the storage host's ZFS pool, not Longhorn.storageClassName: ""on both PV and PVC (explicitly empty ⇒ statically provisioned; no provisioner is ever asked to create anything, and the default-StorageClass annotation is suppressed). Deliberately NOT thebulkStorageClass: a StorageClassprovisioneris immutable, so re-pointing that stub would make Flux fail permanently on it.- Binding is pinned from both ends —
claimRefon the PV,volumeNameon the PVC — so no other claim of matching size/mode can win a race for the volume. persistentVolumeReclaimPolicy: Retain, always. Bulk data pre-exists the cluster, is far larger than the cluster, and may have consumers outside it; nothing here may reclaim it.- Access mode follows the read/write posture and is always
…Many, because one export is consumed by several namespaces and several nodes at once:ReadOnlyManywhenreadOnly,ReadWriteManyotherwise.ReadWriteOncecannot express either. mountOptionsare a platform decision held in the renderer, not a per-service field:nfsvers=4.1,hard,noatime,timeo=600,retrans=2. NFSv4 carries mount and lock in-protocol, so the only port that must be reachable is tcp/2049 — no rpcbind/111, mountd or statd, which is what keeps the firewall surface to a single rule.hardis chosen oversoftbecause a soft mount converts a server hiccup into a short read handed to the application.sizeon a bulk entry is the PV's advertised capacity and nothing more — no component enforces it (the ZFS pool is the real limit), so it should state the dataset's actual size sokubectl get pvreports something true.- fsGroup does not apply. The in-tree NFS volume plugin performs no ownership management, so
podSecurityContext.fsGroupis a no-op on a bulk volume. A bulk mount that must be written therefore requires the container's own uid/gid to be one the export's ownership permits (e.g. Plex'sPLEX_UID/PLEX_GID, LinuxServer images'PUID/PGID) — or a server-side squash. A mount that succeeds but cannot write is the characteristic failure of this tier.
mountPath(any class) overrides the/data/<name>default verbatim.subPath(any class) mounts a subtree instead of the volume root. TwosubPathmounts of one export are two separate mounts, solink()between them failsEXDEV; a workload that needs hardlinks or atomic moves between two directories (Sonarr/Radarr importing from a download path into a library) must mount their common ancestor once instead.readOnly(any class) marks the volumeMount read-only, and on a bulk entry also sets the PV'snfs.readOnlyso the kernel mount really isro. Note this is a client-side guarantee: one export serving both a reader and a writer that may land on any node cannot distinguish them server-side.- State separation rule: anything that must survive a rebuild lives on a named volume here, never on ephemeral fs/hostPath — so the workload can be destroyed and re-applied from git with data intact.
- Requirements: R-STATE-01 (rebuild survival), R-STATE-02 (Longhorn replication), R-STATE-04 (bulk tier). (docs/04)
hardware (gpu) — host device capability
-
The field is INTENT, never MECHANISM.
hardware.gpuis an enum of capability classes (today:intel-quicksync), not a device path, not a hostPath, not a resource request. The ServiceDefinition says what class of host device this workload needs; how the platform hands it over iscatalog/render.py's business and lives nowhere else. Do not "simplify" this into a raw/dev/...path. The entire payoff is that migrating from the hostPath mechanism below to the Intel GPU device plugin (resources.limits: gpu.intel.com/i915) is a renderer change with zero catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely different capability class (e.g.nvidia-nvenc); never to name a second route to the same one. -
gpu: intel-quicksync→ on the app container only (never a sidecar), exactly two things:- a hostPath volume
dri→/dev/dri,type: DirectoryOrCreate; - a read-write volumeMount of
driat/dev/drion the app container (VA-API opens the render node read-write, so it is deliberately notreadOnly).
Mounting the device is sufficient. That is the whole mapping.
- a hostPath volume
-
podSecurityContext.supplementalGroupsis deliberately NOT rendered, and adding it would be a regression rather than a hardening./dev/dri/renderD128isroot:render0660, so the obvious move is to push the hostrenderGID in — and it does not work. Measured in the running container on prod-2, 2026-09-30: the image's own/etc/cont-init.d/45-plex-hw-transcode-and-connected-tunerhookfinds every char device under/dev/dri,stat -c '%g'each one's GID,groupadds a matching group inside the container when none exists, andusermod -a -Gthe app user into it — every start, nothing hardcoded.cont-init.dcompletes beforeservices.d, so the account database is already correct when the service execs. The service then starts viaexec s6-setuidgid plex …, ands6-setuidgidreplaces the process group list from/etc/grouprather than appending to it:Plex Media Serverruns asUid: 1000 Gid: 1000 Groups: 100 1000, with root's gid0— which PID 1 (s6-svscan) does hold — gone. A KubernetessupplementalGroupstherefore reachess6-svscanand is discarded before the app ever starts. It would be inert and it would mislead whoever debugs this next. The renderer asserts its own absence inrender.py --selftest.- A corollary worth having: because the hook re-
stats the device on every start, the host GID drifting (a node rebuild renumbering the systemd dynamicrendergroup) is self-healing. No GID is pinned anywhere in this repo. - Scope of that measurement, stated so nobody over-reads it. "Mounting the device is sufficient" was established for an image that resolves the device group itself. A FUTURE consumer that runs non-root and has no equivalent init hook would still need the group solved somehow — and the answer for that case is a renderer change (per-image handling, or the device plugin), not a new ServiceDefinition field. The enum is what keeps that option open. The first such consumer must re-measure rather than assume this bullet covers it.
- A corollary worth having: because the hook re-
-
DirectoryOrCreateis load-bearing — do not tighten it toDirectory. A node that comes back without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the workload to software transcoding, not fail the mount.apps-wave-0iswait: trueandapps-wave-1dependsOnit, so a failed hostPath mount stalls the whole app tier for the sake of a feature that is not load-bearing./devis tmpfs, so the empty directory the kubelet creates on such a node leaves nothing durable behind.- The cost of that choice, stated plainly: a missing device is silent. The workload
software-transcodes and nothing reports it. There is no metric to alert on — media apps here
render no
metricsEndpoint, so no ServiceMonitor exists — and an alert that cannot fire is worse than none (see §observability's 2026-09-13 correction). The only proof of hardware transcode is an operator observing it in the application's own dashboard.
- The cost of that choice, stated plainly: a missing device is silent. The workload
software-transcodes and nothing reports it. There is no metric to alert on — media apps here
render no
-
Deliberately NOT rendered from this field: no
privileged: true(Kyvernorestrict-host-namespacesis Enforce on app namespaces and would reject it, and the render node does not need it); nonodeSelector(placement is a separate concern with its own justification — this field is about the device, not about where the pod lands); no resource requests/limits (there is no app-container resource render path, and inventing one here would be scope creep). -
Absent ⇒ nothing at all is rendered; every service that does not opt in stays byte-identical. Ignored for
kind: vm— a VM receives devices through KubeVirt's owndomain.devices. -
Requirements: R-PLAT-02 (the PaaS provides the host capability; the app declares need).
ingressRoutes[] (host / port) — HTTPRoute + Certificate + DNS
- Each entry renders an HTTPRoute attaching
host→ the serviceporton the shared Gateway. - TLS: the Gateway terminates with the cert-manager wildcard
*.home.keithjackson.devcertificate (Let's Encrypt via DNS-01, scoped Cloudflare token), so no per-service cert step — trusted, non-self-signed, auto-renewed, including forinternalservices. Per-serviceCertificateonly where isolation warrants. - DNS: ExternalDNS watches the HTTPRoute and manages the
<host>record — but only for apublicservice. Aninternalservice's route is rendered with ExternalDNS's controller annotation set tonone, which removes it from ExternalDNS's view entirely (see §exposure), so its<host>exists only in the internal split-horizon answer PiHole serves, never in the public zone. Apublicrecord's TARGET is still the Gateway VIP, not a routable address — open operator decision, see §exposure. - The
hostpattern is enforced to<label>.home.keithjackson.devby the schema. - Requirements: R-TLS-01/02/03 (docs/08), R-DNS-01/02/03 (docs/09).
inClusterClients[] (namespace / port / protocol / name) — direct east-west clients
- Each entry renders ONE CiliumNetworkPolicy ingress allow on this service:
endpointSelector= this service,fromEndpointsmatching only the reservedk8s:io.kubernetes.pod.namespace: <namespace>label,toPortscarrying only the declaredport/protocol. Name:<service>-allow-<name>-ingress,namedefaulting tonamespace. - Why the field exists.
ingressRoutesopens the north-south path only (the gateway allow admitsfromEntities: [ingress], the Gateway proxy identity, and nothing else), anddependenciescompiles edges only between catalog services — both halves come from a catalog entry's own declaration. A platform controller that dials this service's ClusterIP directly was therefore unauthorable, and under default-deny it is dropped. Prod hit exactly that: itsGitRepositorypoints at the in-cluster Gitea Service (http://gitea.gitea.svc.cluster.local:3000/..., clusters/prod/kustomization.yaml) so GitOps depends on no public DNS/Cloudflare/Gateway/${BASE_DOMAIN}, butflux-system's source-controller was denied — and the HTTPRoute that would have served a public name lives in a downstream tier that cannot apply while the source is down (source down → routes never rename → source stays down). - Least privilege is part of the contract: a namespace and a port, never
fromEntities: all, never a port-less allow. The client is selected by namespace rather than pod labels because an upstream controller's labels are its chart's to change, not this platform's contract — so name a platform-controller namespace whose entire membership may hold that reach, never a shared one. - Additive, and not a dependency: the namespace default-deny baseline is unchanged
(
ingress: []), every undeclared source/port stays denied, and the entry creates no wave ordering edge — the client is a platform component, not a catalog service. - Requirements: R-NET-01 (default-deny baseline stands), R-NET-02 (explicit allow / deny undeclared), R-GITOPS-02 (the allowance is authored in the one reviewable object rather than surviving as a live-applied cluster object). Auditable via Hubble (R-SEC-03, R-OBS-04).
auth (mode / groups[] / apiRoutes[]) — identity integration (Keycloak / oauth2-proxy)
oidc→ register the app as a Keycloak OIDC client (declarative realm/client config, GitOps; client secret SOPS-encrypted). App does its own login redirect.forward-auth→ protect the service'singressRoutesat the Gateway with oauth2-proxy → Keycloak so auth happens before traffic reaches the app (bare UIs without usable OIDC).native→ no platform SSO wiring; app owns its auth (documented exception, e.g. Plex/Seerr; and Keycloak itself as the IdP, admin login local + MFA).none→ no auth wiring (internal-only, discouraged).groups[]→ the Keycloak groups permitted access, enforced by the OIDC client mappers / oauth2-proxyallowed_groups. Schema requires ≥1 group foroidc/forward-auth— access is "in an authorized group," not merely "logged in."apiRoutes[](forward-authonly) → one repeatable--api-route=<regex>per entry on the per-app oauth2-proxy sidecar, making matching paths answer 401 instead of a 302 to Keycloak. Default[]renders no flag at all, which is the common case. It exists because oauth2-proxy's built-in AJAX guard fires only on an exactAccept: application/json, which these apps' JavaScript does not consistently send: the XHR is 302'd, the browser follows it invisibly, the live SSO session satisfies it silently, and the call resolves 200 carrying the app's HTML where the JavaScript expected JSON — and a POST following a 302 is re-issued as a GET with its body dropped. It is per-service and must never become a renderer constant: the path shape belongs to the app (radarr/signalr/, jackett/api/, pihole none), and a wrong constant does not merely fail to help — a path a browser navigates to, answered 401, shows a logged-out operator a bare error page and never offers a sign-in. Each entry must be anchored at^/and continue with literal path characters;^/,^/.*,.*, an unanchored/api/and any alternation are rejected by the schema and by the renderer (alternation is refused rather than supported because the flag is repeatable).- Keycloak, oauth2-proxy, and app→Keycloak / proxy→Keycloak / Keycloak→DB flows are ordinary
workloads under default-deny with explicit allows (ties back to
dependencies). - Requirements: R-IDP-02 (protect OIDC-capable and bare UIs), R-IDP-06 (group-based authorization). (docs/15, ADR-0014)
update (channel / range / cadence / autoRollbackOnFailedHealth / pin)
- container
kind→ a Flux ImagePolicy (+ ImageRepository) encodingchannel/range; Flux image-automation advances the running version on its scan interval (cadence).pinfreezes the service (stops automation). No human/AI edits a tag. vmkind→ in-guest unattended package updates oncadence(cloud-init), per docs/11 §3.autoRollbackOnFailedHealth: true→ health-gated rollout that reverts on failed health and alerts.- Requirements: R-UPD-01 (stable only), R-UPD-02 (no human/AI authoring), R-UPD-03 (health-gated auto-rollback). (docs/11, ADR-0009)
observability (metrics / slis[] / metricsEndpoint / livenessWorkload) — ServiceMonitor + alerts
metrics: true→ the platform collects signals for this service: logs to Loki via the node agent, and kube-state-metrics readiness for theupSLI below. Both are platform-wide scrapes; neither needs a per-service CR. It does not assert that the workload serves a Prometheus endpoint. (Corrected 2026-09-13. It used to render a ServiceMonitor unconditionally, hardcoded to a Service port namedmetrics. No rendered Service ever defined such a port, so all twelve app monitors discovered ZERO targets — verified on prod: 24 jobs inup, not one of the twelve — and everyup{job=...}SLI built on them was structurally inert.metrics: truecould not have been right: Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud serve no Prometheus endpoint at all without a separate exporter.)metricsEndpoint: {port, name?, path?, interval?}→ the workload does serve metrics, here. Renders all three halves of a working scrape from the one declaration, because any one of them missing makes the scrape silently yield nothing:- a named Service port (
name, defaultmetrics) on thatport; - a ServiceMonitor endpoint selecting that port name, on
pathatinterval; - a CiliumNetworkPolicy admitting the
monitoringnamespace to that port — the namespace default-deny baseline isingress: [], so without it the scrape is dropped by policy. The metrics port is deliberately not added to the Gateway ingress allow: it is east-west for Prometheus, never north-south.
- a named Service port (
- each entry in
slis[]→ at least one alert rule routed by Alertmanager to email; a per-service Grafana dashboard row.up→ a<name>-downcritical alert reading the readiness oflivenessWorkloadfrom kube-state-metrics (kube_deployment_status_replicas_available,kube_statefulset_status_replicas_ready,kube_daemonset_status_number_available) or, for aVirtualMachineInstance, the existence ofkubevirt_vmi_info{phase="running"}. Each rule carries anor absent(...)half, so the signal DISAPPEARING fires the alert rather than silencing it. livenessWorkload: {kind, name}→ which object's readiness is this service. Defaults to the object this SD renders; override it when the running workload is a different one (keycloak's server is a StatefulSet owned by the Keycloak operator — a Deployment-shaped rule wouldabsent()and fire forever).- Requirements: R-OBS-01 (per-service SLIs/alerts). Platform-health alerts (R-OBS-04) are rendered platform-wide, not per ServiceDefinition. (docs/10)
backup (policy)
- a schedule ref (e.g.
daily-offcluster) → Velero + Longhorn off-cluster backup schedule for this service's volumes/namespace. - literal
reacquirable→ no 1:1 backup (bulk media only). Irreplaceable state (configs, DBs, Nextcloud, Gitea, PiHole config, Keycloak DB) MUST NOT bereacquirable. - Requirements: R-STATE-03 (off-cluster DR backup). (docs/04)
Cross-field invariants the renderer / L0-L1 tests rely on
- kind ↔ source: container⇒
image, vm⇒vmTemplate(schema-enforced). - auth ↔ groups:
oidc/forward-auth⇒ non-emptygroups(schema-enforced; R-IDP-06). 2a. auth ↔ apiRoutes: a non-emptyauth.apiRoutes⇒auth.mode: forward-auth(schema-enforced). It renders onto the per-app oauth2-proxy sidecar, and no sidecar exists foroidc/native/none, so declaring it elsewhere would be a silent no-op — the operator believes the app's XHRs now answer 401 and nothing changed. Every entry must be anchored at^/(schema-enforced and enforced incatalog/render.py, so a breadth mistake is caught byrender-goldeneven on a run where the schema stage did not execute). - default-deny is unconditional: every namespace gets the deny baseline regardless of
dependencies;dependenciesonly adds allows (R-NET-01/02). The same holds forinClusterClients: it adds one namespace+port ingress allow and never relaxes the baseline. - egress is enforced outside the workload:
vpnrenders gateway steering + kill-switch, never a per-service sidecar the app could bypass (R-NET-05/06). - TLS is never self-signed and needs no per-service step: the wildcard Gateway cert covers every
ingressRoutes.host,internalincluded (R-TLS-01/03). - irreplaceable ⇒ off-cluster backup: DR-critical services must not carry
backup.policy: reacquirable(R-STATE-03). (Advisory: not enforced by the schema — an L0 policy check should assert it; see "Ambiguities / escalations" in schemas/README.md.) exposure: internal⇒ no public DNS record, structurally. Every HTTPRoute rendered for aninternalservice (including one that omitsexposureand takes the default) carriesexternal-dns.alpha.kubernetes.io/controller: none, and nopublicservice's route carries it. Derived fromexposurealone; enforced on every gate run byconformance/tests/l0-static/internal-services-publish-no-public-dns(R-DNS-03).hardwareis inert until opted into, and its mechanism is renderer-local. An SD with nohardwarekey renders no volume and no volumeMount; an SD with it gains nosecurityContexteither. Both directions are proven byrender.py --selftest. The/dev/drihostPath and theDirectoryOrCreatetype appear incatalog/render.pyand nowhere else, so the capability's implementation can be replaced without touching a single ServiceDefinition.
Requirement traceability (contract-level)
Primary R-GITOPS-02 (single reviewable object wires every requirement). Field-level: R-NET-01/02 (dependencies, inClusterClients), R-NET-05/06 (egress), R-STATE-01/02/04 (storage), R-STATE-03 (backup), R-TLS-01/02/03 + R-DNS-01/02/03 (ingressRoutes + exposure), R-OBS-01 (observability), R-UPD-01/02/03 (update), R-IDP-02/06 (auth), R-PLAT-02/03 (kind/source).