feat(schema): declare a host hardware capability, hardware.gpu: intel-quicksync
A ServiceDefinition could not say it needs the host's Intel QuickSync render
device. The schema is `additionalProperties: false`, so there was no authorable
form for it at all, and `apps/` is generated — a hand-edit would be reverted by
the next render.
Adds optional top-level `hardware`, with one optional member `gpu`, an enum
whose only current value is `intel-quicksync` (schema 1.7.0 -> 1.8.0).
The enum is the point. The SD declares WHAT CLASS of device it needs; the
hostPath and the mount live only in catalog/render.py. Migrating to the Intel
GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) is then
a renderer change with zero catalog edits. Letting an SD name `/dev/dri`
directly would look simpler and make that migration a catalog-wide edit, so
both the schema description and the renderer contract say so explicitly.
When `hardware.gpu == intel-quicksync`, r_deployment emits exactly two things,
on the app container only:
- a hostPath volume `dri` -> /dev/dri, type DirectoryOrCreate;
- a read-write volumeMount of it at /dev/dri (VA-API opens the render node
read-write, so deliberately not readOnly).
Mounting the device is sufficient. In particular NO
podSecurityContext.supplementalGroups is emitted, and adding one would be a
regression rather than a hardening. Measured in the running container on
prod-2, 2026-09-30:
PID 1 s6-svscan Uid: 0 Gid: 0 Groups: 0
pid 246 Plex Media Server Uid: 1000 Gid: 1000 Groups: 100 1000
The service is started with `exec s6-setuidgid plex ...`, and the child's group
list is exactly the /etc/group account-database list — root's gid 0, which
PID 1 holds, is gone. That is a setgroups() that REPLACES rather than appends,
so anything Kubernetes supplies is discarded before the app starts. What works
is the image's own /etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner,
which stats each /dev/dri char device, groupadds a matching group inside the
container and usermods the app user into it, every start, nothing hardcoded.
An inert supplementalGroups would have read as the mechanism to whoever
debugged this next, so the absence is documented at the emission site and
asserted by --selftest. Corollary: the hook re-stats on every start, so host
GID drift is self-healing and no GID is pinned anywhere in this repo.
`DirectoryOrCreate` is load-bearing. A node that comes back without the render
device (rebuild, BIOS change, driver regression, hardware swap) must degrade
the workload to software transcoding, not fail the mount: apps-wave-0 is
`wait: true` and apps-wave-1 dependsOn it, so a failed hostPath mount stalls
the whole app tier for a feature that is not load-bearing. /dev is tmpfs, so
the empty directory the kubelet creates leaves nothing durable behind. Its cost
— a missing device is silent, and no metric exists to alert on — is recorded in
the contract rather than papered over with an alert that could not fire.
Also deliberately NOT emitted: no `privileged: true` (Kyverno restrict-host-
namespaces is Enforce and it is not needed), no nodeSelector (placement is a
separate concern with its own justification), no resource requests (there is no
app-container resource render path).
No catalog entry uses the field. Re-rendering `--all catalog -o apps` is a zero
diff, which is the proof the change is inert until opted into.
Self-tests in the existing check(...) style: the positive (volume + mount, and
no securityContext / privileged / nodeSelector), the negative (an SD without
`hardware` gains no volume, no mount, no securityContext — mirroring the
podSecurityContext negative) and a pass-through case (an SD's own
podSecurityContext survives byte-for-byte). Note that NO gate stage runs
`render.py --selftest`; they were run by hand and the gap is recorded in the
handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
08deb208a1
commit
fa4765bb3e
5 files changed
+391
-2
No files matched your search
+4
-1
@@ -15,7 +15,7 @@ docs/13 exactly. Do not diverge from it here — escalate spec problems, don't f
|
||||
| `servicedefinition.schema.json` | Formal JSON Schema (draft 2020-12) for a ServiceDefinition. Enforces enums, required fields, patterns, and cross-field invariants. |
|
||||
| `renderer-contract.md` | Authoritative field → rendered-resource mapping, with requirement IDs. The contract other workstreams consume. |
|
||||
|
||||
**Schema version:** 1.0.0 (in the schema `description`). Treat any change as an interface change:
|
||||
**Schema version:** 1.8.0 (in the schema `description`, which carries the full per-version changelog — this line had said `1.0.0` since 1.1.0 and is now synced). Treat any change as an interface change:
|
||||
propose on the task board, update dependent renderers/tests in the same change, never silently break a
|
||||
consumer (coordination-protocol §2).
|
||||
|
||||
@@ -35,6 +35,9 @@ consumer (coordination-protocol §2).
|
||||
- `ingressRoutes[].host` must match `<label>.home.keithjackson.dev`.
|
||||
- `storage[].class ∈ {longhorn, bulk}`; `exposure ∈ {internal, public}`; `egress ∈ {internet, vpn}`;
|
||||
`auth.mode ∈ {oidc, forward-auth, native, none}`.
|
||||
- `hardware.gpu ∈ {intel-quicksync}` — a host **capability class**, never a device path. The enum
|
||||
is the contract: the mechanism that satisfies it lives only in `catalog/render.py` (see
|
||||
`renderer-contract.md` §`hardware`).
|
||||
- `additionalProperties: false` everywhere — unknown keys are rejected (catches typos in review/CI).
|
||||
|
||||
## Validate examples against the schema — L1, no cluster (R-TEST-02)
|
||||
|
||||
@@ -160,6 +160,64 @@ platform must render, and cites the requirement each mapping satisfies. Changing
|
||||
- **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication),
|
||||
**R-STATE-04** (bulk tier). (docs/04)
|
||||
|
||||
### `hardware` (`gpu`) — host device capability
|
||||
- **The field is INTENT, never MECHANISM.** `hardware.gpu` is an **enum of capability classes**
|
||||
(today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
|
||||
ServiceDefinition says *what class of host device this workload needs*; how the platform hands it
|
||||
over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw
|
||||
`/dev/...` path.** The entire payoff is that migrating from the hostPath mechanism below to the
|
||||
Intel GPU device plugin (`resources.limits: gpu.intel.com/i915`) is a renderer change with **zero**
|
||||
catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely
|
||||
different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one.
|
||||
- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), exactly two things:
|
||||
1. a **hostPath volume** `dri` → `/dev/dri`, **`type: DirectoryOrCreate`**;
|
||||
2. a **read-write volumeMount** of `dri` at `/dev/dri` on the app container (VA-API opens the
|
||||
render node read-write, so it is deliberately not `readOnly`).
|
||||
|
||||
**Mounting the device is sufficient. That is the whole mapping.**
|
||||
- **`podSecurityContext.supplementalGroups` is deliberately NOT rendered, and adding it would be a
|
||||
regression rather than a hardening.** `/dev/dri/renderD128` is `root:render` `0660`, so the obvious
|
||||
move is to push the host `render` GID in — and it does not work. **Measured in the running
|
||||
container on prod-2, 2026-09-30:** the image's own
|
||||
`/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` hook `find`s every char device under
|
||||
`/dev/dri`, `stat -c '%g'` each one's GID, `groupadd`s a matching group *inside the container* when
|
||||
none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded.
|
||||
`cont-init.d` completes before `services.d`, so the account database is already correct when the
|
||||
service execs. The service then starts via `exec s6-setuidgid plex …`, and `s6-setuidgid`
|
||||
**replaces** the process group list from `/etc/group` rather than appending to it:
|
||||
`Plex Media Server` runs as `Uid: 1000 Gid: 1000 Groups: 100 1000`, with root's gid `0` — which
|
||||
PID 1 (`s6-svscan`) does hold — **gone**. A Kubernetes `supplementalGroups` therefore reaches
|
||||
`s6-svscan` and is discarded before the app ever starts. It would be inert *and* it would mislead
|
||||
whoever debugs this next. The renderer asserts its own absence in `render.py --selftest`.
|
||||
- A corollary worth having: because the hook re-`stat`s the device on every start, the host GID
|
||||
**drifting** (a node rebuild renumbering the *systemd dynamic* `render` group) is self-healing.
|
||||
No GID is pinned anywhere in this repo.
|
||||
- **Scope of that measurement, stated so nobody over-reads it.** "Mounting the device is
|
||||
sufficient" was established for an image that resolves the device group itself. A FUTURE
|
||||
consumer that runs non-root and has no equivalent init hook would still need the group solved
|
||||
somehow — and the answer for that case is a **renderer** change (per-image handling, or the
|
||||
device plugin), not a new ServiceDefinition field. The enum is what keeps that option open. The
|
||||
first such consumer must re-measure rather than assume this bullet covers it.
|
||||
- **`DirectoryOrCreate` is load-bearing — do not tighten it to `Directory`.** A node that comes back
|
||||
without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the
|
||||
workload to **software transcoding**, not fail the mount. `apps-wave-0` is `wait: true` and
|
||||
`apps-wave-1` `dependsOn` it, so a failed hostPath mount stalls the **whole app tier** for the sake
|
||||
of a feature that is not load-bearing. `/dev` is tmpfs, so the empty directory the kubelet creates
|
||||
on such a node leaves nothing durable behind.
|
||||
- **The cost of that choice, stated plainly:** a missing device is **silent**. The workload
|
||||
software-transcodes and nothing reports it. There is no metric to alert on — media apps here
|
||||
render no `metricsEndpoint`, so no ServiceMonitor exists — and an alert that cannot fire is worse
|
||||
than none (see §`observability`'s 2026-09-13 correction). The only proof of hardware transcode is
|
||||
an operator observing it in the application's own dashboard.
|
||||
- **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno
|
||||
`restrict-host-namespaces` is Enforce on app namespaces and would reject it, and the render node
|
||||
does not need it); no `nodeSelector` (placement is a separate concern with its own justification —
|
||||
this field is about the device, not about where the pod lands); no resource requests/limits (there
|
||||
is no app-container resource render path, and inventing one here would be scope creep).
|
||||
- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical.
|
||||
Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`.
|
||||
- **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need).
|
||||
|
||||
### `ingressRoutes[]` (`host` / `port`) — HTTPRoute + Certificate + DNS
|
||||
- Each entry renders an **HTTPRoute** attaching `host` → the service `port` on the shared Gateway.
|
||||
- TLS: the Gateway terminates with the cert-manager **wildcard `*.home.keithjackson.dev`** certificate
|
||||
@@ -286,6 +344,11 @@ platform must render, and cites the requirement each mapping satisfies. Changing
|
||||
`external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it.
|
||||
Derived from `exposure` alone; enforced on every gate run by
|
||||
`conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03).
|
||||
8. **`hardware` is inert until opted into, and its mechanism is renderer-local.** An SD with no
|
||||
`hardware` key renders no volume and no volumeMount; an SD *with* it gains no `securityContext`
|
||||
either. Both directions are proven by `render.py --selftest`. The `/dev/dri` hostPath and the
|
||||
`DirectoryOrCreate` type appear in `catalog/render.py` and nowhere else, so the capability's
|
||||
implementation can be replaced without touching a single ServiceDefinition.
|
||||
|
||||
## Requirement traceability (contract-level)
|
||||
Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
|
||||
"title": "ServiceDefinition",
|
||||
"description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 / R-IDP-02).",
|
||||
"description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 Line truncated
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["name", "source"],
|
||||
@@ -139,6 +139,19 @@
|
||||
"supplementalGroups": { "type": "array", "items": { "type": "integer" }, "description": "Additional GIDs added to the first process in each container." }
|
||||
}
|
||||
},
|
||||
"hardware": {
|
||||
"type": "object",
|
||||
"description": "Optional host HARDWARE CAPABILITY the workload needs, for the container kind (docs/13). This field declares INTENT (`what device class this service needs`), never MECHANISM (`which host path to bind-mount`, `which extended resource to request`). That separation is the whole point of the enum: the platform currently satisfies `intel-quicksync` with a /dev/dri hostPath volume and a matching mount on the app container, but a later migration to the Intel GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) must be a catalog/render.py change and NOTHING else - no ServiceDefinition may need editing, and no reviewer may have to re-derive per-service device wiring. Do NOT 'simplify' this into a raw device path, a hostPath, or a resource request; that would push a platform implementation detail into the operator-facing contract and make the migration a catalog-wide edit. Absent => no device wiring at all (existing apps render byte-identical). Ignored for kind=vm (a VM gets devices through KubeVirt's own domain.devices).",
|
||||
"additionalProperties": false,
|
||||
"minProperties": 1,
|
||||
"properties": {
|
||||
"gpu": {
|
||||
"type": "string",
|
||||
"description": "GPU capability class this service needs. `intel-quicksync` = the host's Intel Quick Sync Video render device (the iGPU's /dev/dri/renderD* node) for hardware video transcode/decode. Enum, not a free-form device path, so the satisfying mechanism stays swappable (see the parent description). Extend the enum when a genuinely different capability class appears (e.g. nvidia-nvenc), never to name a different path to the same one.",
|
||||
"enum": ["intel-quicksync"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"exposure": {
|
||||
"type": "string",
|
||||
"description": "internal = LAN-only (internal DNS + Gateway on LAN); public = additionally edge port-forwarded 80/443 (docs/09; R-DNS/R-NET).",
|
||||
|
||||
Reference in new issue
Block a user