Compare commits

...
Author SHA1 Message Date
jakman85andClaude Opus 5 df1fd1b3d1 chore(decisions): record the rebase onto main @ 08deb20
gate / gate (pull_request) Successful in 1m43s
gate / conformance (pull_request) Has been skipped
The runbooks/README.md conflict was two index entries appended at the
same point, not a semantic one. Both kept, in the file's grouped order.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-01 21:06:34 -05:00
jakman85andClaude Opus 5 0fee584e3a feat(plex): opt into hardware.gpu: intel-quicksync, and write the runbook for the step GitOps cannot reach
Plex has been transcoding in software on hardware that can do it in the iGPU. This
opts its ServiceDefinition into the `hardware` capability from #27 and re-renders.

catalog/plex.yaml gains four lines of YAML and the comment that explains them:

    hardware:
      gpu: intel-quicksync

which renders exactly two things into apps/wave-0/plex.yaml and nothing else — a
`dri` hostPath volume (/dev/dri, type: DirectoryOrCreate) and a read-write /dev/dri
volumeMount on the app container. Whole-tree render diff: 6 added lines, one file.

The comment records what is not obvious from the four lines: the device exists on
ALL THREE prod nodes with i915 bound (so no nodeSelector is rendered and none is
wanted); why `supplementalGroups` would be INERT rather than a hardening (the
image's cont-init hook adds the plex user to the device's group inside the
container, and s6-setuidgid then REPLACES the process group list from /etc/group,
discarding anything Kubernetes supplied); and why `DirectoryOrCreate` is
load-bearing (apps-wave-0 is wait:true with apps-wave-1 depending on it, so a
failed mount would stall the whole app tier for a feature that is not).

THE FEATURE IS INERT UNTIL A HUMAN CLICKS A BOX, and that is the part this commit
is really about. Mounting the device does not enable hardware transcoding; Plex's
`HardwareAcceleratedCodecs` preference does, and it is ABSENT from Preferences.xml
— the toggle has never been on. It cannot be automated: a bounded grep across the
running image's entire init surface (/etc/cont-init.d, /etc/services.d,
/etc/fix-attrs.d, /plex-common.sh, /usr/local/bin) found ZERO PLEX_PREFERENCE*
support, and the six setPref keys the image does write are all unrelated. So
runbooks/plex-quicksync.md is the procedure, registered in the configMapGenerator
(a runbook in git with no generator entry is reviewed, merged, and still a 404) and
indexed in runbooks/README.md. It covers the GitOps/manual boundary, the toggle and
the real preference keys (including the widely-cited one that does not exist and
the upstream misspelling nobody should "fix"), verification via `Transcode (hw)` on
the Plex dashboard — stated plainly as the ONLY proof, because Plex renders no
ServiceMonitor and an alert could never fire — the device-cgroup failure to expect
first with its Intel-device-plugin fallback, and the floating-`:latest` caveat.

Stacked on #27 (feat/sd-hardware-capability), which is open and unmerged; that
branch's commit is in this range by design and this must merge after it.

NOT VERIFIED, and the handoff says so at length: hardware transcoding has not been
observed working. Whether containerd's device cgroup lets a non-privileged pod
open() a bind-mounted char device is an open question that only deploying settles,
and the Plex-side toggle was still unset at the time of writing. A green gate
proves the manifest renders; it proves nothing about transcoding.

Gate (pinned toolbox, worktree): rc 3 — UNDETERMINED, not green. Passed:
decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped (the known
local ceiling): spec-immutability (no sibling spec repo), gitea-settings (no
SETTINGS_API_TOKEN), conformance (CLUSTER_AVAILABLE unset, so L2/L3 did not run).
No fourth skip, nothing failed.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-01 21:05:17 -05:00
jakman85andClaude Opus 5 f716e9d167 chore(decisions): record the rebase onto main @ 08deb20
gate / gate (pull_request) Successful in 1m41s
gate / conformance (pull_request) Has been skipped
The render.py conflict was textual adjacency with PR #28's constants
block, not a semantic one. Both sides kept.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-01 21:01:16 -05:00
jakman85andClaude Opus 5 fa4765bb3e feat(schema): declare a host hardware capability, hardware.gpu: intel-quicksync
A ServiceDefinition could not say it needs the host's Intel QuickSync render
device. The schema is `additionalProperties: false`, so there was no authorable
form for it at all, and `apps/` is generated — a hand-edit would be reverted by
the next render.

Adds optional top-level `hardware`, with one optional member `gpu`, an enum
whose only current value is `intel-quicksync` (schema 1.7.0 -> 1.8.0).

The enum is the point. The SD declares WHAT CLASS of device it needs; the
hostPath and the mount live only in catalog/render.py. Migrating to the Intel
GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) is then
a renderer change with zero catalog edits. Letting an SD name `/dev/dri`
directly would look simpler and make that migration a catalog-wide edit, so
both the schema description and the renderer contract say so explicitly.

When `hardware.gpu == intel-quicksync`, r_deployment emits exactly two things,
on the app container only:

  - a hostPath volume `dri` -> /dev/dri, type DirectoryOrCreate;
  - a read-write volumeMount of it at /dev/dri (VA-API opens the render node
    read-write, so deliberately not readOnly).

Mounting the device is sufficient. In particular NO
podSecurityContext.supplementalGroups is emitted, and adding one would be a
regression rather than a hardening. Measured in the running container on
prod-2, 2026-09-30:

  PID 1   s6-svscan            Uid: 0     Gid: 0     Groups: 0
  pid 246 Plex Media Server    Uid: 1000  Gid: 1000  Groups: 100 1000

The service is started with `exec s6-setuidgid plex ...`, and the child's group
list is exactly the /etc/group account-database list — root's gid 0, which
PID 1 holds, is gone. That is a setgroups() that REPLACES rather than appends,
so anything Kubernetes supplies is discarded before the app starts. What works
is the image's own /etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner,
which stats each /dev/dri char device, groupadds a matching group inside the
container and usermods the app user into it, every start, nothing hardcoded.
An inert supplementalGroups would have read as the mechanism to whoever
debugged this next, so the absence is documented at the emission site and
asserted by --selftest. Corollary: the hook re-stats on every start, so host
GID drift is self-healing and no GID is pinned anywhere in this repo.

`DirectoryOrCreate` is load-bearing. A node that comes back without the render
device (rebuild, BIOS change, driver regression, hardware swap) must degrade
the workload to software transcoding, not fail the mount: apps-wave-0 is
`wait: true` and apps-wave-1 dependsOn it, so a failed hostPath mount stalls
the whole app tier for a feature that is not load-bearing. /dev is tmpfs, so
the empty directory the kubelet creates leaves nothing durable behind. Its cost
— a missing device is silent, and no metric exists to alert on — is recorded in
the contract rather than papered over with an alert that could not fire.

Also deliberately NOT emitted: no `privileged: true` (Kyverno restrict-host-
namespaces is Enforce and it is not needed), no nodeSelector (placement is a
separate concern with its own justification), no resource requests (there is no
app-container resource render path).

No catalog entry uses the field. Re-rendering `--all catalog -o apps` is a zero
diff, which is the proof the change is inert until opted into.

Self-tests in the existing check(...) style: the positive (volume + mount, and
no securityContext / privileged / nodeSelector), the negative (an SD without
`hardware` gains no volume, no mount, no securityContext — mirroring the
podSecurityContext negative) and a pass-through case (an SD's own
podSecurityContext survives byte-for-byte). Note that NO gate stage runs
`render.py --selftest`; they were run by hand and the gap is recorded in the
handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-01 20:54:35 -05:00
11 changed files with 940 additions and 2 deletions

No files matched your search

+6
View File
@@ -143,6 +143,8 @@ spec:
- name: media
mountPath: /media
readOnly: true
- name: dri
mountPath: /dev/dri
volumes:
- name: config
persistentVolumeClaim:
@@ -150,6 +152,10 @@ spec:
- name: media
persistentVolumeClaim:
claimName: plex-media
- name: dri
hostPath:
path: /dev/dri
type: DirectoryOrCreate
---
# Service
apiVersion: v1
+52
View File
@@ -80,6 +80,58 @@ env:
# remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
# clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
# reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
# HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
# INTENT ONLY. This says "this workload needs an Intel Quick Sync render device"; it does NOT name a
# device path. The renderer currently satisfies it with exactly two things — a `/dev/dri` hostPath
# volume (`type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container —
# and nothing else: no nodeSelector, no privileged, no resources (schemas/renderer-contract.md
# `### hardware`). If that mechanism is later replaced by the Intel GPU device plugin
# (`gpu.intel.com/i915`), it is a `catalog/render.py` change and THIS BLOCK DOES NOT MOVE. Do not
# "simplify" it into a raw `/dev/...` path; that is the whole point of the enum.
#
# THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
# anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
# GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Plex happens to run on k3s-prod-2 today, but
# that is scheduler history rather than a requirement — this field deliberately renders no
# nodeSelector, because placement is a separate concern with its own justification.
#
# NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
# The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
# the host `render` GID in as a supplemental group. It cannot work here. MEASURED IN THE RUNNING
# CONTAINER, 2026-09-30: the image's own
# `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device under
# `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
# `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
# GID renumbering self-healing (no GID is pinned anywhere in this repo). The service then execs via
# `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from `/etc/group` rather
# than appending to it: `Plex Media Server` runs with `Groups: 100 1000`, and root's gid `0` — which
# PID 1 (`s6-svscan`) does hold — is GONE. Anything Kubernetes supplies therefore reaches s6-svscan
# and is discarded before Plex ever starts. Mounting the device is sufficient; the group is the
# image's job and it does it correctly. `render.py --selftest` asserts the ABSENCE of a pod
# securityContext for this field precisely so nobody adds an inert one back later.
#
# `DirectoryOrCreate` IS THE SAFE FAILURE MODE, not laziness. A node that comes back without the
# device (rebuild, BIOS reset, driver regression, hardware swap) gets an empty directory and Plex
# degrades to SOFTWARE transcoding. `Directory` would fail the mount instead — and because
# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that would stall the WHOLE app
# tier for a feature that is not load-bearing. The price, stated plainly: a missing device is
# SILENT. This SD renders no `metricsEndpoint`, so there is no ServiceMonitor, no transcode-mode
# metric, and no alert that could ever fire. The only proof is an operator reading the Plex
# dashboard.
#
# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Mounting the device does
# not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
# is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
# set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
# `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin`) found ZERO `PLEX_PREFERENCE*` support,
# and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
# `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
# unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took and the
# device-cgroup failure mode to expect first.
hardware:
gpu: intel-quicksync
exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
egress: internet # normal WAN egress (docs/03; R-NET-05)
storage:
+126
View File
@@ -163,6 +163,15 @@ OAUTH2_COOKIE_REFRESH = "2m30s"
# are added. Do not read this flag as "metrics are working".
OAUTH2_PROXY_METRICS_PORT = 44180
# --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
# The ServiceDefinition declares the CAPABILITY (schemas/servicedefinition.schema.json
# `hardware.gpu`); everything below is the MECHANISM that currently satisfies it and is
# deliberately confined to this file. Swapping to the Intel GPU device plugin
# (`gpu.intel.com/i915` resource requests instead of a hostPath) is a change here and
# nowhere else -- no catalog entry moves.
DRI_VOLUME_NAME = "dri"
DRI_DEVICE_DIR = "/dev/dri"
# =============================================================================
# Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping.
@@ -513,6 +522,46 @@ def r_deployment(s: dict) -> dict:
"persistentVolumeClaim": {"claimName": f"{s['name']}-{vol['name']}"},
}
)
# hardware capability (schema-optional): `hardware.gpu: intel-quicksync` means this
# workload needs the host iGPU's render node for hardware video transcode. The SD says
# only that; the wiring below is the platform's current mechanism for satisfying it.
#
# `type: DirectoryOrCreate` is DELIBERATE and load-bearing -- do NOT "tighten" it to
# `Directory`. A node that comes back without the render device (rebuild, BIOS change,
# driver regression, hardware swap) must degrade the workload to SOFTWARE transcoding,
# not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` dependsOn it, so a
# failed hostPath mount stalls the entire app tier for the sake of a feature that is not
# load-bearing. `DirectoryOrCreate` turns that failure mode into "transcodes in
# software" instead of "the app tier does not reconcile". /dev is tmpfs, so the empty
# directory the kubelet creates on such a node leaves nothing durable behind.
#
# The mount is deliberately NOT readOnly: VA-API opens the render node read-write.
#
# `podSecurityContext.supplementalGroups` is DELIBERATELY NOT SET here, and adding it
# would be a regression, not a hardening. MEASURED in the running container on prod-2,
# 2026-09-30: the image's own cont-init hook
# `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` `find`s every char device
# under /dev/dri, `stat -c '%g'` each one's GID, `groupadd`s a matching group INSIDE the
# container when none exists, and `usermod -a -G` the app user into it -- every start,
# nothing hardcoded. cont-init.d completes before services.d, so the account database
# s6 reads at exec time already carries the group. The service is then started with
# `exec s6-setuidgid plex ...`, and `s6-setuidgid` REPLACES the process group list from
# /etc/group rather than appending to it: `Plex Media Server` runs with
# `Uid: 1000 Gid: 1000 Groups: 100 1000` -- root's gid 0, which PID 1 (s6-svscan) does
# have, is GONE. Anything Kubernetes puts in supplementalGroups reaches s6-svscan and is
# then discarded before the app ever starts. MOUNTING THE DEVICE IS SUFFICIENT; the
# group handling is the image's job and it does it correctly. A supplementalGroups entry
# here would be inert AND would mislead whoever debugs this next.
hw_gpu = (s.get("hardware") or {}).get("gpu")
if hw_gpu == "intel-quicksync":
volume_mounts.append({"name": DRI_VOLUME_NAME, "mountPath": DRI_DEVICE_DIR})
volumes.append(
{
"name": DRI_VOLUME_NAME,
"hostPath": {"path": DRI_DEVICE_DIR, "type": "DirectoryOrCreate"},
}
)
if volume_mounts:
container["volumeMounts"] = volume_mounts
@@ -2530,6 +2579,83 @@ def _selftest() -> int:
"an SD without podSecurityContext must not gain a pod securityContext",
)
# 6b) hardware.gpu: intel-quicksync -> a /dev/dri hostPath volume and a read-write mount
# on the app container, and NOTHING ELSE. The SD declares the CAPABILITY; these
# assertions pin the MECHANISM that currently satisfies it -- including what it
# deliberately does NOT emit -- so a change to it is a deliberate, visible edit.
hw_docs = render(
{
"name": "hw",
"source": {"image": "a:1"},
"hardware": {"gpu": "intel-quicksync"},
}
)
hw_pod = next(
d[1] for d in hw_docs if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]
check(
{
"name": "dri",
"hostPath": {"path": "/dev/dri", "type": "DirectoryOrCreate"},
}
in hw_pod.get("volumes", []),
"hardware.gpu=intel-quicksync must render a /dev/dri hostPath volume "
"with type DirectoryOrCreate (a GPU-less node must degrade, not fail the mount)",
)
hw_mounts = hw_pod["containers"][0].get("volumeMounts", [])
check(
{"name": "dri", "mountPath": "/dev/dri"} in hw_mounts,
"hardware.gpu=intel-quicksync must mount /dev/dri READ-WRITE on the app container",
)
# The device mount is the WHOLE feature: no pod securityContext is invented. The image's
# own cont-init hook adds the app user to the device's group inside the container, and
# s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway (see the
# emission site). Asserting the absence is what stops someone adding an inert one back.
check(
"securityContext" not in hw_pod,
"hardware.gpu must NOT invent a pod securityContext (supplementalGroups is inert "
"under s6-setuidgid; the image's cont-init hook handles the device group)",
)
# Nothing privileged, and no node pinning: Kyverno restrict-host-namespaces (Enforce)
# rejects `privileged: true` in an app namespace, and placement is a separate concern
# with its own justification -- this field is about the DEVICE only.
check(
"privileged" not in hw_pod.get("securityContext", {})
and "securityContext" not in hw_pod["containers"][0]
and "nodeSelector" not in hw_pod,
"hardware.gpu must emit no privileged securityContext and no nodeSelector",
)
# The regression that matters: absence renders NOTHING. Mirrors the podSecurityContext
# negative above -- an SD that does not opt in must be byte-identical to before.
nohw_pod = next(
d[1] for d in render({"name": "nohw", "source": {"image": "a:1"}})
if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]
check(
"volumes" not in nohw_pod
and "volumeMounts" not in nohw_pod["containers"][0]
and "securityContext" not in nohw_pod,
"an SD without `hardware` must gain no dri volume, no mount and no securityContext",
)
# An SD that declares BOTH keeps its own podSecurityContext exactly as written -- the
# hardware field neither adds to it nor rewrites it.
hwm_pod = next(
d[1]
for d in render(
{
"name": "hwm",
"source": {"image": "a:1"},
"hardware": {"gpu": "intel-quicksync"},
"podSecurityContext": {"fsGroup": 1001, "supplementalGroups": [44]},
}
)
if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]
check(
hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]},
"hardware.gpu must leave the SD's own podSecurityContext untouched",
)
# 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that
# fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged.
fa_docs = render(
+174
View File
@@ -0,0 +1,174 @@
---
date: 2026-09-30
branch: feat/plex-quicksync
title: Plex opts into `hardware.gpu: intel-quicksync` — the device mount is GitOps, the Plex-side toggle is a documented manual step, and hardware transcoding has NOT been observed working
---
## What changed
Three things, and the third is the point of the branch.
1. **`catalog/plex.yaml`** — added the four lines the whole feature reduces to:
```yaml
hardware:
gpu: intel-quicksync
```
placed between `env` and `exposure` to match the schema's own property order, with ~48 lines of
comment in the file's established style. The comment records: the field is **intent, not
mechanism** (a later move to the Intel device plugin is a `render.py` change and this block does
not move); that `/dev/dri/renderD128` is present on **all three** prod nodes with `i915` bound;
**why a `supplementalGroups` entry would be inert** (the image's cont-init hook does the group
work inside the container and `s6-setuidgid` replaces the process group list from `/etc/group`);
why `DirectoryOrCreate` is the load-bearing safe failure mode (`apps-wave-0` is `wait: true` with
`apps-wave-1` depending on it, so a failed mount would stall the whole app tier for a feature
that is not load-bearing); and that the Plex-side toggle is a **separate manual operator step**,
pointing at the new runbook.
2. **`apps/wave-0/plex.yaml`** — re-rendered. Six added lines, nothing else, in the whole `apps/`
tree:
```
- name: dri
mountPath: /dev/dri
- name: dri
hostPath:
path: /dev/dri
type: DirectoryOrCreate
```
3. **`runbooks/plex-quicksync.md`** (new), registered in `runbooks/kustomization.yaml`'s
`configMapGenerator` as `- plex-quicksync=plex-quicksync.md` and indexed in `runbooks/README.md`
under "Operator procedures". **The file alone would have been a 404** — the pod serves a
ConfigMap, not the git tree, and `conformance/tests/l0-static/alert-runbooks-resolvable` exists
precisely because "the file is in git" and "the URL resolves" are different claims. The page
covers: the GitOps/manual boundary as a table; the toggle (Settings → Transcoder → "Use hardware
acceleration when available", preference key `HardwareAcceleratedCodecs`); the real neighbouring
keys and the widely-cited one that **does not exist**; that the only proof is `Transcode (hw)` on
the Plex dashboard **and why no alert is possible**; the device-cgroup failure to expect first
with a concrete open() diagnostic and the Intel-device-plugin fallback; and the floating-`:latest`
caveat with a digest check.
**Stacked on PR #27 (`feat/sd-hardware-capability`), which is open and unmerged.** This branch was
cut from it rather than from `main`, so its one commit (`d9baa2a`, the schema + renderer + contract)
is in this PR's range. That is deliberate: merging #27 is not an agent's action (CLAUDE.md §1), and
duplicating its changes here would have been worse than showing them. **This must merge after #27.**
Not done, deliberately: no `--emit-waves` (it writes into `clusters/base/`, a sensitive path that
would force a human reviewer, and it is a no-op unless the *set* of files under `apps/` changes — it
does not here). No `podSecurityContext`. No `nodeSelector`. No alert rule. No `kubectl apply`.
### Two follow-ups left unclaimed, with the reasoning
- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
`TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
transcode scratch performance changes silently on reschedule, with nothing reporting it. Both
candidate fixes need a **schema** change and so neither belongs in this PR: pin Plex to prod-2
(needs a `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory`
emptyDir and transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not
have). **The RAM option is the better one** — it removes the node asymmetry outright instead of
working around it — which is exactly why it deserves its own PR and its own review rather than
being smuggled in behind a four-line catalog change. Documented in the runbook's "Known gaps".
- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
quality only; decode and this feature are unaffected. Changing it needs a kernel parameter plus a
**serialized rolling reboot** — an operator action under CLAUDE.md §5, on a cluster with a
documented history of that going badly (a prod node hung on a warm reboot on 2026-09-03). Kept
well clear of this PR.
### Rebase note (2026-10-01, Ruth Okonkwo) — resolution only, no redesign
Rebased onto `main` @ `08deb20` (was cut from `0fbf544`), restacked on the rebased
`feat/sd-hardware-capability` (PR #27) so the stack stays coherent — this branch still carries
exactly PR #27's commit plus its own, with no duplicate of anything already in `main`. The branch
ref was force-updated with `--force-with-lease`, so PR #32 is the same PR with a new head.
**The `catalog/render.py` conflict disappeared** once PR #27 was rebased; it was never this
branch's own conflict. **One conflict remained, in `runbooks/README.md`, purely textual.** PR #30
(`fix/jackett-oauth2-proxy-session`, merged as `08deb20`) appended an `oauth2-proxy-session` index
entry under *Operator procedures* at the same point this branch appended its `plex-quicksync` entry.
**Both entries were kept, in full.** Ordering follows this file's existing convention, which is
grouped/narrative rather than alphabetical (the *Nodes* section runs down/io/memory/disk/cpu):
`plex-quicksync` sits next to `plex-claim` as the other Plex page, leaving `oauth2-proxy-session`
adjacent to `crafty-admin-credential`, which cross-references the same sidecar gate. The change to
`runbooks/README.md` is an 8-line pure insertion, so PR #30's entry is untouched.
`runbooks/kustomization.yaml` auto-merged and **both** `configMapGenerator` entries are present.
Re-verified after the resolution, on the rebased stack: `--selftest` PASS (rc 0); a full re-render
reproduces every committed golden byte-identically; and the **only** `apps/` diff against the new
`main` is `apps/wave-0/plex.yaml`, confined to the expected six lines (the `dri` mount and its
`DirectoryOrCreate` hostPath volume). No other service's golden moved — PR #28 added ~176 lines of
oauth2-proxy sidecar rendering, and this confirms by measurement rather than assumption that Plex
(`auth: native`, no sidecar) is unaffected by it.
## What I verified
- **The render is exactly the intended change, and nothing else moved.** After
`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` (exit 0,
silent): `git diff --stat apps/` = `apps/wave-0/plex.yaml | 6 ++++++`, and
`git status --porcelain apps/` = a single ` M apps/wave-0/plex.yaml`. Read line by line, the diff
is the `dri` volumeMount and the `dri` hostPath volume, and nothing more. **Specifically checked:**
the same invocation also rewrites `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, which
are git-tracked but excluded from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in
`ci/render-golden.sh` — a spurious diff there would have landed **ungated**. Neither file shows a
diff; whole-repo `git status --porcelain` lists only `apps/wave-0/plex.yaml` and `catalog/plex.yaml`
plus my new files.
- **The rendered output matches what the contract promises** — `schemas/renderer-contract.md`
`### hardware` says exactly two things (a `/dev/dri` hostPath with `type: DirectoryOrCreate` and a
read-write mount on the app container) and exactly two things appeared. No `securityContext`, no
`nodeSelector`, no resources in the diff.
- **The runbook is projected into the server, not merely committed.** `runbooks/kustomization.yaml`
carries `- plex-quicksync=plex-quicksync.md`, whose shape matches the generator-entry regex the
L0 check parses; the extensionless key is what makes `/runbooks/plex-quicksync` a plain static
lookup. Also indexed in `runbooks/README.md`.
- **The runbook clears the L0 quality floor** that `alert-runbooks-resolvable` applies to
alert-linked pages, even though nothing links to this one (it is advisory for an unreferenced
page): all four required sections (`## What this means here`, `## First commands`,
`## Likely causes, most likely first`, `## When to escalate or stop`), well over the 1500-byte
floor, `dev.sh` used throughout, and `--context k3s-prod` on every command.
- **The gate.** See the run recorded in the PR description for the per-stage table and the real rc.
- **No secret was printed.** Every `Preferences.xml` read in the runbook greps one named key, and the
page states explicitly why: the file holds `PlexOnlineToken`. Nothing was `sops -d`'d.
## What I did NOT verify
**Hardware transcoding has NOT been observed working.** Nothing in this branch is evidence that it
does. A green gate here proves the manifest renders and the golden matches — it proves nothing about
runtime behaviour, and `vars.CLUSTER_AVAILABLE` is unset so L2/L3 conformance did not run at all.
- **The device-cgroup question is OPEN, and it is the largest remaining technical risk.** Whether
containerd on this k3s build permits a **non-privileged** pod to `open()` a bind-mounted character
device is **UNVERIFIED**. It can only be settled by deploying and trying. The container inits as
uid 0, which settles *file* permissions, but uid 0 does **not** bypass the cgroup device gate — and
the two failures look identical from the outside. If it is refused, the fallback is the Intel
Device Plugin (`gpu.intel.com/i915`), which is a `catalog/render.py` change plus a re-render with
**no ServiceDefinition edit**; `privileged: true` is not available (Kyverno
`restrict-host-namespaces` is Enforce). The runbook has the diagnostic sequence.
- **The Plex-side toggle was still unset at the time of writing.** `HardwareAcceleratedCodecs` was
**absent** from `Preferences.xml` — the toggle has never been enabled on this server. Until an
operator ticks it in the UI, this branch changes the pod spec and **nothing about transcoding**.
That is the whole reason the runbook exists.
- **No metric, so no alert, so no automated detection — permanently.** `catalog/plex.yaml` declares
no `metricsEndpoint`, so Plex renders no ServiceMonitor; there is no transcode-mode signal for a
rule to evaluate. A silent degradation to software transcoding (a node losing its iGPU, the
`DirectoryOrCreate` path) will **not** be reported by anything. I deliberately did not add an
alert: with no metric behind it, it could never fire, and this repo has been burned at scale by
exactly that. Closing the gap honestly means an exporter first.
- **I did not re-derive this session's node or in-container measurements.** The three-node
`renderD128`/`i915`/GID-992 facts, the cont-init hook behaviour, the `s6-setuidgid` group
replacement, the `PLEX_PREFERENCE*` absence, the six `setPref` keys and the
`HardwareAcceleratedCodecs` string were established earlier in this session and are cited, not
re-measured, by me. Per CLAUDE.md §2b, treat this file as a record of belief and re-measure against
the cluster before relying on any of it.
- **The image is a floating tag.** Everything known about the hook came from digest
`sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), while the Deployment runs
`plexinc/pms-docker:latest` with `imagePullPolicy: Always`. A silent pull can invalidate the
"no `supplementalGroups` needed" conclusion **with no commit in this repo**. Not guarded against —
the runbook tells the reader to re-check the hook after an image bump, which is documentation, not
enforcement.
- **I did not verify the live Gitea gate state** (branch protection, required contexts, runner
online) for this PR. Per §2b that must be queried, not read out of `CLAUDE.md`'s table; I did not
query it.
- **`/runbooks/plex-quicksync` was not fetched.** The L0 check asserts every static precondition for
an HTTP 200; it does not make the request, and I did not either.
@@ -0,0 +1,204 @@
---
date: 2026-09-30
branch: feat/sd-hardware-capability
title: ServiceDefinition gains a `hardware` capability field — `gpu: intel-quicksync` mounts /dev/dri and nothing else; the enum is what keeps the mechanism swappable
---
## What changed
A new optional top-level ServiceDefinition property, `hardware`, with one optional member,
`gpu`, whose enum today has exactly one value: `intel-quicksync`. **No catalog entry uses it.**
This PR delivers the capability; consuming it (Plex) is separate work on top.
- `schemas/servicedefinition.schema.json` — adds `hardware` (object, `additionalProperties: false`,
`minProperties: 1`) with `gpu` (enum `["intel-quicksync"]`). Schema version bumped **1.7.0 →
1.8.0**, changelog entry prepended to the schema `description` per that file's own convention.
- `catalog/render.py` — two module-level constants (`DRI_VOLUME_NAME`, `DRI_DEVICE_DIR`) and the
emission in `r_deployment`.
- `schemas/renderer-contract.md` — new `### hardware` field-mapping section, plus cross-field
invariant **8** in that file's numbered list.
- `schemas/README.md` — its `**Schema version:**` line said `1.0.0`; it had been stale since 1.1.0.
Synced to 1.8.0 and annotated that the schema `description` is the changelog of record. Added one
bullet to "Cross-field rules" for the new enum. That was the whole of the staleness; nothing
ballooned.
**What the renderer emits** for `hardware.gpu: intel-quicksync`, on the app container only (never a
sidecar), and it is exactly two things:
1. a hostPath volume `dri` → `/dev/dri`, `type: DirectoryOrCreate`;
2. a read-write `volumeMount` of `dri` at `/dev/dri` on the app container (deliberately not
`readOnly` — VA-API opens the render node read-write).
**Mounting the device is sufficient.** It emits **no** `privileged: true` (Kyverno
`restrict-host-namespaces` is Enforce on app namespaces and it is not needed), **no** `nodeSelector`
(placement is a separate concern with its own justification), **no** resource requests (there is no
app-container resource render path), and — the one that is worth spelling out — **no
`podSecurityContext.supplementalGroups`**.
**Why no `supplementalGroups`, when `/dev/dri/renderD128` is `root:render 0660`.** An earlier
revision of this branch merged the host `render` GID into `supplementalGroups`. **It was measured to
be a no-op and has been removed.** In the running container on prod-2, 2026-09-30:
PID 1 s6-svscan Uid: 0 Gid: 0 Groups: 0
pid 246 Plex Media Server Uid: 1000 Gid: 1000 Groups: 100 1000
`/etc/services.d/plex/run` ends in `exec s6-setuidgid plex …`, and the child's group list is exactly
the `/etc/group` account-database list — root's gid `0`, which PID 1 holds, has **vanished**. That is
a `setgroups()` that *replaces* rather than appends, so anything Kubernetes puts in
`supplementalGroups` reaches `s6-svscan` and is discarded before the app ever starts. What actually
works is the image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner`, which `find`s
every char device under `/dev/dri`, `stat -c '%g'`s its GID, `groupadd`s a matching group inside the
container when none exists, and `usermod -a -G`s the app user into it — every start, nothing
hardcoded; `cont-init.d` completes before `services.d`, so the database is correct at exec time.
Shipping an inert `supplementalGroups` would have been worse than shipping nothing: it would read as
the mechanism to whoever debugs this next. The reasoning lives in a comment **at the emission site**
and in `renderer-contract.md`, and `render.py --selftest` now asserts the *absence* of a
`securityContext`, so it cannot be "helpfully" added back without a test failing. A pleasant
corollary: because the hook re-`stat`s the device every start, host GID drift is **self-healing**,
and no GID is pinned anywhere in this repo.
**The design decision worth defending: the field is an enum of intent, not a device path.** The SD
says *what class of host device the workload needs*; the hostPath and the mount exist only in
`catalog/render.py`. Migrating to the Intel GPU device plugin (`gpu.intel.com/i915` resource
requests, no hostPath at all) is then a renderer change with zero catalog edits. Both the schema
`description` and the contract section say this explicitly, because the obvious "simplification" —
letting an SD name `/dev/dri` — would push a platform implementation detail into the operator-facing
contract and make that migration a catalog-wide edit.
**`DirectoryOrCreate` is load-bearing.** A node that comes back without the render device (rebuild,
BIOS change, driver regression, hardware swap) must degrade the workload to software transcoding,
not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so a failed
hostPath mount stalls the entire app tier for the sake of a feature that is not load-bearing.
`/dev` is tmpfs, so the empty directory the kubelet creates on such a node leaves nothing durable
behind. Do not "tighten" it to `Directory`. The reasoning is recorded at the emission site.
### Rebase note (2026-10-01, Ruth Okonkwo) — resolution only, no redesign
Rebased onto `main` @ `08deb20` (was cut from `0fbf544`). Re-landed as a single cherry-pick onto
current `main`; the branch ref was force-updated with `--force-with-lease`, so PR #27 is the same
PR with a new head.
**One conflict, in `catalog/render.py`, purely textual.** PR #28
(`fix/oauth2-proxy-cookie-collision-fleet`, merged as `6caa5f2`) inserted its own constants blocks
(`OAUTH2_ACCESS_TOKEN_LIFETIME_S` / `OAUTH2_COOKIE_REFRESH`, and `OAUTH2_PROXY_METRICS_PORT`) at the
same insertion point immediately after `KEYCLOAK_ISSUER` that this branch used for
`DRI_VOLUME_NAME` / `DRI_DEVICE_DIR`. The two blocks are unrelated and do not interact, so **both
sides were kept in full**, oauth2-proxy's first and the hardware block after it. Nothing was
dropped and nothing was rewritten: the rebased commit adds the **same 126 `render.py` lines** as the
pre-rebase commit (verified by comparing the added-line sets) and deletes none. In particular the
`supplementalGroups`-is-deliberately-not-emitted and `DirectoryOrCreate`-is-load-bearing comments
survive verbatim.
Re-verified after the resolution: `--selftest` PASS (rc 0), and a full re-render still produces a
**zero** diff in `apps/` — this PR remains inert, as it must be while no ServiceDefinition opts in.
## What I verified
Measured, in this worktree, on this branch:
- **`python3 catalog/render.py --selftest` → `render.py selftest: PASS`, rc 0.** New cases in the
existing `check(...)` style: the positive (volume + mount, and *no* `securityContext`, *no*
privileged, *no* nodeSelector), the negative that matters (an SD without `hardware` gains no
volume, no volumeMount and no `securityContext` — mirroring the existing `podSecurityContext`
negative), and a pass-through case (an SD declaring both `hardware` and its own
`podSecurityContext: {fsGroup: 1001, supplementalGroups: [44]}` keeps it **exactly** as written).
- **The self-tests actually bite.** I mutated the guard (`if hw_gpu == "intel-quicksync":` → `if
True:`) in a scratch copy and re-ran: **rc 1**, failing on "an SD without `hardware` must gain no
dri volume, no mount and no securityContext". A test that cannot fail is not a test; this one can.
- **Re-render is a zero diff.**
`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` then
`git diff --stat apps/` → **empty**, and `git status --porcelain apps/` → **empty** (so the two
not-onboarded by-products, `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, were
rewritten byte-identically and needed no revert). The change is inert until a service opts in.
- **Schema validity and catalog conformance.** `Draft202012Validator.check_schema` passes; all
**12** `catalog/*.yaml` validate with **0** errors. On the new field specifically: `gpu:
intel-quicksync` accepted; `gpu: /dev/dri` rejected (enum); `hardware: {gpu: …, path: …}` rejected
(`additionalProperties: false`); `hardware: {}` rejected (`minProperties: 1`).
- **Hand-checked the emitted YAML** by rendering a throwaway SD carrying `hardware`, a Longhorn
volume and its own `podSecurityContext: {runAsUser: 1000, supplementalGroups: [44]}`. Result:
`volumeMounts` = config then `{name: dri, mountPath: /dev/dri}`; `volumes` = the PVC then
`{name: dri, hostPath: {path: /dev/dri, type: DirectoryOrCreate}}`; `securityContext` =
`{runAsUser: 1000, supplementalGroups: [44]}` — i.e. **unchanged**, nothing appended. That toy SD
was never written into `catalog/`.
- **GATE: `rc 3` — UNDETERMINED, NOT green.** Run in the pinned toolbox from this worktree with
both CLAUDE.md §3 workarounds (`DEV_KUBECONFIG=/dev/null`, and an `ENGINE` shim bind-mounting the
main checkout's `.git` at its own absolute path). Diff base `origin/main @ 0fbf544d5e2f`; touched
requirements `R-DNS-03 R-GITOPS-02 R-IDP-02`. Run **four** times on this branch with an identical
result each time, most recently against tree `c132893045b1` — the final commit differs from that
run only by this paragraph. Per stage:
| stage | result |
|---|---|
| spec-immutability | **SKIPPED** — sibling spec repo not at `/work/../home-network-k3s`. Expected from a worktree; tolerated by §3. |
| decisions-log | pass |
| lint | pass |
| l0 | pass — 19 offline cases, `0 failure(s), 0 skip(s), 0 unwired` |
| render-golden | pass — *"all 10 onboarded service golden(s) reproduce from the canonical render"* |
| l1 | pass — 3 offline checks, `0 failure(s), 0 skip(s), 0 unwired`; coverage 51/51 requirements scaffolded |
| approval-policy | pass (`event='none'`, not a pull_request event — N/A locally; it will evaluate for real on the PR) |
| gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset. Expected; branch protection and runner registration are therefore UNVERIFIED by this run. |
| conformance (L2/L3) | **SKIPPED** — `CLUSTER_AVAILABLE!=true`. Phase-0 standing skip. |
The gate's own verdict: *"Stages that did not run: spec-immutability gitea-settings"*. **No stage
failed, and no stage skipped for a reason caused by this change.** rc 3 is reported as rc 3; I am
not calling this green.
- **Blast radius of the schema change:** `hardware` is absent from every catalog entry, so
`apply_defaults` never materializes it and `s.get("hardware")` is `None` on all 12 — which is what
the zero diff above demonstrates rather than asserts.
- **Touched no sensitive path.** Changes are confined to `schemas/`, `catalog/`, `decisions/`; per
`infra/gitea/approval-policy.json` none of those are sensitive. `conformance/**` and `clusters/**`
are untouched. I did not run `--emit-waves` (it writes `clusters/base/`, and the set of files
under `apps/` did not change).
## What I did NOT verify
- **Whether containerd's device cgroup permits a non-privileged pod to `open()` a bind-mounted char
device. Still unverified, and it is the single largest technical risk in this workstream.**
containerd installs a per-container **device cgroup** allow-list, and whether
`/dev/dri/renderD128` arriving as a bind-mounted device node is permitted by it on this
k3s/containerd version is **unmeasured**. Settling it requires applying a pod, which is an
operator action, not an agent one. The container's PID 1 running as uid 0 helps on the DAC side
but does **not** bypass the cgroup gate. If it blocks, the mount will succeed, the image's group
hook will do its job, and the `open()` will still fail — and the fallback is the Intel GPU device
plugin, which *sets* the cgroup entry. Note that that fallback is a **renderer-only change**: no
ServiceDefinition moves, because the authoring surface is an intent enum. That is the design
justifying itself, and it is the reason to keep the enum even if the hostPath mechanism loses.
- **That a missing device degrades silently — and nothing will tell you.** This is the accepted cost
of `DirectoryOrCreate`. The workload will software-transcode and **no signal reports it**: media
apps in this catalog declare no `metricsEndpoint`, so no ServiceMonitor exists, and there is no
metric to alert on. I deliberately did **not** propose an alert, because an alert that cannot fire
is worse than none — see the 2026-09-13 correction in `renderer-contract.md` §`observability`,
where twelve monitors discovered zero targets. The only proof of hardware transcode is an operator
seeing it in the application's own dashboard.
- **That "mounting the device is sufficient" generalizes beyond this one image.** It does not, and
I have not verified that it does. The claim was measured for an image that resolves the device
group itself in `cont-init.d`. A future consumer of `hardware.gpu` that runs non-root with no
equivalent hook would still need the group solved — and the fix for that case is a **renderer**
change (per-image handling, or the device plugin), never a new SD field. I scoped the claim in
`renderer-contract.md` rather than letting it read as general; the first such consumer must
re-measure.
- **The feature now depends on a hook inside an image that floats.** The `cont-init.d` behaviour
above was read from digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), but the
deployment runs `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. **A silent pull can
change the behaviour this feature depends on, with no commit anywhere in this repo to show for
it** — and the failure would be the silent software-transcode described above. I have not fixed
that here (it is the Plex wiring's tag/digest decision, not the capability's), but it should not
be forgotten.
- **GID drift is NOT on this list any more, deliberately.** An earlier revision pinned
`INTEL_RENDER_GID = 992` and listed "a node rebuild could move it" as an open risk. The hook
re-`stat`s the device on every start, so drift is self-healing and no GID is pinned anywhere. The
risk is retired, not overlooked.
- **Anything on the live cluster.** I ran no `kubectl` and I opened no shell in any container. The
node facts (device present, `root:render` 0660, `i915` bound, no device plugin, no
`gpu.intel.com/*` advertised), the in-container measurements quoted above, and the Kyverno
posture (`restrict-host-namespaces` Enforce; no hostPath or `supplementalGroups`
restriction; no PSA labels on app namespaces) are **prior recon handed to me, not re-measured by
me**. Per CLAUDE.md §2b, treat them as second-hand until re-queried.
- **That CI exercises the self-tests. It does not.** `git grep -- --selftest ci/` returns nothing —
no gate stage invokes the renderer's self-test. They are the executable spec and house style, and
I ran them by hand, but a future regression in this code path **will not be caught by the gate**.
Wiring `--selftest` into a gate stage is a real gap and is not part of this PR (`ci/` is a
sensitive path and would drag in a human reviewer for a change unrelated to this one).
- **Runtime behaviour of any kind.** The gate proves the L0/L1 offline subset only;
`vars.CLUSTER_AVAILABLE` is unset, so L2/L3 conformance does not run in CI.
+8
View File
@@ -115,6 +115,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
- **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod
(`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware
acceleration when available"**, and no environment variable can do it (the image supports no
`PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`,
read out of the binary), says which widely-cited key **does not exist**, and is explicit that the
**only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no
alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin
fallback, and why a `:latest` pull can silently break it.
- **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
(the four lines that matter, including the upstream `-` that means the proxy answered a request the
+6
View File
@@ -95,6 +95,12 @@ configMapGenerator:
# because the plex.tv claim window is short and is not a thing to go looking
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
- plex-claim=plex-claim.md
# Operator procedure, not an alert page, and NO alert is possible for it: plex renders no
# metricsEndpoint, so there is no ServiceMonitor, no transcode-mode metric and nothing a rule
# could evaluate. Served because the feature it completes is INERT until a human ticks a box in
# the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped
# feature stays switched off for months.
- plex-quicksync=plex-quicksync.md
- pvc-capacity=pvc-capacity.md
- targets-down=targets-down.md
- update-rollback=update-rollback.md
+283
View File
@@ -0,0 +1,283 @@
# plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did
**Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**,
not an incident page. It is indexed here because the feature it completes is **inert until a human
clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
to prevent.
**Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the
rendered golden — the `dri` hostPath volume and the `/dev/dri` mount), `catalog/render.py` (the
mechanism), `schemas/renderer-contract.md` `### hardware` (the contract).
**Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile.
## What this means here
Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync
moves that work off the CPU — which matters on a three-node cluster where the CPU is also running
everything else. Two independent things have to be true before that happens, and **only one of them
is in git**:
| Half | Who does it | State |
|---|---|---|
| The render device is inside the container at `/dev/dri` | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders a `dri` hostPath volume (`/dev/dri`, `type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container. Flux applies it. | Done by merging that change. |
| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
**There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`,
`/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin` — found **zero** `PLEX_PREFERENCE*` support.
Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentifier`,
`ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`,
`TranscoderTempDirectory`), none of them the transcode toggle. No environment variable, no ConfigMap
and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this;
there isn't one, and the search has already been done.
What the device mount does **not** do, so nobody reads more into a green gate than is there: it does
not enable hardware transcoding, it does not prove the device can be *opened* from inside a
non-privileged pod (see "Likely causes" #2), and it renders no `nodeSelector` — Plex can land on any
of the three nodes and all three carry the device.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation; the toolchain lives only in the pinned dev
container. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
picks a current-context **per invocation**, and that default has already changed once. Do not run two
`dev.sh` invocations at once.
**Never `cat` `Preferences.xml`.** It contains `PlexOnlineToken`, a plex.tv account credential. Every
read of that file below greps a single named key — do not widen one.
## First commands
**1. Is the device mounted into the pod at all?** (Proves Flux applied the change, not that it works.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-o jsonpath="{.items[0].spec.containers[0].volumeMounts[*].mountPath}"; echo'
```
`/dev/dri` in that list ⇒ the GitOps half landed. Absent ⇒ the running pod predates the merge; compare
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 \
-o jsonpath="{.status.lastAppliedRevision}"; echo'
```
against the merge commit before blaming anything else.
**2. Is the device actually visible inside the container?**
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'
```
Expect `renderD128` (usually `card0` too), mode `crw-rw----`, group `render`. An **empty directory**
here is the `DirectoryOrCreate` degradation: the kubelet created an empty `/dev/dri` because the node
had none. That is the designed-safe outcome — the app tier still reconciles — but it means software
transcoding, silently. Find which node the pod is on and ask why that node lost its iGPU:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
```
**3. Can Plex *open* it?** This is the question the manifest cannot answer — see "Likely causes" #2.
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- \
bash -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
```
`RW-OPEN-DENIED` (or `Operation not permitted` / `Permission denied`) with the device plainly visible
in step 2 is the **device-cgroup** failure, not a file-permission failure. The two look identical from
the outside and are not the same problem.
**4. Is the toggle on?** Named key only — never the whole file:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- \
grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
"/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'
```
`ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on,
and the remaining question is whether it *works* — go to "How to verify".
## The one manual step
Plex web UI → **Settings → Transcoder → "Use hardware acceleration when available"** → tick → Save.
(It sits under *Show Advanced*.) A transcode already in flight is unaffected; the next one picks it up.
The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by
extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from
recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise
them and not invent them:
| Key | What it is |
|---|---|
| `HardwareAcceleratedCodecs` | the toggle this page is about |
| `HardwareAcceleratedEncoders` | hardware **encode**, as distinct from decode |
| `HardwareDevicePath` | how Plex is pointed at a *specific* render node — relevant only if a host ever exposes more than one |
| `TranscoderToneMapping`, `TranscoderToneMappingAgorithm` | HDR→SDR tone mapping. **That misspelling of "Algorithm" is upstream's, in the binary. It is not a typo in this page — do not "fix" it** here or anywhere else; the corrected spelling is not a key Plex reads. |
| `TranscoderHEVCEncoding` | HEVC output |
**`HardwareAcceleratedCodecsEnableHEVC` does not exist.** It is widely cited and it is not in the
binary. Recorded explicitly so nobody spends an evening hunting for it or, worse, writes it into
`Preferences.xml` and concludes hardware transcoding is broken.
Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn
tier, it is in no git commit, and the UI writes the key correctly. Use the UI.
## How to verify — and why there is no alert
Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a
lower quality explicitly), then read Plex's own **Dashboard → Activity**:
- **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.**
- plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says.
**That dashboard is the *only* proof available, and nothing else will ever tell you.**
`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work
to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently
inert, and this repo has already been burned at scale by rules that evaluated against nothing while
looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that
gap is ever worth closing, the honest order is a real exporter first and a rule second.
Corroborating, not a substitute — the server log on the config PVC:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- \
sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
\"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'
```
Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions.
Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence
of hardware. The dashboard remains the verdict.
## Likely causes, most likely first
When the dashboard still says plain `Transcode`:
1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists.
First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
investigating anything technical.
2. **containerd's device cgroup is refusing the `open()` — UNVERIFIED, and the largest remaining
technical risk in this feature.** Whether a non-privileged pod may `open()` a bind-mounted
character device on this k3s/containerd build **has not been established**; it can only be settled
by deploying and trying. The container inits as uid 0, which settles *file* permissions, but
**uid 0 does not bypass the cgroup device gate.** Symptom: step 2 shows `renderD128` present and
correctly owned, step 3 returns `RW-OPEN-DENIED`.
**The documented fallback is the Intel Device Plugin for Kubernetes** (`gpu.intel.com/i915` as a
resource request, no hostPath). Because the ServiceDefinition field expresses *intent* rather than
a device path, that migration is a **`catalog/render.py` change plus a re-render — with no
ServiceDefinition edit anywhere**; `catalog/plex.yaml` keeps saying exactly what it says today.
**`privileged: true` is not an available escape hatch:** Kyverno `restrict-host-namespaces` is in
**Enforce** mode and rejects it in app namespaces. Do not reach for it.
3. **The pod landed on a node without the device.** Step 2 shows an empty `/dev/dri`. All three prod
nodes carried `renderD128` with `i915` bound when this was measured (2026-09-30), so this means a
node changed — a rebuild, a BIOS reset, or a driver regression. `DirectoryOrCreate` is why the app
tier still came up; the cost is that the loss is silent. **Fix the node, do not tighten the mount:**
`Directory` would stall `apps-wave-0` (`wait: true`, with `apps-wave-1` depending on it) for a
feature that is not load-bearing.
4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
mechanism this feature leans on with no commit in this repo.
5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so
`Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as
a hardware failure.
6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which
Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
sessions show `(hw)` and others do not. Not a fault.
## The floating-image caveat
Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
user to the device's group, the absence of `PLEX_PREFERENCE*`, the six `setPref` keys, the
`HardwareAcceleratedCodecs` string — was read from digest **`sha256:e0ab27395614…`**
(`plexmediaserver 1.43.4.10903`).
The deployment runs **`plexinc/pms-docker:latest` with `imagePullPolicy: Always`**. A silent pull can
therefore change the behaviour this feature depends on **with no commit in this repo**, while the
manifest still looks identical. Symptom to watch for: hardware transcoding that worked and quietly
stopped, with no git change to point at.
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-o jsonpath="{.items[0].status.containerStatuses[0].imageID}"; echo'
```
If that digest is not the one above, **re-verify the hook before trusting this page**:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- ls /etc/cont-init.d/'
```
`45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one
described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups`
needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that
hook exists.
## When to escalate or stop
- **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be
**inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's
`cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its
group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces**
the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with
`Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
- **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
reject it. If the cgroup gate is the blocker, the answer is the Intel device plugin (cause #2).
- **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles
`main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating
`kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change
through a branch, a PR and a green gate.
- **Do not edit `Preferences.xml` on the PVC**, never `cat` it, and never `sops -d` anything into a
terminal. It holds `PlexOnlineToken`.
- **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and
watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
lose the DB.
- **Stop and get the operator** if the device cannot be opened from the pod (cause #2 confirmed). The
next step is a platform change — the Intel device plugin — with its own PR, not more poking at this
workload.
## Known gaps, deliberately not fixed here
Recorded so they are findable, and so nobody assumes they were overlooked:
- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
`TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
transcode scratch performance changes silently whenever the pod is rescheduled. Both candidate fixes
need a **schema** change and are therefore out of scope here: pin Plex to prod-2 (needs a
`nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory` emptyDir and
transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not have). The RAM option
is the better one, because it removes the node asymmetry outright instead of working around it.
- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
quality only — decode, and the feature on this page, are unaffected. Changing it needs a kernel
parameter and a **serialized rolling reboot**, which is an operator action (CLAUDE.md §5), and this
cluster has a documented history of that going badly (a prod node hung on a warm reboot on
2026-09-03). Not worth coupling to this change.
---
Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
The node measurements (all three prod nodes) and the in-container image measurements cited here were
taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
(`plexmediaserver 1.43.4.10903`). Hardware transcoding had **not** been observed working at the time
of writing, and the Plex-side toggle was still unset.
+4 -1
View File
@@ -15,7 +15,7 @@ docs/13 exactly. Do not diverge from it here — escalate spec problems, don't f
| `servicedefinition.schema.json` | Formal JSON Schema (draft 2020-12) for a ServiceDefinition. Enforces enums, required fields, patterns, and cross-field invariants. |
| `renderer-contract.md` | Authoritative field → rendered-resource mapping, with requirement IDs. The contract other workstreams consume. |
**Schema version:** 1.0.0 (in the schema `description`). Treat any change as an interface change:
**Schema version:** 1.8.0 (in the schema `description`, which carries the full per-version changelog — this line had said `1.0.0` since 1.1.0 and is now synced). Treat any change as an interface change:
propose on the task board, update dependent renderers/tests in the same change, never silently break a
consumer (coordination-protocol §2).
@@ -35,6 +35,9 @@ consumer (coordination-protocol §2).
- `ingressRoutes[].host` must match `<label>.home.keithjackson.dev`.
- `storage[].class ∈ {longhorn, bulk}`; `exposure ∈ {internal, public}`; `egress ∈ {internet, vpn}`;
`auth.mode ∈ {oidc, forward-auth, native, none}`.
- `hardware.gpu ∈ {intel-quicksync}` — a host **capability class**, never a device path. The enum
is the contract: the mechanism that satisfies it lives only in `catalog/render.py` (see
`renderer-contract.md` §`hardware`).
- `additionalProperties: false` everywhere — unknown keys are rejected (catches typos in review/CI).
## Validate examples against the schema — L1, no cluster (R-TEST-02)
+63
View File
@@ -160,6 +160,64 @@ platform must render, and cites the requirement each mapping satisfies. Changing
- **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication),
**R-STATE-04** (bulk tier). (docs/04)
### `hardware` (`gpu`) — host device capability
- **The field is INTENT, never MECHANISM.** `hardware.gpu` is an **enum of capability classes**
(today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
ServiceDefinition says *what class of host device this workload needs*; how the platform hands it
over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw
`/dev/...` path.** The entire payoff is that migrating from the hostPath mechanism below to the
Intel GPU device plugin (`resources.limits: gpu.intel.com/i915`) is a renderer change with **zero**
catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely
different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one.
- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), exactly two things:
1. a **hostPath volume** `dri` → `/dev/dri`, **`type: DirectoryOrCreate`**;
2. a **read-write volumeMount** of `dri` at `/dev/dri` on the app container (VA-API opens the
render node read-write, so it is deliberately not `readOnly`).
**Mounting the device is sufficient. That is the whole mapping.**
- **`podSecurityContext.supplementalGroups` is deliberately NOT rendered, and adding it would be a
regression rather than a hardening.** `/dev/dri/renderD128` is `root:render` `0660`, so the obvious
move is to push the host `render` GID in — and it does not work. **Measured in the running
container on prod-2, 2026-09-30:** the image's own
`/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` hook `find`s every char device under
`/dev/dri`, `stat -c '%g'` each one's GID, `groupadd`s a matching group *inside the container* when
none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded.
`cont-init.d` completes before `services.d`, so the account database is already correct when the
service execs. The service then starts via `exec s6-setuidgid plex …`, and `s6-setuidgid`
**replaces** the process group list from `/etc/group` rather than appending to it:
`Plex Media Server` runs as `Uid: 1000 Gid: 1000 Groups: 100 1000`, with root's gid `0` — which
PID 1 (`s6-svscan`) does hold — **gone**. A Kubernetes `supplementalGroups` therefore reaches
`s6-svscan` and is discarded before the app ever starts. It would be inert *and* it would mislead
whoever debugs this next. The renderer asserts its own absence in `render.py --selftest`.
- A corollary worth having: because the hook re-`stat`s the device on every start, the host GID
**drifting** (a node rebuild renumbering the *systemd dynamic* `render` group) is self-healing.
No GID is pinned anywhere in this repo.
- **Scope of that measurement, stated so nobody over-reads it.** "Mounting the device is
sufficient" was established for an image that resolves the device group itself. A FUTURE
consumer that runs non-root and has no equivalent init hook would still need the group solved
somehow — and the answer for that case is a **renderer** change (per-image handling, or the
device plugin), not a new ServiceDefinition field. The enum is what keeps that option open. The
first such consumer must re-measure rather than assume this bullet covers it.
- **`DirectoryOrCreate` is load-bearing — do not tighten it to `Directory`.** A node that comes back
without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the
workload to **software transcoding**, not fail the mount. `apps-wave-0` is `wait: true` and
`apps-wave-1` `dependsOn` it, so a failed hostPath mount stalls the **whole app tier** for the sake
of a feature that is not load-bearing. `/dev` is tmpfs, so the empty directory the kubelet creates
on such a node leaves nothing durable behind.
- **The cost of that choice, stated plainly:** a missing device is **silent**. The workload
software-transcodes and nothing reports it. There is no metric to alert on — media apps here
render no `metricsEndpoint`, so no ServiceMonitor exists — and an alert that cannot fire is worse
than none (see §`observability`'s 2026-09-13 correction). The only proof of hardware transcode is
an operator observing it in the application's own dashboard.
- **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno
`restrict-host-namespaces` is Enforce on app namespaces and would reject it, and the render node
does not need it); no `nodeSelector` (placement is a separate concern with its own justification —
this field is about the device, not about where the pod lands); no resource requests/limits (there
is no app-container resource render path, and inventing one here would be scope creep).
- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical.
Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`.
- **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need).
### `ingressRoutes[]` (`host` / `port`) — HTTPRoute + Certificate + DNS
- Each entry renders an **HTTPRoute** attaching `host` → the service `port` on the shared Gateway.
- TLS: the Gateway terminates with the cert-manager **wildcard `*.home.keithjackson.dev`** certificate
@@ -286,6 +344,11 @@ platform must render, and cites the requirement each mapping satisfies. Changing
`external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it.
Derived from `exposure` alone; enforced on every gate run by
`conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03).
8. **`hardware` is inert until opted into, and its mechanism is renderer-local.** An SD with no
`hardware` key renders no volume and no volumeMount; an SD *with* it gains no `securityContext`
either. Both directions are proven by `render.py --selftest`. The `/dev/dri` hostPath and the
`DirectoryOrCreate` type appear in `catalog/render.py` and nowhere else, so the capability's
implementation can be replaced without touching a single ServiceDefinition.
## Requirement traceability (contract-level)
Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02
+14 -1
View File
@@ -2,7 +2,7 @@
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
"title": "ServiceDefinition",
"description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 / R-IDP-02).",
"description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 Line truncated
"type": "object",
"additionalProperties": false,
"required": ["name", "source"],
@@ -139,6 +139,19 @@
"supplementalGroups": { "type": "array", "items": { "type": "integer" }, "description": "Additional GIDs added to the first process in each container." }
}
},
"hardware": {
"type": "object",
"description": "Optional host HARDWARE CAPABILITY the workload needs, for the container kind (docs/13). This field declares INTENT (`what device class this service needs`), never MECHANISM (`which host path to bind-mount`, `which extended resource to request`). That separation is the whole point of the enum: the platform currently satisfies `intel-quicksync` with a /dev/dri hostPath volume and a matching mount on the app container, but a later migration to the Intel GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) must be a catalog/render.py change and NOTHING else - no ServiceDefinition may need editing, and no reviewer may have to re-derive per-service device wiring. Do NOT 'simplify' this into a raw device path, a hostPath, or a resource request; that would push a platform implementation detail into the operator-facing contract and make the migration a catalog-wide edit. Absent => no device wiring at all (existing apps render byte-identical). Ignored for kind=vm (a VM gets devices through KubeVirt's own domain.devices).",
"additionalProperties": false,
"minProperties": 1,
"properties": {
"gpu": {
"type": "string",
"description": "GPU capability class this service needs. `intel-quicksync` = the host's Intel Quick Sync Video render device (the iGPU's /dev/dri/renderD* node) for hardware video transcode/decode. Enum, not a free-form device path, so the satisfying mechanism stays swappable (see the parent description). Extend the enum when a genuinely different capability class appears (e.g. nvidia-nvenc), never to name a different path to the same one.",
"enum": ["intel-quicksync"]
}
}
},
"exposure": {
"type": "string",
"description": "internal = LAN-only (internal DNS + Gateway on LAN); public = additionally edge port-forwarded 80/443 (docs/09; R-DNS/R-NET).",