feat(plex): hardware transcoding via hardware.gpu: intel-quicksync, plus the runbook for the manual last mile #32

Merged
jakman85 merged 2 commits from feat/plex-quicksync into main 2026-10-02 02:17:48 +00:00
Owner

This depends on #27 (feat/sd-hardware-capability) and MUST merge after it. This branch was
cut from #27, not from main, so #27's commit (d9baa2a) appears in this PR's diff. That is
deliberate: per CLAUDE.md §1 merging #27 is not mine to do, and copying its changes here as
duplicate commits would have been worse than showing them. Review only 7a9d37e. Once #27
merges, this range collapses to that one commit on its own.

Plex has been transcoding in software on hardware that can do it in the iGPU. This opts its
ServiceDefinition into the hardware capability #27 delivers, re-renders the golden, and — the part
that actually makes the feature real — writes the runbook for the one step that cannot be automated.

The change is four lines of YAML

hardware:
  gpu: intel-quicksync

catalog/plex.yaml, between env and exposure to match the schema's property order. The comment
around it is long on purpose, in that file's house style: it records that the render device exists on
all three prod nodes with i915 bound (so no nodeSelector is rendered and none is wanted); why
a supplementalGroups entry would be inert rather than a hardening (the image's cont-init hook
adds the plex user to the device's group inside the container, and s6-setuidgid then replaces
the process group list from /etc/group, discarding anything Kubernetes supplied); why
DirectoryOrCreate is load-bearing (apps-wave-0 is wait: true with apps-wave-1 depending on it,
so a failed mount would stall the whole app tier for a feature that is not load-bearing); and that
enabling the Plex-side toggle is a separate manual operator step, with a pointer to the runbook.

The whole rendered diff

python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation →
apps/wave-0/plex.yaml | 6 ++++++, one file in the entire tree:

         - name: media
           mountPath: /media
           readOnly: true
+        - name: dri
+          mountPath: /dev/dri
       volumes:
       - name: config
         persistentVolumeClaim:
           claimName: plex-config
       - name: media
         persistentVolumeClaim:
           claimName: plex-media
+      - name: dri
+        hostPath:
+          path: /dev/dri
+          type: DirectoryOrCreate

No --emit-waves (it writes into clusters/base/, a sensitive path that would force a human
reviewer, and it is a no-op unless the set of files under apps/ changes — it does not).
apps/wave-1/keycloak.yaml and apps/wave-0/vm-demo.yaml — rewritten by the same command and
excluded from the golden comparison by RENDER_GOLDEN_NOT_ONBOARDED, so a spurious diff there would
land ungated — were checked and are unchanged.

The feature is inert until a human clicks a box

Mounting the device does not enable hardware transcoding. Plex's HardwareAcceleratedCodecs
preference does, and it was absent from Preferences.xml — the toggle has never been on.

It cannot be automated, and that is verified rather than assumed. A bounded grep across the
running image's entire init surface (/etc/cont-init.d, /etc/services.d, /etc/fix-attrs.d,
/plex-common.sh, /usr/local/bin) found zero PLEX_PREFERENCE* support; the six setPref keys
the image does write (MachineIdentifier, ProcessedMachineIdentifier, PlexOnlineToken,
customConnections, allowedNetworks, TranscoderTempDirectory) are all unrelated. There is no
GitOps path to the toggle.

So runbooks/plex-quicksync.md is new, registered in runbooks/kustomization.yaml's
configMapGenerator
(a runbook in git with no generator entry is reviewed, merged, and still a 404
— the pod serves a ConfigMap, not the git tree) and indexed in runbooks/README.md. It covers:

  • the GitOps / manual boundary as an explicit two-row table;
  • the toggle — Settings → Transcoder → "Use hardware acceleration when available" — and the real
    preference keys, HardwareAcceleratedCodecs confirmed by extracting strings from the 23 MB
    Plex Media Server binary in the running container rather than from recall. Including
    HardwareAcceleratedCodecsEnableHEVC does not exist, recorded so nobody hunts for it, and that
    TranscoderToneMappingAgorithm's misspelling is upstream's and must not be "fixed";
  • verification: Transcode (hw) on the Plex dashboard, stated plainly as the only proof
    available. Plex renders no metricsEndpoint, so there is no ServiceMonitor, no transcode-mode
    metric and no alert that could ever fire. I deliberately did not propose one — this repo has
    been burned at scale by rules evaluating against nothing while looking like coverage;
  • the failure mode to expect first: whether containerd's device cgroup lets a non-privileged pod
    open() a bind-mounted char device is UNVERIFIED and only deploying settles it (uid 0 fixes
    file permissions but does not bypass the cgroup gate). Concrete diagnostic sequence, then the
    fallback — the Intel Device Plugin (gpu.intel.com/i915), which because the SD field is intent
    rather than a device path is a catalog/render.py change plus a re-render with no
    ServiceDefinition edit anywhere
    . privileged: true is not an option: Kyverno
    restrict-host-namespaces is Enforce and rejects it;
  • the floating-image caveat: everything known about the hook came from digest
    sha256:e0ab27395614… (plexmediaserver 1.43.4.10903), while the Deployment runs
    plexinc/pms-docker:latest with imagePullPolicy: Always. A silent pull can change the behaviour
    this depends on with no commit in this repo.

Gate — rc 3, UNDETERMINED, not green

Pinned toolbox, from a worktree (DEV_KUBECONFIG=/dev/null + an ENGINE shim bind-mounting the main
checkout's .git), run on this exact commit:

stage result
spec-immutability SKIP — no sibling spec repo (expected)
decisions-log pass — archive frozen, 9 entries well-formed
lint pass
l0 pass — incl. alert-runbooks-resolvable (R-OBS-04) scope=whole-repo
render-golden pass — all 10 onboarded goldens reproduce from the canonical render
l1 pass — incl. container-workload-render (R-PLAT-02)
approval-policy pass (event='none', N/A locally)
gitea-settings SKIP — no SETTINGS_API_TOKEN
conformance SKIP — CLUSTER_AVAILABLE unset, L2/L3 did not run

rc 3. Three skips — the known local ceiling — no fourth skip, and nothing failed. Touched
requirements picked up R-PLAT-02 and R-OBS-04 as expected.

What this does NOT prove

Hardware transcoding has not been observed working. A green gate proves the manifest renders and
the golden matches; it says nothing about runtime. The device-cgroup question is open, and the
Plex-side toggle was still unset at the time of writing. Full detail in
decisions/2026-09-30-plex-quicksync.md.

Two follow-ups documented, deliberately not implemented

  • Transcode scratch is on the wrong medium and it is invisible. TranscoderTempDirectory=/transcode
    is not a volume — it is the container's writable overlay, i.e. the node's /data: NVMe on
    prod-2, SATA on prod-1 and prod-3, so scratch performance changes silently on reschedule. Both
    fixes need a schema change (a nodeSelector field, or an emptyDir medium: Memory field), so
    neither belongs here. The RAM option is the better one — it removes the node asymmetry outright
    instead of working around it — which is exactly why it deserves its own review.
  • GuC/HuC firmware is not loaded (i915.enable_guc=0). Costs low-power H.264 encode quality
    only. Needs a kernel parameter plus a serialized rolling reboot — an operator action under
    CLAUDE.md §5, on a cluster with a documented history of that going badly (a prod node hung on a
    warm reboot on 2026-09-03).

🤖 Generated with Claude Code

> **This depends on #27 (`feat/sd-hardware-capability`) and MUST merge after it.** This branch was > cut from #27, not from `main`, so #27's commit (`d9baa2a`) appears in this PR's diff. That is > deliberate: per CLAUDE.md §1 merging #27 is not mine to do, and copying its changes here as > duplicate commits would have been worse than showing them. **Review only `7a9d37e`.** Once #27 > merges, this range collapses to that one commit on its own. Plex has been transcoding in **software** on hardware that can do it in the iGPU. This opts its ServiceDefinition into the `hardware` capability #27 delivers, re-renders the golden, and — the part that actually makes the feature real — writes the runbook for the one step that cannot be automated. ## The change is four lines of YAML ```yaml hardware: gpu: intel-quicksync ``` `catalog/plex.yaml`, between `env` and `exposure` to match the schema's property order. The comment around it is long on purpose, in that file's house style: it records that the render device exists on **all three** prod nodes with `i915` bound (so no `nodeSelector` is rendered and none is wanted); why a `supplementalGroups` entry would be **inert rather than a hardening** (the image's cont-init hook adds the `plex` user to the device's group *inside* the container, and `s6-setuidgid` then **replaces** the process group list from `/etc/group`, discarding anything Kubernetes supplied); why `DirectoryOrCreate` is load-bearing (`apps-wave-0` is `wait: true` with `apps-wave-1` depending on it, so a failed mount would stall the whole app tier for a feature that is not load-bearing); and that enabling the Plex-side toggle is a separate manual operator step, with a pointer to the runbook. ## The whole rendered diff `python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` → `apps/wave-0/plex.yaml | 6 ++++++`, one file in the entire tree: ```diff - name: media mountPath: /media readOnly: true + - name: dri + mountPath: /dev/dri volumes: - name: config persistentVolumeClaim: claimName: plex-config - name: media persistentVolumeClaim: claimName: plex-media + - name: dri + hostPath: + path: /dev/dri + type: DirectoryOrCreate ``` No `--emit-waves` (it writes into `clusters/base/`, a sensitive path that would force a human reviewer, and it is a no-op unless the *set* of files under `apps/` changes — it does not). `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml` — rewritten by the same command and excluded from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED`, so a spurious diff there would land **ungated** — were checked and are unchanged. ## The feature is inert until a human clicks a box Mounting the device does **not** enable hardware transcoding. Plex's `HardwareAcceleratedCodecs` preference does, and it was **absent** from `Preferences.xml` — the toggle has never been on. **It cannot be automated, and that is verified rather than assumed.** A bounded grep across the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`, `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin`) found **zero** `PLEX_PREFERENCE*` support; the six `setPref` keys the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all unrelated. There is no GitOps path to the toggle. So `runbooks/plex-quicksync.md` is new, **registered in `runbooks/kustomization.yaml`'s `configMapGenerator`** (a runbook in git with no generator entry is reviewed, merged, and still a 404 — the pod serves a ConfigMap, not the git tree) and indexed in `runbooks/README.md`. It covers: - the **GitOps / manual boundary** as an explicit two-row table; - the toggle — Settings → Transcoder → "Use hardware acceleration when available" — and the real preference keys, `HardwareAcceleratedCodecs` confirmed by extracting strings from the 23 MB `Plex Media Server` binary in the running container rather than from recall. Including **`HardwareAcceleratedCodecsEnableHEVC` does not exist**, recorded so nobody hunts for it, and that `TranscoderToneMappingAgorithm`'s misspelling is **upstream's** and must not be "fixed"; - **verification**: `Transcode (hw)` on the Plex dashboard, stated plainly as the **only** proof available. Plex renders no `metricsEndpoint`, so there is no ServiceMonitor, no transcode-mode metric and **no alert that could ever fire**. I deliberately did not propose one — this repo has been burned at scale by rules evaluating against nothing while looking like coverage; - the **failure mode to expect first**: whether containerd's device cgroup lets a non-privileged pod `open()` a bind-mounted char device is **UNVERIFIED** and only deploying settles it (uid 0 fixes file permissions but does **not** bypass the cgroup gate). Concrete diagnostic sequence, then the fallback — the **Intel Device Plugin** (`gpu.intel.com/i915`), which because the SD field is intent rather than a device path is a `catalog/render.py` change plus a re-render with **no ServiceDefinition edit anywhere**. `privileged: true` is not an option: Kyverno `restrict-host-namespaces` is Enforce and rejects it; - the **floating-image caveat**: everything known about the hook came from digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), while the Deployment runs `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. A silent pull can change the behaviour this depends on with no commit in this repo. ## Gate — rc 3, UNDETERMINED, not green Pinned toolbox, from a worktree (`DEV_KUBECONFIG=/dev/null` + an `ENGINE` shim bind-mounting the main checkout's `.git`), run on this exact commit: | stage | result | |---|---| | spec-immutability | **SKIP** — no sibling spec repo (expected) | | decisions-log | pass — archive frozen, 9 entries well-formed | | lint | pass | | l0 | pass — incl. `alert-runbooks-resolvable (R-OBS-04) scope=whole-repo` | | **render-golden** | **pass** — all 10 onboarded goldens reproduce from the canonical render | | l1 | pass — incl. `container-workload-render (R-PLAT-02)` | | approval-policy | pass (`event='none'`, N/A locally) | | gitea-settings | **SKIP** — no `SETTINGS_API_TOKEN` | | conformance | **SKIP** — `CLUSTER_AVAILABLE` unset, L2/L3 did not run | `rc 3`. Three skips — the known local ceiling — **no fourth skip, and nothing failed**. Touched requirements picked up `R-PLAT-02` and `R-OBS-04` as expected. ## What this does NOT prove **Hardware transcoding has not been observed working.** A green gate proves the manifest renders and the golden matches; it says nothing about runtime. The device-cgroup question is open, and the Plex-side toggle was still unset at the time of writing. Full detail in `decisions/2026-09-30-plex-quicksync.md`. ## Two follow-ups documented, deliberately not implemented - **Transcode scratch is on the wrong medium and it is invisible.** `TranscoderTempDirectory=/transcode` is **not a volume** — it is the container's writable overlay, i.e. the node's `/data`: NVMe on prod-2, **SATA on prod-1 and prod-3**, so scratch performance changes silently on reschedule. Both fixes need a **schema** change (a `nodeSelector` field, or an emptyDir `medium: Memory` field), so neither belongs here. The RAM option is the better one — it removes the node asymmetry outright instead of working around it — which is exactly why it deserves its own review. - **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). Costs low-power H.264 **encode** quality only. Needs a kernel parameter plus a serialized rolling reboot — an operator action under CLAUDE.md §5, on a cluster with a documented history of that going badly (a prod node hung on a warm reboot on 2026-09-03). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
gitea-admin added 2 commits 2026-09-30 21:11:50 +00:00
feat(schema): declare a host hardware capability, hardware.gpu: intel-quicksync
gate / gate (pull_request) Successful in 1m40s
gate / conformance (pull_request) Has been skipped
d9baa2a482
A ServiceDefinition could not say it needs the host's Intel QuickSync render
device. The schema is `additionalProperties: false`, so there was no authorable
form for it at all, and `apps/` is generated — a hand-edit would be reverted by
the next render.

Adds optional top-level `hardware`, with one optional member `gpu`, an enum
whose only current value is `intel-quicksync` (schema 1.7.0 -> 1.8.0).

The enum is the point. The SD declares WHAT CLASS of device it needs; the
hostPath and the mount live only in catalog/render.py. Migrating to the Intel
GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) is then
a renderer change with zero catalog edits. Letting an SD name `/dev/dri`
directly would look simpler and make that migration a catalog-wide edit, so
both the schema description and the renderer contract say so explicitly.

When `hardware.gpu == intel-quicksync`, r_deployment emits exactly two things,
on the app container only:

  - a hostPath volume `dri` -> /dev/dri, type DirectoryOrCreate;
  - a read-write volumeMount of it at /dev/dri (VA-API opens the render node
    read-write, so deliberately not readOnly).

Mounting the device is sufficient. In particular NO
podSecurityContext.supplementalGroups is emitted, and adding one would be a
regression rather than a hardening. Measured in the running container on
prod-2, 2026-09-30:

  PID 1   s6-svscan            Uid: 0     Gid: 0     Groups: 0
  pid 246 Plex Media Server    Uid: 1000  Gid: 1000  Groups: 100 1000

The service is started with `exec s6-setuidgid plex ...`, and the child's group
list is exactly the /etc/group account-database list — root's gid 0, which
PID 1 holds, is gone. That is a setgroups() that REPLACES rather than appends,
so anything Kubernetes supplies is discarded before the app starts. What works
is the image's own /etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner,
which stats each /dev/dri char device, groupadds a matching group inside the
container and usermods the app user into it, every start, nothing hardcoded.
An inert supplementalGroups would have read as the mechanism to whoever
debugged this next, so the absence is documented at the emission site and
asserted by --selftest. Corollary: the hook re-stats on every start, so host
GID drift is self-healing and no GID is pinned anywhere in this repo.

`DirectoryOrCreate` is load-bearing. A node that comes back without the render
device (rebuild, BIOS change, driver regression, hardware swap) must degrade
the workload to software transcoding, not fail the mount: apps-wave-0 is
`wait: true` and apps-wave-1 dependsOn it, so a failed hostPath mount stalls
the whole app tier for a feature that is not load-bearing. /dev is tmpfs, so
the empty directory the kubelet creates leaves nothing durable behind. Its cost
— a missing device is silent, and no metric exists to alert on — is recorded in
the contract rather than papered over with an alert that could not fire.

Also deliberately NOT emitted: no `privileged: true` (Kyverno restrict-host-
namespaces is Enforce and it is not needed), no nodeSelector (placement is a
separate concern with its own justification), no resource requests (there is no
app-container resource render path).

No catalog entry uses the field. Re-rendering `--all catalog -o apps` is a zero
diff, which is the proof the change is inert until opted into.

Self-tests in the existing check(...) style: the positive (volume + mount, and
no securityContext / privileged / nodeSelector), the negative (an SD without
`hardware` gains no volume, no mount, no securityContext — mirroring the
podSecurityContext negative) and a pass-through case (an SD's own
podSecurityContext survives byte-for-byte). Note that NO gate stage runs
`render.py --selftest`; they were run by hand and the gap is recorded in the
handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
feat(plex): opt into hardware.gpu: intel-quicksync, and write the runbook for the step GitOps cannot reach
gate / gate (pull_request) Successful in 1m42s
gate / conformance (pull_request) Has been skipped
7a9d37e601
Plex has been transcoding in software on hardware that can do it in the iGPU. This
opts its ServiceDefinition into the `hardware` capability from #27 and re-renders.

catalog/plex.yaml gains four lines of YAML and the comment that explains them:

    hardware:
      gpu: intel-quicksync

which renders exactly two things into apps/wave-0/plex.yaml and nothing else — a
`dri` hostPath volume (/dev/dri, type: DirectoryOrCreate) and a read-write /dev/dri
volumeMount on the app container. Whole-tree render diff: 6 added lines, one file.

The comment records what is not obvious from the four lines: the device exists on
ALL THREE prod nodes with i915 bound (so no nodeSelector is rendered and none is
wanted); why `supplementalGroups` would be INERT rather than a hardening (the
image's cont-init hook adds the plex user to the device's group inside the
container, and s6-setuidgid then REPLACES the process group list from /etc/group,
discarding anything Kubernetes supplied); and why `DirectoryOrCreate` is
load-bearing (apps-wave-0 is wait:true with apps-wave-1 depending on it, so a
failed mount would stall the whole app tier for a feature that is not).

THE FEATURE IS INERT UNTIL A HUMAN CLICKS A BOX, and that is the part this commit
is really about. Mounting the device does not enable hardware transcoding; Plex's
`HardwareAcceleratedCodecs` preference does, and it is ABSENT from Preferences.xml
— the toggle has never been on. It cannot be automated: a bounded grep across the
running image's entire init surface (/etc/cont-init.d, /etc/services.d,
/etc/fix-attrs.d, /plex-common.sh, /usr/local/bin) found ZERO PLEX_PREFERENCE*
support, and the six setPref keys the image does write are all unrelated. So
runbooks/plex-quicksync.md is the procedure, registered in the configMapGenerator
(a runbook in git with no generator entry is reviewed, merged, and still a 404) and
indexed in runbooks/README.md. It covers the GitOps/manual boundary, the toggle and
the real preference keys (including the widely-cited one that does not exist and
the upstream misspelling nobody should "fix"), verification via `Transcode (hw)` on
the Plex dashboard — stated plainly as the ONLY proof, because Plex renders no
ServiceMonitor and an alert could never fire — the device-cgroup failure to expect
first with its Intel-device-plugin fallback, and the floating-`:latest` caveat.

Stacked on #27 (feat/sd-hardware-capability), which is open and unmerged; that
branch's commit is in this range by design and this must merge after it.

NOT VERIFIED, and the handoff says so at length: hardware transcoding has not been
observed working. Whether containerd's device cgroup lets a non-privileged pod
open() a bind-mounted char device is an open question that only deploying settles,
and the Plex-side toggle was still unset at the time of writing. A green gate
proves the manifest renders; it proves nothing about transcoding.

Gate (pinned toolbox, worktree): rc 3 — UNDETERMINED, not green. Passed:
decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped (the known
local ceiling): spec-immutability (no sibling spec repo), gitea-settings (no
SETTINGS_API_TOKEN), conformance (CLUSTER_AVAILABLE unset, so L2/L3 did not run).
No fourth skip, nothing failed.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
gitea-admin force-pushed feat/plex-quicksync from 7a9d37e601 to df1fd1b3d1 2026-10-02 02:08:22 +00:00 Compare
jakman85 approved these changes 2026-10-02 02:17:43 +00:00
jakman85 merged commit cdf3e14e27 into main 2026-10-02 02:17:48 +00:00
Sign in to join this conversation.
No Reviewers
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#32