feat(plex): opt into hardware.gpu: intel-quicksync, and write the runbook for the step GitOps cannot reach
gate / gate (pull_request) Successful in 1m42s
gate / conformance (pull_request) Has been skipped

Plex has been transcoding in software on hardware that can do it in the iGPU. This
opts its ServiceDefinition into the `hardware` capability from #27 and re-renders.

catalog/plex.yaml gains four lines of YAML and the comment that explains them:

    hardware:
      gpu: intel-quicksync

which renders exactly two things into apps/wave-0/plex.yaml and nothing else — a
`dri` hostPath volume (/dev/dri, type: DirectoryOrCreate) and a read-write /dev/dri
volumeMount on the app container. Whole-tree render diff: 6 added lines, one file.

The comment records what is not obvious from the four lines: the device exists on
ALL THREE prod nodes with i915 bound (so no nodeSelector is rendered and none is
wanted); why `supplementalGroups` would be INERT rather than a hardening (the
image's cont-init hook adds the plex user to the device's group inside the
container, and s6-setuidgid then REPLACES the process group list from /etc/group,
discarding anything Kubernetes supplied); and why `DirectoryOrCreate` is
load-bearing (apps-wave-0 is wait:true with apps-wave-1 depending on it, so a
failed mount would stall the whole app tier for a feature that is not).

THE FEATURE IS INERT UNTIL A HUMAN CLICKS A BOX, and that is the part this commit
is really about. Mounting the device does not enable hardware transcoding; Plex's
`HardwareAcceleratedCodecs` preference does, and it is ABSENT from Preferences.xml
— the toggle has never been on. It cannot be automated: a bounded grep across the
running image's entire init surface (/etc/cont-init.d, /etc/services.d,
/etc/fix-attrs.d, /plex-common.sh, /usr/local/bin) found ZERO PLEX_PREFERENCE*
support, and the six setPref keys the image does write are all unrelated. So
runbooks/plex-quicksync.md is the procedure, registered in the configMapGenerator
(a runbook in git with no generator entry is reviewed, merged, and still a 404) and
indexed in runbooks/README.md. It covers the GitOps/manual boundary, the toggle and
the real preference keys (including the widely-cited one that does not exist and
the upstream misspelling nobody should "fix"), verification via `Transcode (hw)` on
the Plex dashboard — stated plainly as the ONLY proof, because Plex renders no
ServiceMonitor and an alert could never fire — the device-cgroup failure to expect
first with its Intel-device-plugin fallback, and the floating-`:latest` caveat.

Stacked on #27 (feat/sd-hardware-capability), which is open and unmerged; that
branch's commit is in this range by design and this must merge after it.

NOT VERIFIED, and the handoff says so at length: hardware transcoding has not been
observed working. Whether containerd's device cgroup lets a non-privileged pod
open() a bind-mounted char device is an open question that only deploying settles,
and the Plex-side toggle was still unset at the time of writing. A green gate
proves the manifest renders; it proves nothing about transcoding.

Gate (pinned toolbox, worktree): rc 3 — UNDETERMINED, not green. Passed:
decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped (the known
local ceiling): spec-immutability (no sibling spec repo), gitea-settings (no
SETTINGS_API_TOKEN), conformance (CLUSTER_AVAILABLE unset, so L2/L3 did not run).
No fourth skip, nothing failed.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-30 13:09:20 -05:00
1 parent d9baa2a482
commit 7a9d37e601
6 files changed
+504

No files matched your search

+6
View File
@@ -143,6 +143,8 @@ spec:
- name: media
mountPath: /media
readOnly: true
- name: dri
mountPath: /dev/dri
volumes:
- name: config
persistentVolumeClaim:
@@ -150,6 +152,10 @@ spec:
- name: media
persistentVolumeClaim:
claimName: plex-media
- name: dri
hostPath:
path: /dev/dri
type: DirectoryOrCreate
---
# Service
apiVersion: v1
+52
View File
@@ -80,6 +80,58 @@ env:
# remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
# clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
# reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
# HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
# INTENT ONLY. This says "this workload needs an Intel Quick Sync render device"; it does NOT name a
# device path. The renderer currently satisfies it with exactly two things — a `/dev/dri` hostPath
# volume (`type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container —
# and nothing else: no nodeSelector, no privileged, no resources (schemas/renderer-contract.md
# `### hardware`). If that mechanism is later replaced by the Intel GPU device plugin
# (`gpu.intel.com/i915`), it is a `catalog/render.py` change and THIS BLOCK DOES NOT MOVE. Do not
# "simplify" it into a raw `/dev/...` path; that is the whole point of the enum.
#
# THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
# anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
# GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Plex happens to run on k3s-prod-2 today, but
# that is scheduler history rather than a requirement — this field deliberately renders no
# nodeSelector, because placement is a separate concern with its own justification.
#
# NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
# The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
# the host `render` GID in as a supplemental group. It cannot work here. MEASURED IN THE RUNNING
# CONTAINER, 2026-09-30: the image's own
# `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device under
# `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
# `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
# GID renumbering self-healing (no GID is pinned anywhere in this repo). The service then execs via
# `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from `/etc/group` rather
# than appending to it: `Plex Media Server` runs with `Groups: 100 1000`, and root's gid `0` — which
# PID 1 (`s6-svscan`) does hold — is GONE. Anything Kubernetes supplies therefore reaches s6-svscan
# and is discarded before Plex ever starts. Mounting the device is sufficient; the group is the
# image's job and it does it correctly. `render.py --selftest` asserts the ABSENCE of a pod
# securityContext for this field precisely so nobody adds an inert one back later.
#
# `DirectoryOrCreate` IS THE SAFE FAILURE MODE, not laziness. A node that comes back without the
# device (rebuild, BIOS reset, driver regression, hardware swap) gets an empty directory and Plex
# degrades to SOFTWARE transcoding. `Directory` would fail the mount instead — and because
# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that would stall the WHOLE app
# tier for a feature that is not load-bearing. The price, stated plainly: a missing device is
# SILENT. This SD renders no `metricsEndpoint`, so there is no ServiceMonitor, no transcode-mode
# metric, and no alert that could ever fire. The only proof is an operator reading the Plex
# dashboard.
#
# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Mounting the device does
# not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
# is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
# set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
# `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin`) found ZERO `PLEX_PREFERENCE*` support,
# and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
# `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
# unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took and the
# device-cgroup failure mode to expect first.
hardware:
gpu: intel-quicksync
exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
egress: internet # normal WAN egress (docs/03; R-NET-05)
storage:
+149
View File
@@ -0,0 +1,149 @@
---
date: 2026-09-30
branch: feat/plex-quicksync
title: Plex opts into `hardware.gpu: intel-quicksync` — the device mount is GitOps, the Plex-side toggle is a documented manual step, and hardware transcoding has NOT been observed working
---
## What changed
Three things, and the third is the point of the branch.
1. **`catalog/plex.yaml`** — added the four lines the whole feature reduces to:
```yaml
hardware:
gpu: intel-quicksync
```
placed between `env` and `exposure` to match the schema's own property order, with ~48 lines of
comment in the file's established style. The comment records: the field is **intent, not
mechanism** (a later move to the Intel device plugin is a `render.py` change and this block does
not move); that `/dev/dri/renderD128` is present on **all three** prod nodes with `i915` bound;
**why a `supplementalGroups` entry would be inert** (the image's cont-init hook does the group
work inside the container and `s6-setuidgid` replaces the process group list from `/etc/group`);
why `DirectoryOrCreate` is the load-bearing safe failure mode (`apps-wave-0` is `wait: true` with
`apps-wave-1` depending on it, so a failed mount would stall the whole app tier for a feature
that is not load-bearing); and that the Plex-side toggle is a **separate manual operator step**,
pointing at the new runbook.
2. **`apps/wave-0/plex.yaml`** — re-rendered. Six added lines, nothing else, in the whole `apps/`
tree:
```
- name: dri
mountPath: /dev/dri
- name: dri
hostPath:
path: /dev/dri
type: DirectoryOrCreate
```
3. **`runbooks/plex-quicksync.md`** (new), registered in `runbooks/kustomization.yaml`'s
`configMapGenerator` as `- plex-quicksync=plex-quicksync.md` and indexed in `runbooks/README.md`
under "Operator procedures". **The file alone would have been a 404** — the pod serves a
ConfigMap, not the git tree, and `conformance/tests/l0-static/alert-runbooks-resolvable` exists
precisely because "the file is in git" and "the URL resolves" are different claims. The page
covers: the GitOps/manual boundary as a table; the toggle (Settings → Transcoder → "Use hardware
acceleration when available", preference key `HardwareAcceleratedCodecs`); the real neighbouring
keys and the widely-cited one that **does not exist**; that the only proof is `Transcode (hw)` on
the Plex dashboard **and why no alert is possible**; the device-cgroup failure to expect first
with a concrete open() diagnostic and the Intel-device-plugin fallback; and the floating-`:latest`
caveat with a digest check.
**Stacked on PR #27 (`feat/sd-hardware-capability`), which is open and unmerged.** This branch was
cut from it rather than from `main`, so its one commit (`d9baa2a`, the schema + renderer + contract)
is in this PR's range. That is deliberate: merging #27 is not an agent's action (CLAUDE.md §1), and
duplicating its changes here would have been worse than showing them. **This must merge after #27.**
Not done, deliberately: no `--emit-waves` (it writes into `clusters/base/`, a sensitive path that
would force a human reviewer, and it is a no-op unless the *set* of files under `apps/` changes — it
does not here). No `podSecurityContext`. No `nodeSelector`. No alert rule. No `kubectl apply`.
### Two follow-ups left unclaimed, with the reasoning
- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
`TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
transcode scratch performance changes silently on reschedule, with nothing reporting it. Both
candidate fixes need a **schema** change and so neither belongs in this PR: pin Plex to prod-2
(needs a `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory`
emptyDir and transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not
have). **The RAM option is the better one** — it removes the node asymmetry outright instead of
working around it — which is exactly why it deserves its own PR and its own review rather than
being smuggled in behind a four-line catalog change. Documented in the runbook's "Known gaps".
- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
quality only; decode and this feature are unaffected. Changing it needs a kernel parameter plus a
**serialized rolling reboot** — an operator action under CLAUDE.md §5, on a cluster with a
documented history of that going badly (a prod node hung on a warm reboot on 2026-09-03). Kept
well clear of this PR.
## What I verified
- **The render is exactly the intended change, and nothing else moved.** After
`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` (exit 0,
silent): `git diff --stat apps/` = `apps/wave-0/plex.yaml | 6 ++++++`, and
`git status --porcelain apps/` = a single ` M apps/wave-0/plex.yaml`. Read line by line, the diff
is the `dri` volumeMount and the `dri` hostPath volume, and nothing more. **Specifically checked:**
the same invocation also rewrites `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, which
are git-tracked but excluded from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in
`ci/render-golden.sh` — a spurious diff there would have landed **ungated**. Neither file shows a
diff; whole-repo `git status --porcelain` lists only `apps/wave-0/plex.yaml` and `catalog/plex.yaml`
plus my new files.
- **The rendered output matches what the contract promises** — `schemas/renderer-contract.md`
`### hardware` says exactly two things (a `/dev/dri` hostPath with `type: DirectoryOrCreate` and a
read-write mount on the app container) and exactly two things appeared. No `securityContext`, no
`nodeSelector`, no resources in the diff.
- **The runbook is projected into the server, not merely committed.** `runbooks/kustomization.yaml`
carries `- plex-quicksync=plex-quicksync.md`, whose shape matches the generator-entry regex the
L0 check parses; the extensionless key is what makes `/runbooks/plex-quicksync` a plain static
lookup. Also indexed in `runbooks/README.md`.
- **The runbook clears the L0 quality floor** that `alert-runbooks-resolvable` applies to
alert-linked pages, even though nothing links to this one (it is advisory for an unreferenced
page): all four required sections (`## What this means here`, `## First commands`,
`## Likely causes, most likely first`, `## When to escalate or stop`), well over the 1500-byte
floor, `dev.sh` used throughout, and `--context k3s-prod` on every command.
- **The gate.** See the run recorded in the PR description for the per-stage table and the real rc.
- **No secret was printed.** Every `Preferences.xml` read in the runbook greps one named key, and the
page states explicitly why: the file holds `PlexOnlineToken`. Nothing was `sops -d`'d.
## What I did NOT verify
**Hardware transcoding has NOT been observed working.** Nothing in this branch is evidence that it
does. A green gate here proves the manifest renders and the golden matches — it proves nothing about
runtime behaviour, and `vars.CLUSTER_AVAILABLE` is unset so L2/L3 conformance did not run at all.
- **The device-cgroup question is OPEN, and it is the largest remaining technical risk.** Whether
containerd on this k3s build permits a **non-privileged** pod to `open()` a bind-mounted character
device is **UNVERIFIED**. It can only be settled by deploying and trying. The container inits as
uid 0, which settles *file* permissions, but uid 0 does **not** bypass the cgroup device gate — and
the two failures look identical from the outside. If it is refused, the fallback is the Intel
Device Plugin (`gpu.intel.com/i915`), which is a `catalog/render.py` change plus a re-render with
**no ServiceDefinition edit**; `privileged: true` is not available (Kyverno
`restrict-host-namespaces` is Enforce). The runbook has the diagnostic sequence.
- **The Plex-side toggle was still unset at the time of writing.** `HardwareAcceleratedCodecs` was
**absent** from `Preferences.xml` — the toggle has never been enabled on this server. Until an
operator ticks it in the UI, this branch changes the pod spec and **nothing about transcoding**.
That is the whole reason the runbook exists.
- **No metric, so no alert, so no automated detection — permanently.** `catalog/plex.yaml` declares
no `metricsEndpoint`, so Plex renders no ServiceMonitor; there is no transcode-mode signal for a
rule to evaluate. A silent degradation to software transcoding (a node losing its iGPU, the
`DirectoryOrCreate` path) will **not** be reported by anything. I deliberately did not add an
alert: with no metric behind it, it could never fire, and this repo has been burned at scale by
exactly that. Closing the gap honestly means an exporter first.
- **I did not re-derive this session's node or in-container measurements.** The three-node
`renderD128`/`i915`/GID-992 facts, the cont-init hook behaviour, the `s6-setuidgid` group
replacement, the `PLEX_PREFERENCE*` absence, the six `setPref` keys and the
`HardwareAcceleratedCodecs` string were established earlier in this session and are cited, not
re-measured, by me. Per CLAUDE.md §2b, treat this file as a record of belief and re-measure against
the cluster before relying on any of it.
- **The image is a floating tag.** Everything known about the hook came from digest
`sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), while the Deployment runs
`plexinc/pms-docker:latest` with `imagePullPolicy: Always`. A silent pull can invalidate the
"no `supplementalGroups` needed" conclusion **with no commit in this repo**. Not guarded against —
the runbook tells the reader to re-check the hook after an image bump, which is documentation, not
enforcement.
- **I did not verify the live Gitea gate state** (branch protection, required contexts, runner
online) for this PR. Per §2b that must be queried, not read out of `CLAUDE.md`'s table; I did not
query it.
- **`/runbooks/plex-quicksync` was not fetched.** The L0 check asserts every static precondition for
an HTTP 200; it does not make the request, and I did not either.
+8
View File
@@ -115,6 +115,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
- **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod
(`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware
acceleration when available"**, and no environment variable can do it (the image supports no
`PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`,
read out of the binary), says which widely-cited key **does not exist**, and is explicit that the
**only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no
alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin
fallback, and why a `:latest` pull can silently break it.
- **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap
+6
View File
@@ -90,6 +90,12 @@ configMapGenerator:
# because the plex.tv claim window is short and is not a thing to go looking
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
- plex-claim=plex-claim.md
# Operator procedure, not an alert page, and NO alert is possible for it: plex renders no
# metricsEndpoint, so there is no ServiceMonitor, no transcode-mode metric and nothing a rule
# could evaluate. Served because the feature it completes is INERT until a human ticks a box in
# the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped
# feature stays switched off for months.
- plex-quicksync=plex-quicksync.md
- pvc-capacity=pvc-capacity.md
- targets-down=targets-down.md
- update-rollback=update-rollback.md
+283
View File
@@ -0,0 +1,283 @@
# plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did
**Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**,
not an incident page. It is indexed here because the feature it completes is **inert until a human
clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
to prevent.
**Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the
rendered golden — the `dri` hostPath volume and the `/dev/dri` mount), `catalog/render.py` (the
mechanism), `schemas/renderer-contract.md` `### hardware` (the contract).
**Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile.
## What this means here
Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync
moves that work off the CPU — which matters on a three-node cluster where the CPU is also running
everything else. Two independent things have to be true before that happens, and **only one of them
is in git**:
| Half | Who does it | State |
|---|---|---|
| The render device is inside the container at `/dev/dri` | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders a `dri` hostPath volume (`/dev/dri`, `type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container. Flux applies it. | Done by merging that change. |
| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
**There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`,
`/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin` — found **zero** `PLEX_PREFERENCE*` support.
Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentifier`,
`ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`,
`TranscoderTempDirectory`), none of them the transcode toggle. No environment variable, no ConfigMap
and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this;
there isn't one, and the search has already been done.
What the device mount does **not** do, so nobody reads more into a green gate than is there: it does
not enable hardware transcoding, it does not prove the device can be *opened* from inside a
non-privileged pod (see "Likely causes" #2), and it renders no `nodeSelector` — Plex can land on any
of the three nodes and all three carry the device.
## Running the commands in this runbook
`kubectl` does not work on the operator workstation; the toolchain lives only in the pinned dev
container. Every command below runs as:
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
picks a current-context **per invocation**, and that default has already changed once. Do not run two
`dev.sh` invocations at once.
**Never `cat` `Preferences.xml`.** It contains `PlexOnlineToken`, a plex.tv account credential. Every
read of that file below greps a single named key — do not widen one.
## First commands
**1. Is the device mounted into the pod at all?** (Proves Flux applied the change, not that it works.)
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-o jsonpath="{.items[0].spec.containers[0].volumeMounts[*].mountPath}"; echo'
```
`/dev/dri` in that list ⇒ the GitOps half landed. Absent ⇒ the running pod predates the merge; compare
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 \
-o jsonpath="{.status.lastAppliedRevision}"; echo'
```
against the merge commit before blaming anything else.
**2. Is the device actually visible inside the container?**
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'
```
Expect `renderD128` (usually `card0` too), mode `crw-rw----`, group `render`. An **empty directory**
here is the `DirectoryOrCreate` degradation: the kubelet created an empty `/dev/dri` because the node
had none. That is the designed-safe outcome — the app tier still reconciles — but it means software
transcoding, silently. Find which node the pod is on and ask why that node lost its iGPU:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
```
**3. Can Plex *open* it?** This is the question the manifest cannot answer — see "Likely causes" #2.
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- \
bash -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
```
`RW-OPEN-DENIED` (or `Operation not permitted` / `Permission denied`) with the device plainly visible
in step 2 is the **device-cgroup** failure, not a file-permission failure. The two look identical from
the outside and are not the same problem.
**4. Is the toggle on?** Named key only — never the whole file:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- \
grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
"/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'
```
`ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on,
and the remaining question is whether it *works* — go to "How to verify".
## The one manual step
Plex web UI → **Settings → Transcoder → "Use hardware acceleration when available"** → tick → Save.
(It sits under *Show Advanced*.) A transcode already in flight is unaffected; the next one picks it up.
The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by
extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from
recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise
them and not invent them:
| Key | What it is |
|---|---|
| `HardwareAcceleratedCodecs` | the toggle this page is about |
| `HardwareAcceleratedEncoders` | hardware **encode**, as distinct from decode |
| `HardwareDevicePath` | how Plex is pointed at a *specific* render node — relevant only if a host ever exposes more than one |
| `TranscoderToneMapping`, `TranscoderToneMappingAgorithm` | HDR→SDR tone mapping. **That misspelling of "Algorithm" is upstream's, in the binary. It is not a typo in this page — do not "fix" it** here or anywhere else; the corrected spelling is not a key Plex reads. |
| `TranscoderHEVCEncoding` | HEVC output |
**`HardwareAcceleratedCodecsEnableHEVC` does not exist.** It is widely cited and it is not in the
binary. Recorded explicitly so nobody spends an evening hunting for it or, worse, writes it into
`Preferences.xml` and concludes hardware transcoding is broken.
Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn
tier, it is in no git commit, and the UI writes the key correctly. Use the UI.
## How to verify — and why there is no alert
Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a
lower quality explicitly), then read Plex's own **Dashboard → Activity**:
- **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.**
- plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says.
**That dashboard is the *only* proof available, and nothing else will ever tell you.**
`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work
to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently
inert, and this repo has already been burned at scale by rules that evaluated against nothing while
looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that
gap is ever worth closing, the honest order is a real exporter first and a rule second.
Corroborating, not a substitute — the server log on the config PVC:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- \
sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
\"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'
```
Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions.
Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence
of hardware. The dashboard remains the verdict.
## Likely causes, most likely first
When the dashboard still says plain `Transcode`:
1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists.
First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
investigating anything technical.
2. **containerd's device cgroup is refusing the `open()` — UNVERIFIED, and the largest remaining
technical risk in this feature.** Whether a non-privileged pod may `open()` a bind-mounted
character device on this k3s/containerd build **has not been established**; it can only be settled
by deploying and trying. The container inits as uid 0, which settles *file* permissions, but
**uid 0 does not bypass the cgroup device gate.** Symptom: step 2 shows `renderD128` present and
correctly owned, step 3 returns `RW-OPEN-DENIED`.
**The documented fallback is the Intel Device Plugin for Kubernetes** (`gpu.intel.com/i915` as a
resource request, no hostPath). Because the ServiceDefinition field expresses *intent* rather than
a device path, that migration is a **`catalog/render.py` change plus a re-render — with no
ServiceDefinition edit anywhere**; `catalog/plex.yaml` keeps saying exactly what it says today.
**`privileged: true` is not an available escape hatch:** Kyverno `restrict-host-namespaces` is in
**Enforce** mode and rejects it in app namespaces. Do not reach for it.
3. **The pod landed on a node without the device.** Step 2 shows an empty `/dev/dri`. All three prod
nodes carried `renderD128` with `i915` bound when this was measured (2026-09-30), so this means a
node changed — a rebuild, a BIOS reset, or a driver regression. `DirectoryOrCreate` is why the app
tier still came up; the cost is that the loss is silent. **Fix the node, do not tighten the mount:**
`Directory` would stall `apps-wave-0` (`wait: true`, with `apps-wave-1` depending on it) for a
feature that is not load-bearing.
4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
mechanism this feature leans on with no commit in this repo.
5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so
`Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as
a hardware failure.
6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which
Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
sessions show `(hw)` and others do not. Not a fault.
## The floating-image caveat
Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
user to the device's group, the absence of `PLEX_PREFERENCE*`, the six `setPref` keys, the
`HardwareAcceleratedCodecs` string — was read from digest **`sha256:e0ab27395614…`**
(`plexmediaserver 1.43.4.10903`).
The deployment runs **`plexinc/pms-docker:latest` with `imagePullPolicy: Always`**. A silent pull can
therefore change the behaviour this feature depends on **with no commit in this repo**, while the
manifest still looks identical. Symptom to watch for: hardware transcoding that worked and quietly
stopped, with no git change to point at.
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-o jsonpath="{.items[0].status.containerStatuses[0].imageID}"; echo'
```
If that digest is not the one above, **re-verify the hook before trusting this page**:
```
infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n plex exec deploy/plex -- ls /etc/cont-init.d/'
```
`45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one
described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups`
needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that
hook exists.
## When to escalate or stop
- **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be
**inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's
`cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its
group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces**
the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with
`Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
- **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
reject it. If the cgroup gate is the blocker, the answer is the Intel device plugin (cause #2).
- **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles
`main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating
`kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change
through a branch, a PR and a green gate.
- **Do not edit `Preferences.xml` on the PVC**, never `cat` it, and never `sops -d` anything into a
terminal. It holds `PlexOnlineToken`.
- **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and
watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
lose the DB.
- **Stop and get the operator** if the device cannot be opened from the pod (cause #2 confirmed). The
next step is a platform change — the Intel device plugin — with its own PR, not more poking at this
workload.
## Known gaps, deliberately not fixed here
Recorded so they are findable, and so nobody assumes they were overlooked:
- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
`TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
transcode scratch performance changes silently whenever the pod is rescheduled. Both candidate fixes
need a **schema** change and are therefore out of scope here: pin Plex to prod-2 (needs a
`nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory` emptyDir and
transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not have). The RAM option
is the better one, because it removes the node asymmetry outright instead of working around it.
- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
quality only — decode, and the feature on this page, are unaffected. Changing it needs a kernel
parameter and a **serialized rolling reboot**, which is an operator action (CLAUDE.md §5), and this
cluster has a documented history of that going badly (a prod node hung on a warm reboot on
2026-09-03). Not worth coupling to this change.
---
Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
The node measurements (all three prod nodes) and the in-container image measurements cited here were
taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
(`plexmediaserver 1.43.4.10903`). Hardware transcoding had **not** been observed working at the time
of writing, and the Plex-side toggle was still unset.