# plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did **Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**, not an incident page. It is indexed here because the feature it completes is **inert until a human clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists to prevent. **Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the rendered golden — the `resources.limits: gpu.intel.com/i915: 1` block), `catalog/render.py` (the mechanism), `schemas/renderer-contract.md` `### hardware` (the contract), `platform/intel-gpu-plugin/` (the DaemonSet that advertises the resource). **Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile. ## STOP HERE IF PLEX IS `Pending` / `Unschedulable` That is this feature's **one serious failure mode** and it **stalls the whole app tier** (`apps-wave-0` is `wait: true`; `apps-wave-1` `dependsOn` it). Since 2026-10-01 Plex requests the extended resource `gpu.intel.com/i915: 1`, which is a **hard scheduling constraint** — if the `intel-gpu-plugin` DaemonSet is not advertising it on at least one schedulable node, Plex does not start at all. Go to **"Plex will not schedule"** below; it is the first troubleshooting section on purpose. **The escape hatch, in one sentence:** delete the two `hardware:` lines from `catalog/plex.yaml`, re-render, and merge — Plex returns to software transcoding and the tier unblocks. Exact commands are in **"Plex will not schedule" → remedy B**. ## What this means here Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync moves that work off the CPU — which matters on a three-node cluster where the CPU is also running everything else. Two independent things have to be true before that happens, and **only one of them is in git**: | Half | Who does it | State | |---|---|---| | The render device is inside the container **and openable** | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders `resources.limits: {gpu.intel.com/i915: 1}` on the app container. The `intel-gpu-plugin` DaemonSet (`platform/intel-gpu-plugin/`) injects the device **and the matching device-cgroup allow rule**. Flux applies it. | Done by merging that change. | | Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. | **There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`, `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin` — found **zero** `PLEX_PREFERENCE*` support. Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentifier`, `ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`), none of them the transcode toggle. No environment variable, no ConfigMap and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this; there isn't one, and the search has already been done. ### Why a resource request and not a `/dev/dri` hostPath Because the hostPath was **tried and measured not to work**, and knowing that is the only thing that will stop someone "simplifying" it back. Merged 2026-09-30, measured in the live pod on k3s-prod-2 on **2026-10-01**: the device node was plainly **visible** in the container (`crw-rw---- root video2 226,128 renderD128`) and **could not be opened** — | attempt | result | |---|---| | `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" | | `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" | `EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions. The two look identical from the outside and are not the same problem. Proven three ways: root on the **host** opened the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc `config.json` read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with `linux.devices: None` — **one deny-all rule, no allow entry for major 226**. A hostPath is a *filesystem* operation: it binds the device node in and adds **no cgroup allow rule**, because Kubernetes cannot infer one from a volume. Two non-workarounds, so nobody spends an evening on them: **`uid 0` failed too**, so running as root does not bypass the cgroup gate; and **`privileged: true` is not available** — Kyverno `restrict-host-namespaces` is **Enforce** on app namespaces and rejects it. The device plugin is what produces the allow rule — in the v0.37.0 source, `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns into both the device injection and the matching cgroup allow. **The failure mode that trade bought, stated plainly.** Before, a missing device meant the pod ran and **silently software-transcoded** — invisible, but harmless to everything else. Now, a missing *resource* means Plex is **Unschedulable** and the app tier stalls. That is strictly worse in blast radius and it is **unavoidable**: an extended resource is a hard scheduling constraint, and it is also the only thing that produces the cgroup allow rule. There is no graceful-degradation variant. The one improvement is that the loss is no longer silent. What this does **not** do, so nobody reads more into a green gate than is there: it does not enable hardware transcoding (that is the manual toggle below), a green gate only proves the **manifest renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin advertises, which is the scheduler's doing and not a selector in this repo. ## Running the commands in this runbook `kubectl` does not work on the operator workstation; the toolchain lives only in the pinned dev container. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and picks a current-context **per invocation**, and that default has already changed once. Do not run two `dev.sh` invocations at once. **Never `cat` `Preferences.xml`.** It contains `PlexOnlineToken`, a plex.tv account credential. Every read of that file below greps a single named key — do not widen one. ## First commands **0. Is Plex running at all?** Ask this first now — the current mechanism can stop it starting. ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide' ``` `Pending` ⇒ go straight to **"Plex will not schedule"**. Running ⇒ carry on. **1. Do the nodes advertise the resource, and is the plugin Ready?** This replaces the old "is the hostPath mounted" check, and it is now the first thing that can be wrong. ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get nodes -o json \ | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""' ``` Expect `i915=1` on each of the three prod nodes (measured 2026-10-01: all three). Any `ABSENT` ⇒ that node cannot run Plex. Then the DaemonSet behind it: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n intel-gpu-plugin get daemonset intel-gpu-plugin' ``` Expect **3 desired / 3 ready / 3 available** (measured 2026-10-01). Anything less is the cause, not a symptom — fix it there, in `platform/intel-gpu-plugin/`, not here. **2. Did Flux apply the current manifest?** `gpu.intel.com/i915` on the app container means the GitOps half landed: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \ -o jsonpath="{.items[0].spec.containers[0].resources.limits}"; echo' ``` A `/dev/dri` **volumeMount** instead ⇒ the running pod predates the 2026-10-01 merge and is on the mechanism that does not work. Compare ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 \ -o jsonpath="{.status.lastAppliedRevision}"; echo' ``` against the merge commit before blaming anything else. **3. Is the device visible inside the container, and can Plex *open* it?** This is the decisive test of the whole mechanism, and **it must be run as uid 1000**, because that is what Plex runs as. ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri' ``` Expect `renderD128`, mode `crw-rw----`. The plugin injects it; there is no longer any `DirectoryOrCreate` empty-directory case — if the resource were missing the pod would not be running to exec into. Then the open, **as the app user**: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n plex exec deploy/plex -- \ s6-setuidgid plex sh -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"' ``` `RW-OPEN-OK` **as uid 1000** is the pass condition. **Passing as `uid 0` is weaker evidence and must not be accepted as proof** — Plex runs as 1000, and the whole point of the 2026-10-01 change was a cgroup rule that uid 0 could not buy its way past either. `RW-OPEN-DENIED` with the device plainly visible is the **device-cgroup** failure — i.e. the plugin's allow rule did not arrive; see "Likely causes" #2. **4. Is the toggle on?** Named key only — never the whole file: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n plex exec deploy/plex -- \ grep -o "HardwareAcceleratedCodecs=\"[01]\"" \ "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT' ``` `ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on, and the remaining question is whether it *works* — go to "How to verify". ## The one manual step Plex web UI → **Settings → Transcoder → "Use hardware acceleration when available"** → tick → Save. (It sits under *Show Advanced*.) A transcode already in flight is unaffected; the next one picks it up. The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise them and not invent them: | Key | What it is | |---|---| | `HardwareAcceleratedCodecs` | the toggle this page is about | | `HardwareAcceleratedEncoders` | hardware **encode**, as distinct from decode | | `HardwareDevicePath` | how Plex is pointed at a *specific* render node — relevant only if a host ever exposes more than one | | `TranscoderToneMapping`, `TranscoderToneMappingAgorithm` | HDR→SDR tone mapping. **That misspelling of "Algorithm" is upstream's, in the binary. It is not a typo in this page — do not "fix" it** here or anywhere else; the corrected spelling is not a key Plex reads. | | `TranscoderHEVCEncoding` | HEVC output | **`HardwareAcceleratedCodecsEnableHEVC` does not exist.** It is widely cited and it is not in the binary. Recorded explicitly so nobody spends an evening hunting for it or, worse, writes it into `Preferences.xml` and concludes hardware transcoding is broken. Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn tier, it is in no git commit, and the UI writes the key correctly. Use the UI. ## How to verify — and why there is no alert Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a lower quality explicitly), then read Plex's own **Dashboard → Activity**: - **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.** - plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says. **That dashboard is the *only* proof available, and nothing else will ever tell you.** `catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** — there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently inert, and this repo has already been burned at scale by rules that evaluated against nothing while looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that gap is ever worth closing, the honest order is a real exporter first and a rule second. Corroborating, not a substitute — the server log on the config PVC: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n plex exec deploy/plex -- \ sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \ \"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""' ``` Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions. Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence of hardware. The dashboard remains the verdict. ## Likely causes, most likely first When the dashboard still says plain `Transcode`: 1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists. First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before** investigating anything technical. 2. **The device cannot be `open()`ed — the plugin's cgroup allow rule did not arrive.** Symptom: first-command #3 shows `renderD128` present and correctly owned, and the open returns `RW-OPEN-DENIED` / `Operation not permitted`. This is `EPERM` from the **device cgroup**, not `EACCES` from file permissions. **This is no longer a design risk — it is the measured reason the design is what it is** (2026-10-01; see "Why a resource request and not a `/dev/dri` hostPath"). If it recurs *under the current mechanism*, the plugin is the suspect, not the manifest: confirm the pod really did consume the resource ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \ -o jsonpath="{.items[0].spec.containers[0].resources}"; echo' ``` and then check the plugin pod on Plex's node (first-command #1). **Do not reach for `privileged: true`** — Kyverno `restrict-host-namespaces` is Enforce and rejects it, and uid 0 failed the same open in 2026-09-30's measurement anyway. **Do not re-add a `/dev/dri` hostPath**; that is the mechanism this replaced, and it was measured not to work. 3. **The pod landed on a node whose GPU went away.** Under the current mechanism this does **not** present as an empty `/dev/dri` — the node simply stops advertising `gpu.intel.com/i915` and Plex either never schedules there or gets evicted off. So this cause now shows up as **`Pending`**, not as silent software transcoding: see "Plex will not schedule". All three prod nodes carried `renderD128` with `i915` bound when measured (2026-09-30) and all three advertised the resource on 2026-10-01, so this means a node changed — a rebuild, a BIOS reset, or a driver regression. **Fix the node.** 4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the mechanism this feature leans on with no commit in this repo. 5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so `Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as a hardware failure. 6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some sessions show `(hw)` and others do not. Not a fault. ## Plex will not schedule **Symptom:** the Plex pod sits `Pending`, and `apps-wave-0` does not go Ready, which holds `apps-wave-1` behind it. This is the cost of the 2026-10-01 mechanism and the section you want at 2am. ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n plex describe pod -l app.kubernetes.io/name=plex | tail -20' ``` `0/3 nodes are available: 3 Insufficient gpu.intel.com/i915` (or similar) confirms it: **nothing is advertising the resource Plex requires**. Plex's schedulability is coupled to the `intel-gpu-plugin` DaemonSet, deliberately and unavoidably — the plugin is also the only thing that produces the device-cgroup allow rule, so there is no variant of this feature that degrades gracefully. **Remedy A — fix the plugin (preferred).** Run first-command #1. If the DaemonSet is not 3/3, or the nodes show `i915=ABSENT`, the fault is in `platform/intel-gpu-plugin/` or on the node, and that is where to fix it. Check its Flux Kustomization reconciled: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n flux-system get kustomization intel-gpu-plugin' ``` A plugin that is merely **not yet reconciled** resolves itself; one that is failing does not. **Remedy B — the escape hatch: give up hardware transcoding and unblock the tier.** One field, no code change. In a branch (never on `main`, never `kubectl edit` — Flux reconciles `main` and would revert you): 1. delete these two lines from `catalog/plex.yaml`: ```yaml hardware: gpu: intel-quicksync ``` 2. re-render the golden: ``` python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation ``` The only change must be `apps/wave-0/plex.yaml`, losing its `resources` block. 3. gate, PR, merge. Plex schedules anywhere and **software-transcodes**; the app tier unblocks. Re-adding the field later is the same edit in reverse. Nothing about the Plex-side toggle or the `/config` PVC is affected either way — do **not** delete the PVC as part of this. ## The floating-image caveat Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app user to the device's group, the absence of `PLEX_PREFERENCE*`, the six `setPref` keys, the `HardwareAcceleratedCodecs` string — was read from digest **`sha256:e0ab27395614…`** (`plexmediaserver 1.43.4.10903`). The deployment runs **`plexinc/pms-docker:latest` with `imagePullPolicy: Always`**. A silent pull can therefore change the behaviour this feature depends on **with no commit in this repo**, while the manifest still looks identical. Symptom to watch for: hardware transcoding that worked and quietly stopped, with no git change to point at. ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \ -o jsonpath="{.items[0].status.containerStatuses[0].imageID}"; echo' ``` If that digest is not the one above, **re-verify the hook before trusting this page**: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n plex exec deploy/plex -- ls /etc/cont-init.d/' ``` `45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups` needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that hook exists. ## When to escalate or stop - **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be **inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's `cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces** the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is discarded before Plex starts. `render.py --selftest` asserts that absence deliberately. - **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will reject it — and it would not help: `uid 0` failed the same `open()` in the 2026-09-30 measurement, because root does not bypass the cgroup device gate. - **Do not re-add a `/dev/dri` hostPath, and do not "simplify" the resource request into one.** That was the original mechanism and it was measured non-functional on 2026-10-01 (device-cgroup `EPERM` with the device plainly visible). The device plugin's `DeviceSpec{… Permissions: "rw"}` is what grants access; a volume cannot. - **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles `main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating `kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change through a branch, a PR and a green gate. - **Do not edit `Preferences.xml` on the PVC**, never `cat` it, and never `sops -d` anything into a terminal. It holds `PlexOnlineToken`. - **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to lose the DB. - **Stop and get the operator** if the device cannot be opened from the pod as uid 1000 (cause #2 confirmed) *under the current mechanism*. The next step is a platform change in `platform/intel-gpu-plugin/`, with its own PR, not more poking at this workload. ## Known gaps, deliberately not fixed here Recorded so they are findable, and so nobody assumes they were overlooked: - **Transcode scratch is on the wrong medium, and it is invisible.** The image sets `TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so transcode scratch performance changes silently whenever the pod is rescheduled. Both candidate fixes need a **schema** change and are therefore out of scope here: pin Plex to prod-2 (needs a `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory` emptyDir and transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not have). The RAM option is the better one, because it removes the node asymmetry outright instead of working around it. - **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode** quality only — decode, and the feature on this page, are unaffected. Changing it needs a kernel parameter and a **serialized rolling reboot**, which is an operator action (CLAUDE.md §5), and this cluster has a documented history of that going badly (a prod node hung on a warm reboot on 2026-09-03). Not worth coupling to this change. --- Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/` The node measurements (all three prod nodes) and the in-container image measurements cited here were taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`). The `EPERM` open measurements, the `Groups: 44 100 992 1000` reading and the `gpu.intel.com/i915: 1` allocatable/DaemonSet readings were taken **2026-10-01**. Hardware transcoding has **still not been observed working**: the open test has not been re-run under the device-plugin mechanism (it cannot be, until that merges and Flux reconciles), and the Plex-side toggle is still unset.