From 0fee584e3ad30d54ab37a7a8d9f2039d900a92e8 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Wed, 30 Sep 2026 13:09:20 -0500 Subject: [PATCH] feat(plex): opt into `hardware.gpu: intel-quicksync`, and write the runbook for the step GitOps cannot reach MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plex has been transcoding in software on hardware that can do it in the iGPU. This opts its ServiceDefinition into the `hardware` capability from #27 and re-renders. catalog/plex.yaml gains four lines of YAML and the comment that explains them: hardware: gpu: intel-quicksync which renders exactly two things into apps/wave-0/plex.yaml and nothing else — a `dri` hostPath volume (/dev/dri, type: DirectoryOrCreate) and a read-write /dev/dri volumeMount on the app container. Whole-tree render diff: 6 added lines, one file. The comment records what is not obvious from the four lines: the device exists on ALL THREE prod nodes with i915 bound (so no nodeSelector is rendered and none is wanted); why `supplementalGroups` would be INERT rather than a hardening (the image's cont-init hook adds the plex user to the device's group inside the container, and s6-setuidgid then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied); and why `DirectoryOrCreate` is load-bearing (apps-wave-0 is wait:true with apps-wave-1 depending on it, so a failed mount would stall the whole app tier for a feature that is not). THE FEATURE IS INERT UNTIL A HUMAN CLICKS A BOX, and that is the part this commit is really about. Mounting the device does not enable hardware transcoding; Plex's `HardwareAcceleratedCodecs` preference does, and it is ABSENT from Preferences.xml — the toggle has never been on. It cannot be automated: a bounded grep across the running image's entire init surface (/etc/cont-init.d, /etc/services.d, /etc/fix-attrs.d, /plex-common.sh, /usr/local/bin) found ZERO PLEX_PREFERENCE* support, and the six setPref keys the image does write are all unrelated. So runbooks/plex-quicksync.md is the procedure, registered in the configMapGenerator (a runbook in git with no generator entry is reviewed, merged, and still a 404) and indexed in runbooks/README.md. It covers the GitOps/manual boundary, the toggle and the real preference keys (including the widely-cited one that does not exist and the upstream misspelling nobody should "fix"), verification via `Transcode (hw)` on the Plex dashboard — stated plainly as the ONLY proof, because Plex renders no ServiceMonitor and an alert could never fire — the device-cgroup failure to expect first with its Intel-device-plugin fallback, and the floating-`:latest` caveat. Stacked on #27 (feat/sd-hardware-capability), which is open and unmerged; that branch's commit is in this range by design and this must merge after it. NOT VERIFIED, and the handoff says so at length: hardware transcoding has not been observed working. Whether containerd's device cgroup lets a non-privileged pod open() a bind-mounted char device is an open question that only deploying settles, and the Plex-side toggle was still unset at the time of writing. A green gate proves the manifest renders; it proves nothing about transcoding. Gate (pinned toolbox, worktree): rc 3 — UNDETERMINED, not green. Passed: decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped (the known local ceiling): spec-immutability (no sibling spec repo), gitea-settings (no SETTINGS_API_TOKEN), conformance (CLUSTER_AVAILABLE unset, so L2/L3 did not run). No fourth skip, nothing failed. Co-Authored-By: Claude Opus 5 (1M context) --- apps/wave-0/plex.yaml | 6 + catalog/plex.yaml | 52 +++++ decisions/2026-09-30-plex-quicksync.md | 149 +++++++++++++ runbooks/README.md | 8 + runbooks/kustomization.yaml | 6 + runbooks/plex-quicksync.md | 283 +++++++++++++++++++++++++ 6 files changed, 504 insertions(+) create mode 100644 decisions/2026-09-30-plex-quicksync.md create mode 100644 runbooks/plex-quicksync.md diff --git a/apps/wave-0/plex.yaml b/apps/wave-0/plex.yaml index 5c3b21f..2928eaa 100644 --- a/apps/wave-0/plex.yaml +++ b/apps/wave-0/plex.yaml @@ -143,6 +143,8 @@ spec: - name: media mountPath: /media readOnly: true + - name: dri + mountPath: /dev/dri volumes: - name: config persistentVolumeClaim: @@ -150,6 +152,10 @@ spec: - name: media persistentVolumeClaim: claimName: plex-media + - name: dri + hostPath: + path: /dev/dri + type: DirectoryOrCreate --- # Service apiVersion: v1 diff --git a/catalog/plex.yaml b/catalog/plex.yaml index caf51b1..0b14b29 100644 --- a/catalog/plex.yaml +++ b/catalog/plex.yaml @@ -80,6 +80,58 @@ env: # remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443". +# HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02). +# INTENT ONLY. This says "this workload needs an Intel Quick Sync render device"; it does NOT name a +# device path. The renderer currently satisfies it with exactly two things — a `/dev/dri` hostPath +# volume (`type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container — +# and nothing else: no nodeSelector, no privileged, no resources (schemas/renderer-contract.md +# `### hardware`). If that mechanism is later replaced by the Intel GPU device plugin +# (`gpu.intel.com/i915`), it is a `catalog/render.py` change and THIS BLOCK DOES NOT MOVE. Do not +# "simplify" it into a raw `/dev/...` path; that is the whole point of the enum. +# +# THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land +# anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`, +# GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines +# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Plex happens to run on k3s-prod-2 today, but +# that is scheduler history rather than a requirement — this field deliberately renders no +# nodeSelector, because placement is a separate concern with its own justification. +# +# NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this. +# The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push +# the host `render` GID in as a supplemental group. It cannot work here. MEASURED IN THE RUNNING +# CONTAINER, 2026-09-30: the image's own +# `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device under +# `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and +# `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host +# GID renumbering self-healing (no GID is pinned anywhere in this repo). The service then execs via +# `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from `/etc/group` rather +# than appending to it: `Plex Media Server` runs with `Groups: 100 1000`, and root's gid `0` — which +# PID 1 (`s6-svscan`) does hold — is GONE. Anything Kubernetes supplies therefore reaches s6-svscan +# and is discarded before Plex ever starts. Mounting the device is sufficient; the group is the +# image's job and it does it correctly. `render.py --selftest` asserts the ABSENCE of a pod +# securityContext for this field precisely so nobody adds an inert one back later. +# +# `DirectoryOrCreate` IS THE SAFE FAILURE MODE, not laziness. A node that comes back without the +# device (rebuild, BIOS reset, driver regression, hardware swap) gets an empty directory and Plex +# degrades to SOFTWARE transcoding. `Directory` would fail the mount instead — and because +# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that would stall the WHOLE app +# tier for a feature that is not load-bearing. The price, stated plainly: a missing device is +# SILENT. This SD renders no `metricsEndpoint`, so there is no ServiceMonitor, no transcode-mode +# metric, and no alert that could ever fire. The only proof is an operator reading the Plex +# dashboard. +# +# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Mounting the device does +# not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it +# is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can +# set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`, +# `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin`) found ZERO `PLEX_PREFERENCE*` support, +# and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`, +# `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all +# unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI. +# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took and the +# device-cgroup failure mode to expect first. +hardware: + gpu: intel-quicksync exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET) egress: internet # normal WAN egress (docs/03; R-NET-05) storage: diff --git a/decisions/2026-09-30-plex-quicksync.md b/decisions/2026-09-30-plex-quicksync.md new file mode 100644 index 0000000..4e9966f --- /dev/null +++ b/decisions/2026-09-30-plex-quicksync.md @@ -0,0 +1,149 @@ +--- +date: 2026-09-30 +branch: feat/plex-quicksync +title: Plex opts into `hardware.gpu: intel-quicksync` — the device mount is GitOps, the Plex-side toggle is a documented manual step, and hardware transcoding has NOT been observed working +--- + +## What changed + +Three things, and the third is the point of the branch. + +1. **`catalog/plex.yaml`** — added the four lines the whole feature reduces to: + + ```yaml + hardware: + gpu: intel-quicksync + ``` + + placed between `env` and `exposure` to match the schema's own property order, with ~48 lines of + comment in the file's established style. The comment records: the field is **intent, not + mechanism** (a later move to the Intel device plugin is a `render.py` change and this block does + not move); that `/dev/dri/renderD128` is present on **all three** prod nodes with `i915` bound; + **why a `supplementalGroups` entry would be inert** (the image's cont-init hook does the group + work inside the container and `s6-setuidgid` replaces the process group list from `/etc/group`); + why `DirectoryOrCreate` is the load-bearing safe failure mode (`apps-wave-0` is `wait: true` with + `apps-wave-1` depending on it, so a failed mount would stall the whole app tier for a feature + that is not load-bearing); and that the Plex-side toggle is a **separate manual operator step**, + pointing at the new runbook. + +2. **`apps/wave-0/plex.yaml`** — re-rendered. Six added lines, nothing else, in the whole `apps/` + tree: + + ``` + - name: dri + mountPath: /dev/dri + - name: dri + hostPath: + path: /dev/dri + type: DirectoryOrCreate + ``` + +3. **`runbooks/plex-quicksync.md`** (new), registered in `runbooks/kustomization.yaml`'s + `configMapGenerator` as `- plex-quicksync=plex-quicksync.md` and indexed in `runbooks/README.md` + under "Operator procedures". **The file alone would have been a 404** — the pod serves a + ConfigMap, not the git tree, and `conformance/tests/l0-static/alert-runbooks-resolvable` exists + precisely because "the file is in git" and "the URL resolves" are different claims. The page + covers: the GitOps/manual boundary as a table; the toggle (Settings → Transcoder → "Use hardware + acceleration when available", preference key `HardwareAcceleratedCodecs`); the real neighbouring + keys and the widely-cited one that **does not exist**; that the only proof is `Transcode (hw)` on + the Plex dashboard **and why no alert is possible**; the device-cgroup failure to expect first + with a concrete open() diagnostic and the Intel-device-plugin fallback; and the floating-`:latest` + caveat with a digest check. + +**Stacked on PR #27 (`feat/sd-hardware-capability`), which is open and unmerged.** This branch was +cut from it rather than from `main`, so its one commit (`d9baa2a`, the schema + renderer + contract) +is in this PR's range. That is deliberate: merging #27 is not an agent's action (CLAUDE.md §1), and +duplicating its changes here would have been worse than showing them. **This must merge after #27.** + +Not done, deliberately: no `--emit-waves` (it writes into `clusters/base/`, a sensitive path that +would force a human reviewer, and it is a no-op unless the *set* of files under `apps/` changes — it +does not here). No `podSecurityContext`. No `nodeSelector`. No alert rule. No `kubectl apply`. + +### Two follow-ups left unclaimed, with the reasoning + +- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets + `TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable + overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so + transcode scratch performance changes silently on reschedule, with nothing reporting it. Both + candidate fixes need a **schema** change and so neither belongs in this PR: pin Plex to prod-2 + (needs a `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory` + emptyDir and transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not + have). **The RAM option is the better one** — it removes the node asymmetry outright instead of + working around it — which is exactly why it deserves its own PR and its own review rather than + being smuggled in behind a four-line catalog change. Documented in the runbook's "Known gaps". +- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode** + quality only; decode and this feature are unaffected. Changing it needs a kernel parameter plus a + **serialized rolling reboot** — an operator action under CLAUDE.md §5, on a cluster with a + documented history of that going badly (a prod node hung on a warm reboot on 2026-09-03). Kept + well clear of this PR. + +## What I verified + +- **The render is exactly the intended change, and nothing else moved.** After + `python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` (exit 0, + silent): `git diff --stat apps/` = `apps/wave-0/plex.yaml | 6 ++++++`, and + `git status --porcelain apps/` = a single ` M apps/wave-0/plex.yaml`. Read line by line, the diff + is the `dri` volumeMount and the `dri` hostPath volume, and nothing more. **Specifically checked:** + the same invocation also rewrites `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, which + are git-tracked but excluded from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in + `ci/render-golden.sh` — a spurious diff there would have landed **ungated**. Neither file shows a + diff; whole-repo `git status --porcelain` lists only `apps/wave-0/plex.yaml` and `catalog/plex.yaml` + plus my new files. +- **The rendered output matches what the contract promises** — `schemas/renderer-contract.md` + `### hardware` says exactly two things (a `/dev/dri` hostPath with `type: DirectoryOrCreate` and a + read-write mount on the app container) and exactly two things appeared. No `securityContext`, no + `nodeSelector`, no resources in the diff. +- **The runbook is projected into the server, not merely committed.** `runbooks/kustomization.yaml` + carries `- plex-quicksync=plex-quicksync.md`, whose shape matches the generator-entry regex the + L0 check parses; the extensionless key is what makes `/runbooks/plex-quicksync` a plain static + lookup. Also indexed in `runbooks/README.md`. +- **The runbook clears the L0 quality floor** that `alert-runbooks-resolvable` applies to + alert-linked pages, even though nothing links to this one (it is advisory for an unreferenced + page): all four required sections (`## What this means here`, `## First commands`, + `## Likely causes, most likely first`, `## When to escalate or stop`), well over the 1500-byte + floor, `dev.sh` used throughout, and `--context k3s-prod` on every command. +- **The gate.** See the run recorded in the PR description for the per-stage table and the real rc. +- **No secret was printed.** Every `Preferences.xml` read in the runbook greps one named key, and the + page states explicitly why: the file holds `PlexOnlineToken`. Nothing was `sops -d`'d. + +## What I did NOT verify + +**Hardware transcoding has NOT been observed working.** Nothing in this branch is evidence that it +does. A green gate here proves the manifest renders and the golden matches — it proves nothing about +runtime behaviour, and `vars.CLUSTER_AVAILABLE` is unset so L2/L3 conformance did not run at all. + +- **The device-cgroup question is OPEN, and it is the largest remaining technical risk.** Whether + containerd on this k3s build permits a **non-privileged** pod to `open()` a bind-mounted character + device is **UNVERIFIED**. It can only be settled by deploying and trying. The container inits as + uid 0, which settles *file* permissions, but uid 0 does **not** bypass the cgroup device gate — and + the two failures look identical from the outside. If it is refused, the fallback is the Intel + Device Plugin (`gpu.intel.com/i915`), which is a `catalog/render.py` change plus a re-render with + **no ServiceDefinition edit**; `privileged: true` is not available (Kyverno + `restrict-host-namespaces` is Enforce). The runbook has the diagnostic sequence. +- **The Plex-side toggle was still unset at the time of writing.** `HardwareAcceleratedCodecs` was + **absent** from `Preferences.xml` — the toggle has never been enabled on this server. Until an + operator ticks it in the UI, this branch changes the pod spec and **nothing about transcoding**. + That is the whole reason the runbook exists. +- **No metric, so no alert, so no automated detection — permanently.** `catalog/plex.yaml` declares + no `metricsEndpoint`, so Plex renders no ServiceMonitor; there is no transcode-mode signal for a + rule to evaluate. A silent degradation to software transcoding (a node losing its iGPU, the + `DirectoryOrCreate` path) will **not** be reported by anything. I deliberately did not add an + alert: with no metric behind it, it could never fire, and this repo has been burned at scale by + exactly that. Closing the gap honestly means an exporter first. +- **I did not re-derive this session's node or in-container measurements.** The three-node + `renderD128`/`i915`/GID-992 facts, the cont-init hook behaviour, the `s6-setuidgid` group + replacement, the `PLEX_PREFERENCE*` absence, the six `setPref` keys and the + `HardwareAcceleratedCodecs` string were established earlier in this session and are cited, not + re-measured, by me. Per CLAUDE.md §2b, treat this file as a record of belief and re-measure against + the cluster before relying on any of it. +- **The image is a floating tag.** Everything known about the hook came from digest + `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), while the Deployment runs + `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. A silent pull can invalidate the + "no `supplementalGroups` needed" conclusion **with no commit in this repo**. Not guarded against — + the runbook tells the reader to re-check the hook after an image bump, which is documentation, not + enforcement. +- **I did not verify the live Gitea gate state** (branch protection, required contexts, runner + online) for this PR. Per §2b that must be queried, not read out of `CLAUDE.md`'s table; I did not + query it. +- **`/runbooks/plex-quicksync` was not fetched.** The L0 check asserts every static precondition for + an HTTP 200; it does not make the request, and I did not either. diff --git a/runbooks/README.md b/runbooks/README.md index 2368296..f5b7a93 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -115,6 +115,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent. plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`: token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured). Includes why a changed Secret does not restart anything, and why an expired token is harmless. +- **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod + (`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware + acceleration when available"**, and no environment variable can do it (the image supports no + `PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`, + read out of the binary), says which widely-cited key **does not exist**, and is explicit that the + **only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no + alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin + fallback, and why a `:latest` pull can silently break it. - **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log (the four lines that matter, including the upstream `-` that means the proxy answered a request the diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index 4e341bf..e37f2da 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -95,6 +95,12 @@ configMapGenerator: # because the plex.tv claim window is short and is not a thing to go looking # for in git: /runbooks/plex-claim has to resolve the moment it is needed. - plex-claim=plex-claim.md + # Operator procedure, not an alert page, and NO alert is possible for it: plex renders no + # metricsEndpoint, so there is no ServiceMonitor, no transcode-mode metric and nothing a rule + # could evaluate. Served because the feature it completes is INERT until a human ticks a box in + # the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped + # feature stays switched off for months. + - plex-quicksync=plex-quicksync.md - pvc-capacity=pvc-capacity.md - targets-down=targets-down.md - update-rollback=update-rollback.md diff --git a/runbooks/plex-quicksync.md b/runbooks/plex-quicksync.md new file mode 100644 index 0000000..4ccd4d5 --- /dev/null +++ b/runbooks/plex-quicksync.md @@ -0,0 +1,283 @@ +# plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did + +**Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**, +not an incident page. It is indexed here because the feature it completes is **inert until a human +clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists +to prevent. +**Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the +rendered golden — the `dri` hostPath volume and the `/dev/dri` mount), `catalog/render.py` (the +mechanism), `schemas/renderer-contract.md` `### hardware` (the contract). +**Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile. + +## What this means here + +Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync +moves that work off the CPU — which matters on a three-node cluster where the CPU is also running +everything else. Two independent things have to be true before that happens, and **only one of them +is in git**: + +| Half | Who does it | State | +|---|---|---| +| The render device is inside the container at `/dev/dri` | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders a `dri` hostPath volume (`/dev/dri`, `type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container. Flux applies it. | Done by merging that change. | +| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. | + +**There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded +grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`, +`/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin` — found **zero** `PLEX_PREFERENCE*` support. +Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentifier`, +`ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`, +`TranscoderTempDirectory`), none of them the transcode toggle. No environment variable, no ConfigMap +and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this; +there isn't one, and the search has already been done. + +What the device mount does **not** do, so nobody reads more into a green gate than is there: it does +not enable hardware transcoding, it does not prove the device can be *opened* from inside a +non-privileged pod (see "Likely causes" #2), and it renders no `nodeSelector` — Plex can land on any +of the three nodes and all three carry the device. + +## Running the commands in this runbook + +`kubectl` does not work on the operator workstation; the toolchain lives only in the pinned dev +container. Every command below runs as: + + infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' + +Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and +picks a current-context **per invocation**, and that default has already changed once. Do not run two +`dev.sh` invocations at once. + +**Never `cat` `Preferences.xml`.** It contains `PlexOnlineToken`, a plex.tv account credential. Every +read of that file below greps a single named key — do not widen one. + +## First commands + +**1. Is the device mounted into the pod at all?** (Proves Flux applied the change, not that it works.) + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \ + -o jsonpath="{.items[0].spec.containers[0].volumeMounts[*].mountPath}"; echo' +``` + +`/dev/dri` in that list ⇒ the GitOps half landed. Absent ⇒ the running pod predates the merge; compare + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 \ + -o jsonpath="{.status.lastAppliedRevision}"; echo' +``` + +against the merge commit before blaming anything else. + +**2. Is the device actually visible inside the container?** + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri' +``` + +Expect `renderD128` (usually `card0` too), mode `crw-rw----`, group `render`. An **empty directory** +here is the `DirectoryOrCreate` degradation: the kubelet created an empty `/dev/dri` because the node +had none. That is the designed-safe outcome — the app tier still reconciles — but it means software +transcoding, silently. Find which node the pod is on and ask why that node lost its iGPU: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide' +``` + +**3. Can Plex *open* it?** This is the question the manifest cannot answer — see "Likely causes" #2. + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex exec deploy/plex -- \ + bash -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"' +``` + +`RW-OPEN-DENIED` (or `Operation not permitted` / `Permission denied`) with the device plainly visible +in step 2 is the **device-cgroup** failure, not a file-permission failure. The two look identical from +the outside and are not the same problem. + +**4. Is the toggle on?** Named key only — never the whole file: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex exec deploy/plex -- \ + grep -o "HardwareAcceleratedCodecs=\"[01]\"" \ + "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT' +``` + +`ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on, +and the remaining question is whether it *works* — go to "How to verify". + +## The one manual step + +Plex web UI → **Settings → Transcoder → "Use hardware acceleration when available"** → tick → Save. +(It sits under *Show Advanced*.) A transcode already in flight is unaffected; the next one picks it up. + +The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by +extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from +recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise +them and not invent them: + +| Key | What it is | +|---|---| +| `HardwareAcceleratedCodecs` | the toggle this page is about | +| `HardwareAcceleratedEncoders` | hardware **encode**, as distinct from decode | +| `HardwareDevicePath` | how Plex is pointed at a *specific* render node — relevant only if a host ever exposes more than one | +| `TranscoderToneMapping`, `TranscoderToneMappingAgorithm` | HDR→SDR tone mapping. **That misspelling of "Algorithm" is upstream's, in the binary. It is not a typo in this page — do not "fix" it** here or anywhere else; the corrected spelling is not a key Plex reads. | +| `TranscoderHEVCEncoding` | HEVC output | + +**`HardwareAcceleratedCodecsEnableHEVC` does not exist.** It is widely cited and it is not in the +binary. Recorded explicitly so nobody spends an evening hunting for it or, worse, writes it into +`Preferences.xml` and concludes hardware transcoding is broken. + +Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn +tier, it is in no git commit, and the UI writes the key correctly. Use the UI. + +## How to verify — and why there is no alert + +Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a +lower quality explicitly), then read Plex's own **Dashboard → Activity**: + +- **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.** +- plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says. + +**That dashboard is the *only* proof available, and nothing else will ever tell you.** +`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** — +there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work +to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently +inert, and this repo has already been burned at scale by rules that evaluated against nothing while +looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that +gap is ever worth closing, the honest order is a real exporter first and a rule second. + +Corroborating, not a substitute — the server log on the config PVC: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex exec deploy/plex -- \ + sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \ + \"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""' +``` + +Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions. +Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence +of hardware. The dashboard remains the verdict. + +## Likely causes, most likely first + +When the dashboard still says plain `Transcode`: + +1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists. + First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before** + investigating anything technical. + +2. **containerd's device cgroup is refusing the `open()` — UNVERIFIED, and the largest remaining + technical risk in this feature.** Whether a non-privileged pod may `open()` a bind-mounted + character device on this k3s/containerd build **has not been established**; it can only be settled + by deploying and trying. The container inits as uid 0, which settles *file* permissions, but + **uid 0 does not bypass the cgroup device gate.** Symptom: step 2 shows `renderD128` present and + correctly owned, step 3 returns `RW-OPEN-DENIED`. + **The documented fallback is the Intel Device Plugin for Kubernetes** (`gpu.intel.com/i915` as a + resource request, no hostPath). Because the ServiceDefinition field expresses *intent* rather than + a device path, that migration is a **`catalog/render.py` change plus a re-render — with no + ServiceDefinition edit anywhere**; `catalog/plex.yaml` keeps saying exactly what it says today. + **`privileged: true` is not an available escape hatch:** Kyverno `restrict-host-namespaces` is in + **Enforce** mode and rejects it in app namespaces. Do not reach for it. + +3. **The pod landed on a node without the device.** Step 2 shows an empty `/dev/dri`. All three prod + nodes carried `renderD128` with `i915` bound when this was measured (2026-09-30), so this means a + node changed — a rebuild, a BIOS reset, or a driver regression. `DirectoryOrCreate` is why the app + tier still came up; the cost is that the loss is silent. **Fix the node, do not tighten the mount:** + `Directory` would stall `apps-wave-0` (`wait: true`, with `apps-wave-1` depending on it) for a + feature that is not load-bearing. + +4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the + mechanism this feature leans on with no commit in this repo. + +5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so + `Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as + a hardware failure. + +6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which + Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some + sessions show `(hw)` and others do not. Not a fault. + +## The floating-image caveat + +Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app +user to the device's group, the absence of `PLEX_PREFERENCE*`, the six `setPref` keys, the +`HardwareAcceleratedCodecs` string — was read from digest **`sha256:e0ab27395614…`** +(`plexmediaserver 1.43.4.10903`). + +The deployment runs **`plexinc/pms-docker:latest` with `imagePullPolicy: Always`**. A silent pull can +therefore change the behaviour this feature depends on **with no commit in this repo**, while the +manifest still looks identical. Symptom to watch for: hardware transcoding that worked and quietly +stopped, with no git change to point at. + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \ + -o jsonpath="{.items[0].status.containerStatuses[0].imageID}"; echo' +``` + +If that digest is not the one above, **re-verify the hook before trusting this page**: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n plex exec deploy/plex -- ls /etc/cont-init.d/' +``` + +`45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one +described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups` +needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that +hook exists. + +## When to escalate or stop + +- **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be + **inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's + `cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its + group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces** + the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with + `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is + discarded before Plex starts. `render.py --selftest` asserts that absence deliberately. +- **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will + reject it. If the cgroup gate is the blocker, the answer is the Intel device plugin (cause #2). +- **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles + `main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating + `kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change + through a branch, a PR and a green gate. +- **Do not edit `Preferences.xml` on the PVC**, never `cat` it, and never `sops -d` anything into a + terminal. It holds `PlexOnlineToken`. +- **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and + watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to + lose the DB. +- **Stop and get the operator** if the device cannot be opened from the pod (cause #2 confirmed). The + next step is a platform change — the Intel device plugin — with its own PR, not more poking at this + workload. + +## Known gaps, deliberately not fixed here + +Recorded so they are findable, and so nobody assumes they were overlooked: + +- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets + `TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable + overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so + transcode scratch performance changes silently whenever the pod is rescheduled. Both candidate fixes + need a **schema** change and are therefore out of scope here: pin Plex to prod-2 (needs a + `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory` emptyDir and + transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not have). The RAM option + is the better one, because it removes the node asymmetry outright instead of working around it. +- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode** + quality only — decode, and the feature on this page, are unaffected. Changing it needs a kernel + parameter and a **serialized rolling reboot**, which is an operator action (CLAUDE.md §5), and this + cluster has a documented history of that going badly (a prod node hung on a warm reboot on + 2026-09-03). Not worth coupling to this change. + +--- +Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/` +The node measurements (all three prod nodes) and the in-container image measurements cited here were +taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…` +(`plexmediaserver 1.43.4.10903`). Hardware transcoding had **not** been observed working at the time +of writing, and the Plex-side toggle was still unset.