home-network-k3s-impl/runbooks/plex-quicksync.md

plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did

Alert: none, and none is possible — see "How to verify" below. This is an operator procedure, not an incident page. It is indexed here because the feature it completes is inert until a human clicks one button, and a switched-off feature that looks shipped is exactly what this page exists to prevent. Files: catalog/plex.yaml (hardware.gpu: intel-quicksync), apps/wave-0/plex.yaml (the rendered golden — the resources.limits: gpu.intel.com/i915: 1 block), catalog/render.py (the mechanism), schemas/renderer-contract.md ### hardware (the contract), platform/intel-gpu-plugin/ (the DaemonSet that advertises the resource). Related: runbooks/plex-claim.md — the other Plex procedure with a manual last mile.

STOP HERE IF PLEX IS Pending / Unschedulable

That is this feature's one serious failure mode and it stalls the whole app tier (apps-wave-0 is wait: true; apps-wave-1 dependsOn it). Since 2026-10-01 Plex requests the extended resource gpu.intel.com/i915: 1, which is a hard scheduling constraint — if the intel-gpu-plugin DaemonSet is not advertising it on at least one schedulable node, Plex does not start at all. Go to "Plex will not schedule" below; it is the first troubleshooting section on purpose.

The escape hatch, in one sentence: delete the two hardware: lines from catalog/plex.yaml, re-render, and merge — Plex returns to software transcoding and the tier unblocks. Exact commands are in "Plex will not schedule" → remedy B.

What this means here

Plex transcodes in software by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync moves that work off the CPU — which matters on a three-node cluster where the CPU is also running everything else. Two independent things have to be true before that happens, and only one of them is in git:

Half	Who does it	State
The render device is inside the container and openable	GitOps. hardware.gpu: intel-quicksync in catalog/plex.yaml renders resources.limits: {gpu.intel.com/i915: 1} on the app container. The intel-gpu-plugin DaemonSet (platform/intel-gpu-plugin/) injects the device and the matching device-cgroup allow rule. Flux applies it.	Done by merging that change.
Plex is told to use it	The operator, by hand, in the Plex web UI.	NOT done. HardwareAcceleratedCodecs was absent from Preferences.xml when this page was written — the toggle has never been on.

There is no GitOps path to the second half, and that is verified rather than assumed. A bounded grep across the running image's entire init surface — /etc/cont-init.d, /etc/services.d, /etc/fix-attrs.d, /plex-common.sh, /usr/local/bin — found zero PLEX_PREFERENCE* support. Every setPref the image performs is one of six hardcoded keys (MachineIdentifier, ProcessedMachineIdentifier, PlexOnlineToken, customConnections, allowedNetworks, TranscoderTempDirectory), none of them the transcode toggle. No environment variable, no ConfigMap and no postBuild substitution can set it. So do not go looking for the clever way to automate this; there isn't one, and the search has already been done.

Why a resource request and not a /dev/dri hostPath

Because the hostPath was tried and measured not to work, and knowing that is the only thing that will stop someone "simplifying" it back. Merged 2026-09-30, measured in the live pod on k3s-prod-2 on 2026-10-01: the device node was plainly visible in the container (crw-rw---- root video2 226,128 renderD128) and could not be opened —

attempt	result
uid 0, O_RDWR	EPERM "Operation not permitted"
uid 1000, O_RDWR	EPERM "Operation not permitted"

EPERM is the device cgroup refusing, not EACCES from file permissions. The two look identical from the outside and are not the same problem. Proven three ways: root on the host opened the same device rc=0; the node is unified cgroup2fs; and the container's runc config.json read linux.resources.devices: [{allow: False, access: 'rwm'}] with linux.devices: None — one deny-all rule, no allow entry for major 226. A hostPath is a filesystem operation: it binds the device node in and adds no cgroup allow rule, because Kubernetes cannot infer one from a volume.

Two non-workarounds, so nobody spends an evening on them: uid 0 failed too, so running as root does not bypass the cgroup gate; and privileged: true is not available — Kyverno restrict-host-namespaces is Enforce on app namespaces and rejects it.

The device plugin is what produces the allow rule — in the v0.37.0 source, createDeviceSpecsFromDrmFiles emits DeviceSpec{…, Permissions: "rw"}, which the kubelet turns into both the device injection and the matching cgroup allow.

The failure mode that trade bought, stated plainly. Before, a missing device meant the pod ran and silently software-transcoded — invisible, but harmless to everything else. Now, a missing resource means Plex is Unschedulable and the app tier stalls. That is strictly worse in blast radius and it is unavoidable: an extended resource is a hard scheduling constraint, and it is also the only thing that produces the cgroup allow rule. There is no graceful-degradation variant. The one improvement is that the loss is no longer silent.

What this does not do, so nobody reads more into a green gate than is there: it does not enable hardware transcoding (that is the manual toggle below), a green gate only proves the manifest renders, and it writes no nodeSelector — the resource itself constrains Plex to nodes the plugin advertises, which is the scheduler's doing and not a selector in this repo.

Running the commands in this runbook

kubectl does not work on the operator workstation; the toolchain lives only in the pinned dev container. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges both clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once. Do not run two dev.sh invocations at once.

Never cat Preferences.xml. It contains PlexOnlineToken, a plex.tv account credential. Every read of that file below greps a single named key — do not widen one.

First commands

0. Is Plex running at all? Ask this first now — the current mechanism can stop it starting.

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'

Pending ⇒ go straight to "Plex will not schedule". Running ⇒ carry on.

1. Do the nodes advertise the resource, and is the plugin Ready? This replaces the old "is the hostPath mounted" check, and it is now the first thing that can be wrong.

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get nodes -o json \
   | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'

Expect i915=1 on each of the three prod nodes (measured 2026-10-01: all three). Any ABSENT ⇒ that node cannot run Plex. Then the DaemonSet behind it:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n intel-gpu-plugin get daemonset intel-gpu-plugin'

Expect 3 desired / 3 ready / 3 available (measured 2026-10-01). Anything less is the cause, not a symptom — fix it there, in platform/intel-gpu-plugin/, not here.

2. Did Flux apply the current manifest? gpu.intel.com/i915 on the app container means the GitOps half landed:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
     -o jsonpath="{.items[0].spec.containers[0].resources.limits}"; echo'

A /dev/dri volumeMount instead ⇒ the running pod predates the 2026-10-01 merge and is on the mechanism that does not work. Compare

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 \
     -o jsonpath="{.status.lastAppliedRevision}"; echo'

against the merge commit before blaming anything else.

3. Is the device visible inside the container, and can Plex open it? This is the decisive test of the whole mechanism, and it must be run as uid 1000, because that is what Plex runs as.

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'

Expect renderD128, mode crw-rw----. The plugin injects it; there is no longer any DirectoryOrCreate empty-directory case — if the resource were missing the pod would not be running to exec into. Then the open, as the app user:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
     s6-setuidgid plex sh -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'

RW-OPEN-OK as uid 1000 is the pass condition. Passing as uid 0 is weaker evidence and must not be accepted as proof — Plex runs as 1000, and the whole point of the 2026-10-01 change was a cgroup rule that uid 0 could not buy its way past either. RW-OPEN-DENIED with the device plainly visible is the device-cgroup failure — i.e. the plugin's allow rule did not arrive; see "Likely causes" #2.

4. Is the toggle on? Named key only — never the whole file:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
     grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
     "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'

ABSENT ⇒ never enabled (the state this page was written in). ="0" ⇒ explicitly off. ="1" ⇒ on, and the remaining question is whether it works — go to "How to verify".

The one manual step

Plex web UI → Settings → Transcoder → "Use hardware acceleration when available" → tick → Save. (It sits under Show Advanced.) A transcode already in flight is unaffected; the next one picks it up.

The underlying preference key is HardwareAcceleratedCodecs. That name was confirmed by extracting strings from the 23 MB Plex Media Server binary in the running container — not from recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise them and not invent them:

Key	What it is
HardwareAcceleratedCodecs	the toggle this page is about
HardwareAcceleratedEncoders	hardware encode, as distinct from decode
HardwareDevicePath	how Plex is pointed at a specific render node — relevant only if a host ever exposes more than one
TranscoderToneMapping, TranscoderToneMappingAgorithm	HDR→SDR tone mapping. That misspelling of "Algorithm" is upstream's, in the binary. It is not a typo in this page — do not "fix" it here or anywhere else; the corrected spelling is not a key Plex reads.
TranscoderHEVCEncoding	HEVC output

HardwareAcceleratedCodecsEnableHEVC does not exist. It is widely cited and it is not in the binary. Recorded explicitly so nobody spends an evening hunting for it or, worse, writes it into Preferences.xml and concludes hardware transcoding is broken.

Editing Preferences.xml by hand is not the procedure. It is app state on the replicated Longhorn tier, it is in no git commit, and the UI writes the key correctly. Use the UI.

How to verify — and why there is no alert

Start a playback that forces a transcode (a client that cannot direct-play the source, or pick a lower quality explicitly), then read Plex's own Dashboard → Activity:

• Transcode (hw) on the session ⇒ hardware transcoding. This is the proof.
• plain Transcode ⇒ software. The feature is not working, whatever the manifest says.

That dashboard is the only proof available, and nothing else will ever tell you. catalog/plex.yaml declares no metricsEndpoint, so Plex renders no ServiceMonitor at all — there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently inert, and this repo has already been burned at scale by rules that evaluated against nothing while looking like coverage (runbooks/README.md, "Alerts you should not trust the silence of"). If that gap is ever worth closing, the honest order is a real exporter first and a rule second.

Corroborating, not a substitute — the server log on the config PVC:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
     sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
     \"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'

Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions. Absence of a match is not evidence of software transcoding; a VA-API device open is good evidence of hardware. The dashboard remains the verdict.

Likely causes, most likely first

When the dashboard still says plain Transcode:

1. The toggle is off. Overwhelmingly the most likely answer, and the reason this page exists. First-command #4 returns ABSENT or ="0". Remedy: the manual step above. Check this before investigating anything technical.

2. The device cannot be open()ed — the plugin's cgroup allow rule did not arrive. Symptom: first-command #3 shows renderD128 present and correctly owned, and the open returns RW-OPEN-DENIED / Operation not permitted. This is EPERM from the device cgroup, not EACCES from file permissions. This is no longer a design risk — it is the measured reason the design is what it is (2026-10-01; see "Why a resource request and not a /dev/dri hostPath"). If it recurs under the current mechanism, the plugin is the suspect, not the manifest: confirm the pod really did consume the resource

   infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
        -o jsonpath="{.items[0].spec.containers[0].resources}"; echo'
   

   and then check the plugin pod on Plex's node (first-command #1). Do not reach for privileged: true — Kyverno restrict-host-namespaces is Enforce and rejects it, and uid 0 failed the same open in 2026-09-30's measurement anyway. Do not re-add a /dev/dri hostPath; that is the mechanism this replaced, and it was measured not to work.

3. The pod landed on a node whose GPU went away. Under the current mechanism this does not present as an empty /dev/dri — the node simply stops advertising gpu.intel.com/i915 and Plex either never schedules there or gets evicted off. So this cause now shows up as Pending, not as silent software transcoding: see "Plex will not schedule". All three prod nodes carried renderD128 with i915 bound when measured (2026-09-30) and all three advertised the resource on 2026-10-01, so this means a node changed — a rebuild, a BIOS reset, or a driver regression. Fix the node.

4. The image changed under you. See "The floating-image caveat". A :latest pull can remove the mechanism this feature leans on with no commit in this repo.

5. The session was never transcoding. Direct Play / Direct Stream never touches the transcoder, so Transcode (hw) cannot appear. Confirm the session actually says Transcode before treating it as a hardware failure.

6. The source needs a codec Quick Sync on this generation cannot handle — e.g. AV1 decode, which Coffee Lake's UHD 630 has no engine for. Plex falls back to software per stream. Symptom: some sessions show (hw) and others do not. Not a fault.

Plex will not schedule

Symptom: the Plex pod sits Pending, and apps-wave-0 does not go Ready, which holds apps-wave-1 behind it. This is the cost of the 2026-10-01 mechanism and the section you want at 2am.

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex describe pod -l app.kubernetes.io/name=plex | tail -20'

0/3 nodes are available: 3 Insufficient gpu.intel.com/i915 (or similar) confirms it: nothing is advertising the resource Plex requires. Plex's schedulability is coupled to the intel-gpu-plugin DaemonSet, deliberately and unavoidably — the plugin is also the only thing that produces the device-cgroup allow rule, so there is no variant of this feature that degrades gracefully.

Remedy A — fix the plugin (preferred). Run first-command #1. If the DaemonSet is not 3/3, or the nodes show i915=ABSENT, the fault is in platform/intel-gpu-plugin/ or on the node, and that is where to fix it. Check its Flux Kustomization reconciled:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get kustomization intel-gpu-plugin'

A plugin that is merely not yet reconciled resolves itself; one that is failing does not.

Remedy B — the escape hatch: give up hardware transcoding and unblock the tier. One field, no code change. In a branch (never on main, never kubectl edit — Flux reconciles main and would revert you):

1. delete these two lines from catalog/plex.yaml:

   hardware:
     gpu: intel-quicksync
   

2. re-render the golden:

   python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
   

   The only change must be apps/wave-0/plex.yaml, losing its resources block.

3. gate, PR, merge. Plex schedules anywhere and software-transcodes; the app tier unblocks.

Re-adding the field later is the same edit in reverse. Nothing about the Plex-side toggle or the /config PVC is affected either way — do not delete the PVC as part of this.

The floating-image caveat

Everything this page states about the image's behaviour — the cont-init.d hook that adds the app user to the device's group, the absence of PLEX_PREFERENCE*, the six setPref keys, the HardwareAcceleratedCodecs string — was read from digest sha256:e0ab27395614… (plexmediaserver 1.43.4.10903).

The deployment runs plexinc/pms-docker:latest with imagePullPolicy: Always. A silent pull can therefore change the behaviour this feature depends on with no commit in this repo, while the manifest still looks identical. Symptom to watch for: hardware transcoding that worked and quietly stopped, with no git change to point at.

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
     -o jsonpath="{.items[0].status.containerStatuses[0].imageID}"; echo'

If that digest is not the one above, re-verify the hook before trusting this page:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- ls /etc/cont-init.d/'

45-plex-hw-transcode-and-connected-tuner still present ⇒ the group mechanism is still the one described here. Gone or renamed ⇒ stop and re-measure. In particular the "no supplementalGroups needed" conclusion in catalog/plex.yaml and schemas/renderer-contract.md is only true while that hook exists.

When to escalate or stop

• Do not add supplementalGroups or a podSecurityContext to catalog/plex.yaml. It would be inert, and it would mislead the next person. Measured in-container 2026-09-30: the image's cont-init.d hook stats each /dev/dri char device and usermod -a -Gs the plex user into its group inside the container, and the service then execs via s6-setuidgid plex, which replaces the process group list from /etc/group rather than appending — Plex Media Server runs with Groups: 100 1000, root's gid 0 (which PID 1 does hold) gone. Anything Kubernetes supplies is discarded before Plex starts. render.py --selftest asserts that absence deliberately.
• Do not reach for privileged: true. Kyverno restrict-host-namespaces is Enforce and will reject it — and it would not help: uid 0 failed the same open() in the 2026-09-30 measurement, because root does not bypass the cgroup device gate.
• Do not re-add a /dev/dri hostPath, and do not "simplify" the resource request into one. That was the original mechanism and it was measured non-functional on 2026-10-01 (device-cgroup EPERM with the device plainly visible). The device plugin's DeviceSpec{… Permissions: "rw"} is what grants access; a volume cannot.
• Do not kubectl edit/patch the Deployment to add or change the device mount. Flux reconciles main; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating kubectl and flux reconcile are operator actions, and the durable change is a git change through a branch, a PR and a green gate.
• Do not edit Preferences.xml on the PVC, never cat it, and never sops -d anything into a terminal. It holds PlexOnlineToken.
• Do not delete the plex-config PVC to "start clean". It is the library database, metadata and watch state (policy: daily-offcluster). The transcode toggle is one preference, not a reason to lose the DB.
• Stop and get the operator if the device cannot be opened from the pod as uid 1000 (cause #2 confirmed) under the current mechanism. The next step is a platform change in platform/intel-gpu-plugin/, with its own PR, not more poking at this workload.

Known gaps, deliberately not fixed here

Recorded so they are findable, and so nobody assumes they were overlooked:

• Transcode scratch is on the wrong medium, and it is invisible. The image sets TranscoderTempDirectory=/transcode, which is not a volume — it is the container's writable overlay, i.e. the node's /data. That is NVMe on prod-2 but SATA on prod-1 and prod-3, so transcode scratch performance changes silently whenever the pod is rescheduled. Both candidate fixes need a schema change and are therefore out of scope here: pin Plex to prod-2 (needs a nodeSelector field the schema does not have), or give /transcode a medium: Memory emptyDir and transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not have). The RAM option is the better one, because it removes the node asymmetry outright instead of working around it.
• GuC/HuC firmware is not loaded (i915.enable_guc=0). The cost is low-power H.264 encode quality only — decode, and the feature on this page, are unaffected. Changing it needs a kernel parameter and a serialized rolling reboot, which is an operator action (CLAUDE.md §5), and this cluster has a documented history of that going badly (a prod node hung on a warm reboot on 2026-09-03). Not worth coupling to this change.

---

Source of truth: runbooks/plex-quicksync.md. Index: /runbooks/ The node measurements (all three prod nodes) and the in-container image measurements cited here were taken 2026-09-30 against plexinc/pms-docker digest sha256:e0ab27395614… (plexmediaserver 1.43.4.10903). The EPERM open measurements, the Groups: 44 100 992 1000 reading and the gpu.intel.com/i915: 1 allocatable/DaemonSet readings were taken 2026-10-01. Hardware transcoding has still not been observed working: the open test has not been re-run under the device-plugin mechanism (it cannot be, until that merges and Flux reconciles), and the Plex-side toggle is still unset.