home-network-k3s-impl/runbooks/plex-quicksync.md

# plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did

**Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**,
not an incident page. It is indexed here because the feature it completes is **inert until a human
clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
to prevent.
**Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the
rendered golden — the `resources.limits: gpu.intel.com/i915: 1` block), `catalog/render.py` (the
mechanism), `schemas/renderer-contract.md` `### hardware` (the contract),
`platform/intel-gpu-plugin/` (the DaemonSet that advertises the resource).
**Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile.

## STOP HERE IF PLEX IS `Pending` / `Unschedulable`

That is this feature's **one serious failure mode** and it **stalls the whole app tier**
(`apps-wave-0` is `wait: true`; `apps-wave-1` `dependsOn` it). Since 2026-10-01 Plex requests the
extended resource `gpu.intel.com/i915: 1`, which is a **hard scheduling constraint** — if the
`intel-gpu-plugin` DaemonSet is not advertising it on at least one schedulable node, Plex does not
start at all. Go to **"Plex will not schedule"** below; it is the first troubleshooting section on
purpose.

**The escape hatch, in one sentence:** delete the two `hardware:` lines from `catalog/plex.yaml`,
re-render, and merge — Plex returns to software transcoding and the tier unblocks. Exact commands
are in **"Plex will not schedule" → remedy B**.

## What this means here

Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync
moves that work off the CPU — which matters on a three-node cluster where the CPU is also running
everything else. Two independent things have to be true before that happens, and **only one of them
is in git**:

| Half | Who does it | State |
|---|---|---|
| The render device is inside the container **and openable** | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders `resources.limits: {gpu.intel.com/i915: 1}` on the app container. The `intel-gpu-plugin` DaemonSet (`platform/intel-gpu-plugin/`) injects the device **and the matching device-cgroup allow rule**. Flux applies it. | Done by merging that change. |
| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |

**There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`,
`/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin` — found **zero** `PLEX_PREFERENCE*` support.
Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentifier`,
`ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`,
`TranscoderTempDirectory`), none of them the transcode toggle. No environment variable, no ConfigMap
and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this;
there isn't one, and the search has already been done.

### Why a resource request and not a `/dev/dri` hostPath

Because the hostPath was **tried and measured not to work**, and knowing that is the only thing that
will stop someone "simplifying" it back. Merged 2026-09-30, measured in the live pod on k3s-prod-2 on
**2026-10-01**: the device node was plainly **visible** in the container
(`crw-rw---- root video2 226,128 renderD128`) and **could not be opened** —

| attempt | result |
|---|---|
| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |

`EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions. The two look identical
from the outside and are not the same problem. Proven three ways: root on the **host** opened the
same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc `config.json` read
`linux.resources.devices: [{allow: False, access: 'rwm'}]` with `linux.devices: None` — **one
deny-all rule, no allow entry for major 226**. A hostPath is a *filesystem* operation: it binds the
device node in and adds **no cgroup allow rule**, because Kubernetes cannot infer one from a volume.

Two non-workarounds, so nobody spends an evening on them: **`uid 0` failed too**, so running as root
does not bypass the cgroup gate; and **`privileged: true` is not available** — Kyverno
`restrict-host-namespaces` is **Enforce** on app namespaces and rejects it.

The device plugin is what produces the allow rule — in the v0.37.0 source,
`createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns
into both the device injection and the matching cgroup allow.

**The failure mode that trade bought, stated plainly.** Before, a missing device meant the pod ran
and **silently software-transcoded** — invisible, but harmless to everything else. Now, a missing
*resource* means Plex is **Unschedulable** and the app tier stalls. That is strictly worse in blast
radius and it is **unavoidable**: an extended resource is a hard scheduling constraint, and it is
also the only thing that produces the cgroup allow rule. There is no graceful-degradation variant.
The one improvement is that the loss is no longer silent.

What this does **not** do, so nobody reads more into a green gate than is there: it does not enable
hardware transcoding (that is the manual toggle below), a green gate only proves the **manifest
renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
advertises, which is the scheduler's doing and not a selector in this repo.

## Running the commands in this runbook

`kubectl` does not work on the operator workstation; the toolchain lives only in the pinned dev
container. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
picks a current-context **per invocation**, and that default has already changed once. Do not run two
`dev.sh` invocations at once.

**Never `cat` `Preferences.xml`.** It contains `PlexOnlineToken`, a plex.tv account credential. Every
read of that file below greps a single named key — do not widen one.

## First commands

**0. Is Plex running at all?** Ask this first now — the current mechanism can stop it starting.

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
```

`Pending` ⇒ go straight to **"Plex will not schedule"**. Running ⇒ carry on.

**1. Do the nodes advertise the resource, and is the plugin Ready?** This replaces the old "is the
hostPath mounted" check, and it is now the first thing that can be wrong.

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get nodes -o json \
   | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
```

Expect `i915=1` on each of the three prod nodes (measured 2026-10-01: all three). Any `ABSENT` ⇒ that
node cannot run Plex. Then the DaemonSet behind it:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n intel-gpu-plugin get daemonset intel-gpu-plugin'
```

Expect **3 desired / 3 ready / 3 available** (measured 2026-10-01). Anything less is the cause, not a
symptom — fix it there, in `platform/intel-gpu-plugin/`, not here.

**2. Did Flux apply the current manifest?** `gpu.intel.com/i915` on the app container means the
GitOps half landed:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
     -o jsonpath="{.items[0].spec.containers[0].resources.limits}"; echo'
```

A `/dev/dri` **volumeMount** instead ⇒ the running pod predates the 2026-10-01 merge and is on the
mechanism that does not work. Compare

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 \
     -o jsonpath="{.status.lastAppliedRevision}"; echo'
```

against the merge commit before blaming anything else.

**3. Is the device visible inside the container, and can Plex *open* it?** This is the decisive test
of the whole mechanism, and **it must be run as uid 1000**, because that is what Plex runs as.

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'
```

Expect `renderD128`, mode `crw-rw----`. The plugin injects it; there is no longer any
`DirectoryOrCreate` empty-directory case — if the resource were missing the pod would not be running
to exec into. Then the open, **as the app user**:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
     s6-setuidgid plex sh -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
```

`RW-OPEN-OK` **as uid 1000** is the pass condition. **Passing as `uid 0` is weaker evidence and must
not be accepted as proof** — Plex runs as 1000, and the whole point of the 2026-10-01 change was a
cgroup rule that uid 0 could not buy its way past either. `RW-OPEN-DENIED` with the device plainly
visible is the **device-cgroup** failure — i.e. the plugin's allow rule did not arrive; see "Likely
causes" #2.

**4. Is the toggle on?** Named key only — never the whole file:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
     grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
     "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'
```

`ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on,
and the remaining question is whether it *works* — go to "How to verify".

## The one manual step

Plex web UI → **Settings → Transcoder → "Use hardware acceleration when available"** → tick → Save.
(It sits under *Show Advanced*.) A transcode already in flight is unaffected; the next one picks it up.

The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by
extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from
recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise
them and not invent them:

| Key | What it is |
|---|---|
| `HardwareAcceleratedCodecs` | the toggle this page is about |
| `HardwareAcceleratedEncoders` | hardware **encode**, as distinct from decode |
| `HardwareDevicePath` | how Plex is pointed at a *specific* render node — relevant only if a host ever exposes more than one |
| `TranscoderToneMapping`, `TranscoderToneMappingAgorithm` | HDR→SDR tone mapping. **That misspelling of "Algorithm" is upstream's, in the binary. It is not a typo in this page — do not "fix" it** here or anywhere else; the corrected spelling is not a key Plex reads. |
| `TranscoderHEVCEncoding` | HEVC output |

**`HardwareAcceleratedCodecsEnableHEVC` does not exist.** It is widely cited and it is not in the
binary. Recorded explicitly so nobody spends an evening hunting for it or, worse, writes it into
`Preferences.xml` and concludes hardware transcoding is broken.

Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn
tier, it is in no git commit, and the UI writes the key correctly. Use the UI.

## How to verify — and why there is no alert

Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a
lower quality explicitly), then read Plex's own **Dashboard → Activity**:

- **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.**
- plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says.

**That dashboard is the *only* proof available, and nothing else will ever tell you.**
`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work
to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently
inert, and this repo has already been burned at scale by rules that evaluated against nothing while
looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that
gap is ever worth closing, the honest order is a real exporter first and a rule second.

Corroborating, not a substitute — the server log on the config PVC:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
     sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
     \"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'
```

Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions.
Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence
of hardware. The dashboard remains the verdict.

## Likely causes, most likely first

When the dashboard still says plain `Transcode`:

1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists.
   First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
   investigating anything technical.

2. **The device cannot be `open()`ed — the plugin's cgroup allow rule did not arrive.** Symptom:
   first-command #3 shows `renderD128` present and correctly owned, and the open returns
   `RW-OPEN-DENIED` / `Operation not permitted`. This is `EPERM` from the **device cgroup**, not
   `EACCES` from file permissions. **This is no longer a design risk — it is the measured reason the
   design is what it is** (2026-10-01; see "Why a resource request and not a `/dev/dri` hostPath").
   If it recurs *under the current mechanism*, the plugin is the suspect, not the manifest: confirm
   the pod really did consume the resource

   ```
   infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
        -o jsonpath="{.items[0].spec.containers[0].resources}"; echo'
   ```

   and then check the plugin pod on Plex's node (first-command #1). **Do not reach for
   `privileged: true`** — Kyverno `restrict-host-namespaces` is Enforce and rejects it, and uid 0
   failed the same open in 2026-09-30's measurement anyway. **Do not re-add a `/dev/dri` hostPath**;
   that is the mechanism this replaced, and it was measured not to work.

3. **The pod landed on a node whose GPU went away.** Under the current mechanism this does **not**
   present as an empty `/dev/dri` — the node simply stops advertising `gpu.intel.com/i915` and Plex
   either never schedules there or gets evicted off. So this cause now shows up as **`Pending`**, not
   as silent software transcoding: see "Plex will not schedule". All three prod nodes carried
   `renderD128` with `i915` bound when measured (2026-09-30) and all three advertised the resource on
   2026-10-01, so this means a node changed — a rebuild, a BIOS reset, or a driver regression.
   **Fix the node.**

4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
   mechanism this feature leans on with no commit in this repo.

5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so
   `Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as
   a hardware failure.

6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which
   Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
   sessions show `(hw)` and others do not. Not a fault.

## Plex will not schedule

**Symptom:** the Plex pod sits `Pending`, and `apps-wave-0` does not go Ready, which holds
`apps-wave-1` behind it. This is the cost of the 2026-10-01 mechanism and the section you want at
2am.

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex describe pod -l app.kubernetes.io/name=plex | tail -20'
```

`0/3 nodes are available: 3 Insufficient gpu.intel.com/i915` (or similar) confirms it: **nothing is
advertising the resource Plex requires**. Plex's schedulability is coupled to the `intel-gpu-plugin`
DaemonSet, deliberately and unavoidably — the plugin is also the only thing that produces the
device-cgroup allow rule, so there is no variant of this feature that degrades gracefully.

**Remedy A — fix the plugin (preferred).** Run first-command #1. If the DaemonSet is not 3/3, or the
nodes show `i915=ABSENT`, the fault is in `platform/intel-gpu-plugin/` or on the node, and that is
where to fix it. Check its Flux Kustomization reconciled:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get kustomization intel-gpu-plugin'
```

A plugin that is merely **not yet reconciled** resolves itself; one that is failing does not.

**Remedy B — the escape hatch: give up hardware transcoding and unblock the tier.** One field, no
code change. In a branch (never on `main`, never `kubectl edit` — Flux reconciles `main` and would
revert you):

1. delete these two lines from `catalog/plex.yaml`:

   ```yaml
   hardware:
     gpu: intel-quicksync
   ```

2. re-render the golden:

   ```
   python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
   ```

   The only change must be `apps/wave-0/plex.yaml`, losing its `resources` block.
3. gate, PR, merge. Plex schedules anywhere and **software-transcodes**; the app tier unblocks.

Re-adding the field later is the same edit in reverse. Nothing about the Plex-side toggle or the
`/config` PVC is affected either way — do **not** delete the PVC as part of this.

## The floating-image caveat

Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
user to the device's group, the absence of `PLEX_PREFERENCE*`, the six `setPref` keys, the
`HardwareAcceleratedCodecs` string — was read from digest **`sha256:e0ab27395614…`**
(`plexmediaserver 1.43.4.10903`).

The deployment runs **`plexinc/pms-docker:latest` with `imagePullPolicy: Always`**. A silent pull can
therefore change the behaviour this feature depends on **with no commit in this repo**, while the
manifest still looks identical. Symptom to watch for: hardware transcoding that worked and quietly
stopped, with no git change to point at.

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
     -o jsonpath="{.items[0].status.containerStatuses[0].imageID}"; echo'
```

If that digest is not the one above, **re-verify the hook before trusting this page**:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- ls /etc/cont-init.d/'
```

`45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one
described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups`
needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that
hook exists.

## When to escalate or stop

- **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be
  **inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's
  `cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its
  group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces**
  the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with
  `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
  discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
- **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
  reject it — and it would not help: `uid 0` failed the same `open()` in the 2026-09-30 measurement,
  because root does not bypass the cgroup device gate.
- **Do not re-add a `/dev/dri` hostPath, and do not "simplify" the resource request into one.** That
  was the original mechanism and it was measured non-functional on 2026-10-01 (device-cgroup `EPERM`
  with the device plainly visible). The device plugin's `DeviceSpec{… Permissions: "rw"}` is what
  grants access; a volume cannot.
- **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles
  `main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating
  `kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change
  through a branch, a PR and a green gate.
- **Do not edit `Preferences.xml` on the PVC**, never `cat` it, and never `sops -d` anything into a
  terminal. It holds `PlexOnlineToken`.
- **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and
  watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
  lose the DB.
- **Stop and get the operator** if the device cannot be opened from the pod as uid 1000 (cause #2
  confirmed) *under the current mechanism*. The next step is a platform change in
  `platform/intel-gpu-plugin/`, with its own PR, not more poking at this workload.

## Known gaps, deliberately not fixed here

Recorded so they are findable, and so nobody assumes they were overlooked:

- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
  `TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
  overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
  transcode scratch performance changes silently whenever the pod is rescheduled. Both candidate fixes
  need a **schema** change and are therefore out of scope here: pin Plex to prod-2 (needs a
  `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory` emptyDir and
  transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not have). The RAM option
  is the better one, because it removes the node asymmetry outright instead of working around it.
- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
  quality only — decode, and the feature on this page, are unaffected. Changing it needs a kernel
  parameter and a **serialized rolling reboot**, which is an operator action (CLAUDE.md §5), and this
  cluster has a documented history of that going badly (a prod node hung on a warm reboot on
  2026-09-03). Not worth coupling to this change.

---
Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
The node measurements (all three prod nodes) and the in-container image measurements cited here were
taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
(`plexmediaserver 1.43.4.10903`). The `EPERM` open measurements, the `Groups: 44 100 992 1000`
reading and the `gpu.intel.com/i915: 1` allocatable/DaemonSet readings were taken **2026-10-01**.
Hardware transcoding has **still not been observed working**: the open test has not been re-run under
the device-plugin mechanism (it cannot be, until that merges and Flux reconciles), and the Plex-side
toggle is still unset.