diff --git a/decisions/2026-10-02-plex-quicksync-verification.md b/decisions/2026-10-02-plex-quicksync-verification.md new file mode 100644 index 0000000..b0b08e3 --- /dev/null +++ b/decisions/2026-10-02-plex-quicksync-verification.md @@ -0,0 +1,149 @@ +--- +date: 2026-10-02 +branch: fix/plex-quicksync-verification +title: Hardware transcoding works — and `runbooks/plex-quicksync.md`'s verification section was wrong +--- + +## What changed + +**Hardware transcoding is confirmed working on prod.** Measured 2026-10-02 from +`Plex Media Server.log` in the running pod: `vaapi` on `/dev/dri/renderD128` +(`Intel CoffeeLake-S GT2 [UHD Graphics 630]`), `using hardware decode accelerator vaapi`, +`zero-copy support present`, `using zero-copy transcoding`, and +`final decoder: vaapi, final encoder: vaapi`. VAAPI for **both** directions, zero-copy, so frames +never round-trip through the CPU. The device-plugin mechanism from PR #37 works end to end. + +**And `runbooks/plex-quicksync.md`'s verification section was wrong, which is the correction this +branch exists to publish.** The page said the Plex dashboard showing `Transcode (hw)` was "the *only* +proof available, and nothing else will ever tell you". The operator had hardware transcoding fully +working while **Dashboard → Activity showed nothing of the sort** — so the page's authoritative check +was silent on a working system, and sent someone looking for a fault that did not exist. Three +inference errors are now corrected on the page, two of them recorded as named false leads so the next +reader does not re-walk them: + +1. **The dashboard is not proof.** Replaced as the primary check by Plex's own log — the grep is in + the page, in its **redacting** form, because `Plex Media Server.log` carries API URLs containing + `X-Plex-Token` (CLAUDE.md §6; a credential has been leaked into a transcript here before). The + page also now says explicitly that **`Plex Transcoder Statistics.log` is the wrong file** — it + holds XML `` records, not ffmpeg output, and the name invites the mistake — and why + `grep -a` is required. The dashboard is **kept but demoted** to a secondary, best-effort signal, + with its absence stated to prove nothing. +2. **An fd snapshot is not evidence.** A `/proc/*/fd` sweep inside the pod found no holder of + `renderD128` while hardware transcoding was active; transcode sessions are short and the scan + landed in a gap. Recorded as a documented false lead, with the reliable process-level check in its + place: the transcoder's **argv** (`-hwaccel`, `-hwaccel_device`, `vaapi`, `renderD128`), found via + `ps -eo pid,user,comm | grep -i transcoder`. +3. **A missing preference key does not mean the setting is off.** The page told readers to treat an + absent `HardwareAcceleratedCodecs` in `Preferences.xml` as "never enabled". With hardware + acceleration demonstrably on and working, that key was **absent from all 27 attributes** while + `HardwareDevicePath` **was** written and the log said `hardware transcoding: enabled`. First-command + #4 now reads both keys and states plainly that `HardwareDevicePath` present is a positive signal, + absence of `HardwareAcceleratedCodecs` means nothing either way, only `="0"` is conclusive, and the + log settles it. + +Also on the page: + +- **GID `992` is no longer presented as a number to expect.** Under the plugin the injected device + arrives `root:root`, so the image's `cont-init.d` hook resolves GID **`0`** and adds `plex` to the + **root group** — measured `Groups: 0 100 1000`, `id plex` → `groups=1000(plex),0(root),100(users)`. + 992 was the hostPath-era host `render` GID. The page now says so, marks the root-group membership as + an inherent property of the plugin's `DeviceSpec` rather than a defect, and repeats that it is + **not** a reason to reintroduce `supplementalGroups` (still a no-op: `s6-setuidgid` rebuilds the + group list from `/etc/group`). The stale `Groups: 100 1000` claim in the escalation section is gone. +- **A new troubleshooting entry for legitimate per-title fallbacks**, marked as a **hypothesis**. + Between two fully hardware sessions, one logged `enabled, but no hardware decode accelerator found` + and empty final decoder/encoder. UHD 630 decodes H.264, HEVC 8/10-bit and VP9 but **not AV1** and + not 10-bit 4:2:2 HEVC, so the likely cause is the source file, not the platform — explicitly + written as a lead to check first, **not** a confirmed diagnosis of that item. +- The status is stated at the top of the page and in a rewritten provenance footer that separates the + 2026-10-02 confirmations from the superseded 2026-09-30/2026-10-01 hostPath-era readings. +- `runbooks/README.md`'s index entry for the page was stale in the same two ways (it still described a + `/dev/dri` hostPath and the dashboard-only proof) and is corrected to match. + +Unchanged on purpose: the "STOP HERE IF PLEX IS `Pending`" section and the escape hatch (delete the +`hardware:` block from `catalog/plex.yaml`, re-render) keep exactly their existing prominence — they +are correct and they are the 2am material. The "no alert is possible" reasoning is also kept verbatim +in substance: Plex declares no `metricsEndpoint`, renders no ServiceMonitor, and a fallback alert +would be a rule evaluating against nothing. + +**Documentation only.** No change to `catalog/`, `schemas/`, `apps/` or `platform/`. + +## What I verified + +- **Gate, in the pinned toolbox from this worktree** (`DEV_KUBECONFIG=/dev/null`, `ENGINE` shim + bind-mounting the main checkout's `.git`): **rc 3 UNDETERMINED**, diff base + `origin/main @ 948875aad37d`, with exactly the three expected skips and **no failures**. Green: + `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy`. Skipped: + `spec-immutability` (spec repo unavailable → R-TEST-06 not verified), `gitea-settings` + (`SETTINGS_API_TOKEN` unset), `conformance` (`CLUSTER_AVAILABLE!=true` → L2/L3 not run). + Touched requirements: **R-OBS-04**, reported `UNVERIFIED` for lack of a cluster. `approval-policy` + passed as `N/A` locally (`event='none'`, not a `pull_request`) — it evaluates for real on the PR. + **rc 3 is not green**, it is the calibrated worktree result, and it is reported here as rc 3. +- **The gate was run three times, with the same rc 3 / three-skip result on each committed tip** — + including the final one. Worth recording from the first attempt: run *before* committing, with + `HEAD == origin/main`, the touched-requirement set is `NOT DETERMINABLE (on-base)` and the + R-TEST-03 coverage gate cannot be applied at all. Commit before gating from a fresh worktree branch. +- **`runbooks/kustomization.yaml` still carries `- plex-quicksync=plex-quicksync.md`** in the + `configMapGenerator` `files:` list (line 110) — checked explicitly, because without it the page + merges and still 404s. `README=README.md` is likewise present, which is what serves the corrected + index entry. +- **This change touches no `sensitive_paths`.** `infra/gitea/approval-policy.json` lists `ci/**`, + `.gitea/**`, `conformance/**`, `clusters/**`, `bootstrap/**`, `infra/gitea/**`, + `platform/kyverno/**`, `platform-config/kustomization.yaml`, `.sops.yaml`, `**/*.sops.yaml`, + `**/rbac.yaml`, `CLAUDE.md`. `runbooks/**` and `decisions/**` are not among them, and the + `approval-policy` stage passed. +- **No YAML was added or edited**, so `indent-sequences: consistent` (which blocked a sibling branch + today) is not in play here; `lint` passed over the full tracked YAML set regardless. +- **`DECISIONS-LOG.md` is untouched** — this handoff is its own file, per CLAUDE.md §7. +- **The PR's own CI run was read, not assumed** (PR #39, run 208, task 261, head `1677dff5688b`). + **It is RED, and not because of this branch.** Every stage this change could affect is green — + `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy` (`approval-policy` evaluated + for real on the `pull_request` event and passed: no sensitive path touched). The single failure is + **`gitea-settings`**, quoting the measured field: + + [fail] required_approvals: DECLARED 1 but LIVE is 0 + [fail] Gitea has DRIFTED from infra/gitea/expected-branch-protection.json. + [fail] MERGE GATE: BLOCKED + + Every other declared setting matched (`block_admin_merge_override`, `block_on_rejected_reviews`, + `dismiss_stale_approvals`, `enable_status_check`, `enable_push=false`, + `status_check_contexts = ["gate / gate (pull_request)"]`, `runners online: 1`). So **live `main` + currently requires ZERO approvals**, the declaration still says 1, and with + `REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml` that drift **blocks every PR in + the repo**, not just this one. Note the shape of it: the check that would notice approvals are + unenforced is the check that is failing. Local branch `fix/admin-approval-override` (`616a497`, + "fix(ci): let an administrator merge their own green PR") looks like the in-flight work that means + to change the declaration. **I did not touch it** — `infra/gitea/**` is a sensitive path, needs a + non-author approval, and is outside this branch's documentation-only scope. It is an operator + decision: re-assert the protection, or change the declaration deliberately in its own PR. +- The repository-wide grep for `992` to find what else still asserts it: remaining occurrences are in + `catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md`, + `platform/intel-gpu-plugin/README.md`, `platform/intel-gpu-plugin/kustomization.yaml`, and three + older `decisions/` entries. + +## What I did NOT verify + +- **I did not re-measure anything on the live cluster in this session.** Every platform number on the + page — the VAAPI/zero-copy log lines, `RW-OPEN-OK` as uid 1000, the `gpu.intel.com/i915` limits and + requests on the pod spec, `Groups: 0 100 1000`, `id plex`, the 27 `Preferences.xml` attributes, the + empty `/proc/*/fd` sweep — came to me as measurements taken earlier on 2026-10-02 and was + transcribed, not re-run. Per CLAUDE.md §2b, treat the page as reporting those readings rather than + as proof of current state; re-run the primary check before relying on it. +- **The per-title fallback cause is unverified and labelled as such.** The source codec of the title + that logged `no hardware decode accelerator found` was never read. The AV1 / 10-bit-4:2:2-HEVC + explanation is inferred from UHD 630's capability set alone. +- **Whether Plex's dashboard *ever* shows `Transcode (hw)` on this deployment is unknown.** It was + observed silent during working hardware transcoding; whether that is a client/version quirk, a + reporting bug, or something about this setup was not investigated. The page therefore demotes it + rather than explaining it. +- **The commands as written in the page were not executed from the page.** They are the forms that + were run, re-wrapped for `dev.sh`; in particular the `exec -it deploy/plex -- sh` route for the argv + check is noted as possibly lacking a TTY under `dev.sh run` and was not exercised in that shape. +- **Stale `992` copies outside `runbooks/` were left alone**, deliberately — scope was documentation + only, and `catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md` and + `platform/intel-gpu-plugin/` all still carry the hostPath-era `Groups: 44 100 992 1000` reading as + if current. `render.py --selftest` asserts around that comment, so correcting it is a real change + with gate consequences, not a doc edit. **That is open work.** Older `decisions/` entries also cite + 992; those are historical records and should stay as written. +- **CI on the PR.** The branch is pushed and the PR opened; I did not watch the Gitea run report, so + nothing here should be read as "CI green". The only gate result I can quote is the local rc 3 above. diff --git a/runbooks/README.md b/runbooks/README.md index 28f7be3..3f07cc9 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -153,14 +153,18 @@ Each affected runbook says so on its own page and gives the by-hand equivalent. plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`: token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured). Includes why a changed Secret does not restart anything, and why an expired token is harmless. -- **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod - (`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware - acceleration when available"**, and no environment variable can do it (the image supports no - `PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`, - read out of the binary), says which widely-cited key **does not exist**, and is explicit that the - **only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no - alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin - fallback, and why a `:latest` pull can silently break it. +- **`plex-quicksync`** — hardware transcoding on the host iGPU. **Confirmed working 2026-10-02** + (VAAPI decode *and* encode, zero-copy). GitOps requests `gpu.intel.com/i915` on the pod + (`hardware.gpu: intel-quicksync`, satisfied by the `intel-gpu-plugin` DaemonSet — a `/dev/dri` + hostPath was measured not to work: device-cgroup `EPERM`); **a human still has to tick Settings → + Transcoder → "Use hardware acceleration when available"**, and no environment variable can do it + (the image supports no `PLEX_PREFERENCE*`, verified). **The proof is `Plex Media Server.log`'s + `final decoder: / final encoder:` line, not the Plex dashboard** — the dashboard showed nothing + while hardware transcoding was working, so its silence proves nothing; the page carries the + redacting grep (`X-Plex-Token` lives in that log), the WRONG log file to avoid, and two more + documented false leads (an empty `/proc/*/fd` scan, and a missing `HardwareAcceleratedCodecs` key). + Plex renders no ServiceMonitor, so no alert could ever fire. Also: **`Pending` stalls the whole app + tier** and its one-field escape hatch, and why a `:latest` pull can silently break it. - **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log (the four lines that matter, including the upstream `-` that means the proxy answered a request the diff --git a/runbooks/plex-quicksync.md b/runbooks/plex-quicksync.md index 8c59a03..bd57a22 100644 --- a/runbooks/plex-quicksync.md +++ b/runbooks/plex-quicksync.md @@ -1,5 +1,10 @@ # plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did +**Status: hardware transcoding is CONFIRMED WORKING on prod as of 2026-10-02** — VAAPI for both +decode *and* encode, zero-copy, measured in Plex's own server log (see "How to verify"). The manual +toggle has been ticked. This page is therefore now mostly a **verification and regression** +procedure, and its old verification section was **wrong** — see the correction at the top of +"How to verify". **Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**, not an incident page. It is indexed here because the feature it completes is **inert until a human clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists @@ -33,7 +38,7 @@ is in git**: | Half | Who does it | State | |---|---|---| | The render device is inside the container **and openable** | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders `resources.limits: {gpu.intel.com/i915: 1}` on the app container. The `intel-gpu-plugin` DaemonSet (`platform/intel-gpu-plugin/`) injects the device **and the matching device-cgroup allow rule**. Flux applies it. | Done by merging that change. | -| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. | +| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **Done, 2026-10-02**, and confirmed from the **log** rather than from a preference key. `HardwareAcceleratedCodecs` is **still absent** from `Preferences.xml` with hardware transcoding working, so its absence proves nothing — see first-command #4. | **There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`, @@ -79,8 +84,8 @@ also the only thing that produces the cgroup allow rule. There is no graceful-de The one improvement is that the loss is no longer silent. What this does **not** do, so nobody reads more into a green gate than is there: it does not enable -hardware transcoding (that is the manual toggle below), a green gate only proves the **manifest -renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin +hardware transcoding (that is the manual toggle below — done 2026-10-02), a green gate only proves +the **manifest renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin advertises, which is the scheduler's doing and not a selector in this repo. ## Running the commands in this runbook @@ -172,17 +177,28 @@ cgroup rule that uid 0 could not buy its way past either. `RW-OPEN-DENIED` with visible is the **device-cgroup** failure — i.e. the plugin's allow rule did not arrive; see "Likely causes" #2. -**4. Is the toggle on?** Named key only — never the whole file: +**4. Is the toggle on?** **You cannot answer this from `Preferences.xml` alone.** That mistake is on +this page's record: it told readers to treat a missing `HardwareAcceleratedCodecs` as "never +enabled", and on 2026-10-02 that key was **absent from all 27 attributes** while hardware +transcoding was demonstrably working. Named keys only — never the whole file, it holds +`PlexOnlineToken`: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n plex exec deploy/plex -- \ - grep -o "HardwareAcceleratedCodecs=\"[01]\"" \ - "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT' + grep -o -e "HardwareAcceleratedCodecs=\"[01]\"" -e "HardwareDevicePath=\"[^\"]*\"" \ + "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo NEITHER-KEY' ``` -`ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on, -and the remaining question is whether it *works* — go to "How to verify". +| Reading | What it actually means | +|---|---| +| `HardwareDevicePath="…"` **present** | **Positive signal.** Plex writes this when hardware acceleration is configured; it was present on 2026-10-02 alongside a working VAAPI pipeline. | +| `HardwareAcceleratedCodecs` **absent** | **Nothing. Neither on nor off.** Measured absent with the feature working (2026-10-02). Do not diagnose from it, and do not write it into the file to "fix" it. | +| `HardwareAcceleratedCodecs="0"` | explicitly **off** — the only conclusive reading, and only in the negative. | +| `HardwareAcceleratedCodecs="1"` | on. | + +**The log settles it, this file does not.** Go to "How to verify"; the `hardware transcoding: +enabled` and `final decoder/final encoder` lines are the verdict. ## The one manual step @@ -191,8 +207,11 @@ Plex web UI → **Settings → Transcoder → "Use hardware acceleration when av The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from -recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise -them and not invent them: +recall and not from a forum post. **But do not expect to see it in `Preferences.xml` after ticking +the box:** on 2026-10-02, with the toggle on and hardware transcoding working, the key was absent and +`HardwareDevicePath` was what had been written. The key exists in the binary; Plex does not +necessarily persist it. Real neighbouring keys from the same binary, so you can recognise them and +not invent them: | Key | What it is | |---|---| @@ -209,42 +228,121 @@ binary. Recorded explicitly so nobody spends an evening hunting for it or, worse Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn tier, it is in no git commit, and the UI writes the key correctly. Use the UI. -## How to verify — and why there is no alert +## How to verify — the log, not the dashboard -Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a -lower quality explicitly), then read Plex's own **Dashboard → Activity**: +**Correction, 2026-10-02.** Until today this page said the Plex dashboard showing `Transcode (hw)` +was "the *only* proof available, and nothing else will ever tell you". **That was false and it cost +the operator real time:** hardware transcoding was fully working — VAAPI decode *and* encode, with +zero-copy — while **Dashboard → Activity showed nothing of the sort**. A check that is silent while +the feature works does not just fail to prove anything; it sends someone hunting a fault that does +not exist. The authoritative check is **Plex's own server log**. -- **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.** -- plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says. +### The primary check — run this one -**That dashboard is the *only* proof available, and nothing else will ever tell you.** -`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** — -there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work -to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently -inert, and this repo has already been burned at scale by rules that evaluated against nothing while -looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that -gap is ever worth closing, the honest order is a real exporter first and a rule second. +``` +./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \ + grep -aiE "hardware transcoding: (final|using)" \ + "/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log" \ + | sed -E "s/(X-Plex-Token|token)=[^&[:space:]]*/\1=REDACTED/g" | tail -5' +``` -Corroborating, not a substitute — the server log on the config PVC: +| Line | Verdict | +|---|---| +| `hardware transcoding: final decoder: vaapi, final encoder: vaapi` | **PASS.** Hardware in **both** directions. This is the pass condition. | +| `hardware transcoding: using zero-copy transcoding` | Best case — frames never round-trip through the CPU. Observed on prod 2026-10-02. | +| `hardware transcoding: final decoder: , final encoder: ` (both empty) | It **fell back to software** for that session. | +| no matching line at all | No transcode has been attempted since the log rolled — force one (below), then look again. Absence is not a failure. | + +The 2026-10-02 reading on prod, in full, for the shape to recognise: + +``` +Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD128' + (Intel CoffeeLake-S GT2 [UHD Graphics 630]) +hardware transcoding: using hardware decode accelerator vaapi +hardware transcoding: zero-copy support present +hardware transcoding: using zero-copy transcoding +hardware transcoding: final decoder: vaapi, final encoder: vaapi +``` + +To make a line appear, start a playback that **forces a transcode**: a client that cannot +direct-play the source, or an explicitly lower quality. Direct Play / Direct Stream never reaches +the transcoder and logs nothing. + +### Three things to get right about that command + +1. **Always keep the `sed`.** `Plex Media Server.log` carries API URLs containing + **`X-Plex-Token=…`**, a live plex.tv credential. Any output that might land in a ticket, a + transcript or a chat window must be redacted — CLAUDE.md §6, and a credential has already been + leaked into a transcript in this repo exactly this way. The redacting form above is the form to + copy; do not "simplify" the pipe out of it. +2. **`Plex Transcoder Statistics.log` is the WRONG FILE.** The name is irresistible and someone will + reach for it. It holds XML `` session records — **not** ffmpeg/VAAPI output. There + is no decoder/encoder verdict anywhere in it. The file you want is `Plex Media Server.log`. +3. `grep -a` is deliberate: the log carries occasional non-UTF-8 bytes, and without `-a` GNU grep + reports "binary file matches" and prints nothing — which reads exactly like "no hardware + transcoding". + +### Secondary and unreliable: the Plex dashboard + +**Dashboard → Activity** *may* label a transcoding session `Transcode (hw)`. If it does, it agrees +with the log and you are done. **If it does not, that proves nothing at all** — measured 2026-10-02: +hardware transcoding active and zero-copy, dashboard silent. Never conclude "software transcoding" +from the dashboard and never open an investigation on its say-so. Kept here only because it is the +fastest glance available; it is a convenience, **not evidence**. + +### Process-level, while a transcode is in flight: the transcoder's argv + +This shows Plex *requesting* hardware, as distinct from reporting it afterwards. Get a shell in the +pod (if `-it` has no TTY under `dev.sh run`, wrap each line in `sh -c "…"` instead): ``` infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n plex exec deploy/plex -- \ - sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \ - \"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""' + 'kubectl --context k3s-prod -n plex exec -it deploy/plex -- sh' ``` -Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions. -Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence -of hardware. The dashboard remains the verdict. +then, inside the pod: + +``` +ps -eo pid,user,comm | grep -i transcoder +tr '\0' ' ' < /proc//cmdline | grep -oE '(-hwaccel[a-z_]*|vaapi|renderD[0-9]+)' +``` + +`-hwaccel`, `-hwaccel_device`, `vaapi`, `renderD128` present ⇒ Plex is asking for hardware. Keep the +`grep -oE`: the raw argv contains media paths and session identifiers you do not want in a +transcript. This only works while a session is actually transcoding. + +### Documented false lead: scanning `/proc/*/fd` for a holder of `renderD128` + +**It returns nothing even when hardware transcoding is working.** Measured 2026-10-02: a +`/proc/*/fd` sweep inside the pod found **no** process holding `/dev/dri/renderD128`, while the log +showed `final decoder: vaapi, final encoder: vaapi` for sessions either side of the scan. Transcode +sessions are **short** and the scan landed in a gap between two of them. An empty fd scan is a +**timing artefact, not a finding** — do not report it as evidence of software transcoding, and do +not escalate on it. Use the argv check above (which has the same timing constraint but tells you +something when it does catch a session), or just read the log, which is retrospective and therefore +does not care about timing. + +### Why there is no alert — this half was always right + +`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** — +there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open +work to "alert on software-transcode fallback": with no metric behind it such a rule would be +permanently inert, and this repo has already been burned at scale by rules that evaluated against +nothing while looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence +of"). If that gap is ever worth closing, the honest order is a real exporter first and a rule +second. ## Likely causes, most likely first -When the dashboard still says plain `Transcode`: +When the **log** says it fell back to software — `final decoder: , final encoder: ` with both +fields empty. (Not when the dashboard is quiet: the dashboard's silence means nothing, see "How to +verify".) -1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists. - First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before** - investigating anything technical. +1. **The toggle is off.** Historically the most likely answer and the reason this page exists. It was + ticked on 2026-10-02 and should stay ticked, but a `/config` restore or a stray UI save can unset + it. Remedy: the manual step above. **Do not diagnose this from a missing + `HardwareAcceleratedCodecs`** — see first-command #4; only `="0"` is conclusive, and the log line + `hardware transcoding: enabled` is the real signal that the toggle took. 2. **The device cannot be `open()`ed — the plugin's cgroup allow rule did not arrive.** Symptom: first-command #3 shows `renderD128` present and correctly owned, and the open returns @@ -276,13 +374,27 @@ When the dashboard still says plain `Transcode`: 4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the mechanism this feature leans on with no commit in this repo. -5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so - `Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as - a hardware failure. +5. **The session was never transcoding.** Direct Play / Direct Stream never reaches the transcoder, + so no `hardware transcoding:` line is written for it at all. An *absent* line is this, not a + failure — confirm a transcode actually happened before treating anything as a hardware fault. -6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which - Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some - sessions show `(hw)` and others do not. Not a fault. +6. **The *source* codec may be one this iGPU cannot decode — HYPOTHESIS, not a confirmed + diagnosis.** The symptom that points here is fallback on **particular titles** while other + sessions are hardware. Measured on prod 2026-10-02, *between* two fully hardware sessions, one + session logged: + + ``` + hardware transcoding: enabled, but no hardware decode accelerator found + hardware transcoding: final decoder: , final encoder: + ``` + + The platform was demonstrably working either side of it, so the likely explanation is the **file**, + not the cluster: UHD 630's media engine decodes H.264, HEVC 8-bit and 10-bit, and VP9 — but **not + AV1**, and not 10-bit 4:2:2 HEVC. **This was NOT verified against that particular title.** It is + inferred from the hardware's capability set and is written here as the lead to check first, not as + a finding. So: fallback on *all* files is a platform problem (causes 1-4); fallback on *some* + files, check the source codec before suspecting the device, and expect occasional legitimate + per-title fallbacks to be normal rather than a fault. ## Plex will not schedule @@ -360,19 +472,30 @@ infra/devcontainer/dev.sh run bash -c \ ``` `45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one -described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups` -needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that -hook exists. +described here: the hook reads the injected device's **own** ownership, which under the device plugin +is `root:root`, so it resolves GID `0` (see "When to escalate or stop"). Gone or renamed ⇒ stop and +re-measure. In particular the "no `supplementalGroups` needed" conclusion in `catalog/plex.yaml` and +`schemas/renderer-contract.md` is only true while that hook exists. ## When to escalate or stop - **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be - **inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's - `cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its - group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces** - the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with - `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is + **inert**, and it would mislead the next person. The image's `cont-init.d` hook stats each + `/dev/dri` char device and `usermod -a -G`s the `plex` user into **that device's** group, inside + the container; the service then execs via `s6-setuidgid plex`, which **rebuilds** the process group + list from `/etc/group` rather than appending to it. Anything Kubernetes supplies is therefore discarded before Plex starts. `render.py --selftest` asserts that absence deliberately. +- **GID `992` is a hostPath-era number — do not expect it, and do not read the root group as a + defect.** Under the device plugin the injected `/dev/dri/renderD128` arrives **`root:root`**, so + the hook resolves GID **`0`** and adds `plex` to the **root group**. Measured 2026-10-02: + `Plex Media Server` runs with `Groups: 0 100 1000`, and `id plex` reports + `groups=1000(plex),0(root),100(users)`. Under the old hostPath the device was `root:render` + (GID 992 on these nodes) and the reading was `Groups: 44 100 992 1000`. **992 is now irrelevant**: + if any page, comment or older handoff tells you to expect it, that text is stale and the system + you are looking at is working. The root-group membership is an inherent property of the plugin's + `DeviceSpec` — it injects the host device with its host ownership — **not** a defect, and **not** a + reason to reintroduce `supplementalGroups`, which would still be a no-op for the reason in the + bullet above. - **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will reject it — and it would not help: `uid 0` failed the same `open()` in the 2026-09-30 measurement, because root does not bypass the cgroup device gate. @@ -390,7 +513,8 @@ hook exists. watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to lose the DB. - **Stop and get the operator** if the device cannot be opened from the pod as uid 1000 (cause #2 - confirmed) *under the current mechanism*. The next step is a platform change in + confirmed) *under the current mechanism* — note that this open **passed** on 2026-10-02 + (`RW-OPEN-OK` as uid 1000 via `s6-setuidgid plex`), so a failure now is a regression. The next step is a platform change in `platform/intel-gpu-plugin/`, with its own PR, not more poking at this workload. ## Known gaps, deliberately not fixed here @@ -413,10 +537,25 @@ Recorded so they are findable, and so nobody assumes they were overlooked: --- Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/` -The node measurements (all three prod nodes) and the in-container image measurements cited here were -taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…` -(`plexmediaserver 1.43.4.10903`). The `EPERM` open measurements, the `Groups: 44 100 992 1000` -reading and the `gpu.intel.com/i915: 1` allocatable/DaemonSet readings were taken **2026-10-01**. -Hardware transcoding has **still not been observed working**: the open test has not been re-run under -the device-plugin mechanism (it cannot be, until that merges and Flux reconciles), and the Plex-side -toggle is still unset. + +**Hardware transcoding is CONFIRMED WORKING on prod, measured 2026-10-02.** From +`Plex Media Server.log` in the running pod: API `vaapi` on `/dev/dri/renderD128` +(`Intel CoffeeLake-S GT2 [UHD Graphics 630]`), `using hardware decode accelerator vaapi`, +`zero-copy support present`, `using zero-copy transcoding`, and +`final decoder: vaapi, final encoder: vaapi`. Also measured in the same pod: `RW-OPEN-OK` opening +`/dev/dri/renderD128` as uid 1000 via `s6-setuidgid plex`; the pod spec carrying +`{"limits":{"gpu.intel.com/i915":"1"},"requests":{"gpu.intel.com/i915":"1"}}`; `Groups: 0 100 1000` +and `id plex` → `groups=1000(plex),0(root),100(users)`; and `HardwareDevicePath` present in +`Preferences.xml` while `HardwareAcceleratedCodecs` was absent from all 27 attributes. + +Earlier readings, kept for provenance and **superseded** where they conflict: the node measurements +(all three prod nodes) and the in-container image measurements taken **2026-09-30** against +`plexinc/pms-docker` digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`); the hostPath +`EPERM` open results and the `Groups: 44 100 992 1000` / GID-992 group reading, **2026-10-01, under +the hostPath mechanism that no longer exists**; and the `gpu.intel.com/i915: 1` allocatable and +`intel-gpu-plugin` DaemonSet readings, **2026-10-01**. + +What this page still does **not** establish: that any particular title's software fallback was caused +by its source codec (cause 6 is an unverified hypothesis), and that Plex's dashboard *ever* labels a +session `Transcode (hw)` on this deployment — it has not been observed doing so, which is exactly why +the log is now the primary check.