fix(runbooks): Plex hw transcoding WORKS — the dashboard was never the proof #39

2026-10-02T05:51:16Z

gitea-admin commented

2026-10-02 05:51:16 +00:00

The good news: hardware transcoding works

Confirmed on prod, 2026-10-02, from Plex Media Server.log in the running pod:

Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD128'
                              (Intel CoffeeLake-S GT2 [UHD Graphics 630])
hardware transcoding: using hardware decode accelerator vaapi
hardware transcoding: zero-copy support present
hardware transcoding: using zero-copy transcoding
hardware transcoding: final decoder: vaapi, final encoder: vaapi

VAAPI for both decode and encode, with zero-copy — frames never round-trip through the CPU.
The device-plugin mechanism from #37 works end to end.

The bad news: the runbook's verification section was wrong

runbooks/plex-quicksync.md said the Plex dashboard showing Transcode (hw) was "the only proof
available, and nothing else will ever tell you". That is false. The operator had hardware
transcoding fully working while Dashboard → Activity showed nothing of the sort. A check that stays
silent on a healthy system does not merely fail to prove anything — it sends someone hunting a fault
that does not exist, and it did.

This PR is documentation only. No change to catalog/, schemas/, apps/ or platform/.

Error 1 — the dashboard is not proof. The log is.

The primary check is now:

./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
  grep -aiE "hardware transcoding: (final|using)" \
  "/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log" \
  | sed -E "s/(X-Plex-Token|token)=[^&[:space:]]*/\1=REDACTED/g" | tail -5'

final decoder: vaapi, final encoder: vaapi is the pass condition; both fields empty means it fell
back to software; using zero-copy transcoding is the best case.

Three things the page now insists on:

The sed is not optional. That log carries API URLs containing X-Plex-Token, a live plex.tv
credential. CLAUDE.md §6, and a credential has been leaked into a transcript in this repo exactly
this way. The redacting form is the only form shown.
Plex Transcoder Statistics.log is the WRONG file — XML <Variant .../> records, no ffmpeg
output, no decoder/encoder verdict. Someone will reach for it because of the name, so the page says
so outright.
grep -a is required: without it the occasional non-UTF-8 byte makes GNU grep print
"binary file matches" and nothing else, which reads exactly like "no hardware transcoding".

The dashboard is kept but demoted to a secondary, best-effort signal, marked unreliable, with its
absence stated to prove nothing.

The other half of that section was right and is unchanged: Plex declares no metricsEndpoint,
renders no ServiceMonitor, and so no alert is possible — a fallback rule would evaluate against
nothing, which is a failure mode this repo has already been burned by at scale.

Error 2 — an fd snapshot is not evidence either

A /proc/*/fd sweep inside the pod found no holder of renderD128 while hardware transcoding
was active. Transcode sessions are short; the scan landed in a gap. Recorded as a documented false
lead, because it is an attractive wrong turn, with the reliable process-level check in its place —
the transcoder's argv:

ps -eo pid,user,comm | grep -i transcoder
tr '\0' ' ' < /proc/<transcoder-pid>/cmdline | grep -oE '(-hwaccel[a-z_]*|vaapi|renderD[0-9]+)'

Error 3 — a missing preference key does not mean the setting is off

The page told readers to treat an absent HardwareAcceleratedCodecs in Preferences.xml as "never
enabled". With hardware acceleration demonstrably on and working, that key was absent from all 27
attributes, while HardwareDevicePath was written and the log said hardware transcoding: enabled. First-command #4 now reads both keys: HardwareDevicePath present is a positive signal,
absence of HardwareAcceleratedCodecs means nothing either way, only ="0" is conclusive, and the
log settles it. The correct statements are kept — no PLEX_PREFERENCE_* env var can set the toggle,
and the UI path is Settings → Transcoder → Show Advanced → "Use hardware acceleration when available".

Also stale: GID 992

992 was the hostPath-era host render GID. Under the plugin the injected device arrives root:root,
so the image's hook resolves GID 0 and adds plex to the root group — measured
Groups: 0 100 1000, and id plex → groups=1000(plex),0(root),100(users). The page now says 992 is
irrelevant (a reader expecting it would think a working system was broken), marks the root-group
membership as an inherent property of the plugin's DeviceSpec rather than a defect, and repeats that
it is not a reason to reintroduce supplementalGroups — still a no-op, because s6-setuidgid
rebuilds the group list from /etc/group. The stale Groups: 100 1000 claim is gone.

New: expect occasional legitimate per-title fallbacks

Between two fully hardware sessions, one logged enabled, but no hardware decode accelerator found
with empty final decoder/encoder. UHD 630 decodes H.264, HEVC 8/10-bit and VP9 but not AV1, and
not 10-bit 4:2:2 HEVC — so if fallback happens on particular files rather than all of them, check
the source codec before suspecting the device. Written as a hypothesis from the hardware's
capability set, not a confirmed diagnosis — that title's codec was never read, and the page says so.

Kept exactly as prominent

The "STOP HERE IF PLEX IS Pending" section and the escape hatch (delete the hardware: block
from catalog/plex.yaml, re-render) are untouched. They are correct and they are the 2am material.

runbooks/README.md's index entry was stale in the same two ways (still described a /dev/dri
hostPath and the dashboard-only proof) and is corrected to match.

Gate

DEV_KUBECONFIG=/dev/null ENGINE=<shim> ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
→ rc 3 UNDETERMINED, which is the calibrated worktree result and is not green:

stage	result
`spec-immutability`	SKIP — spec repo not checked out beside this one (R-TEST-06 not verified)
`decisions-log`	green
`lint`	green
`l0`	green
`render-golden`	green
`l1`	green (touched requirements: `R-OBS-04`)
`approval-policy`	green (`N/A` locally — `event='none'`; it evaluates for real on this PR)
`gitea-settings`	SKIP — `SETTINGS_API_TOKEN` unset in the container
`conformance`	SKIP — `CLUSTER_AVAILABLE!=true`, L2/L3 not run

Three skips, no failures — the expected set. R-OBS-04 is reported UNVERIFIED for want of a
cluster.

Also checked, because it has bitten before: runbooks/kustomization.yaml still carries
- plex-quicksync=plex-quicksync.md in the configMapGenerator files: list, so the page will
actually serve rather than 404 after merge.

What is NOT verified

Nothing was re-measured on the live cluster in this session. Every platform number on the page
was transcribed from measurements taken earlier today, not re-run (CLAUDE.md §2b).
The per-title fallback's cause — that title's source codec was never read.
Whether Plex's dashboard ever labels a session Transcode (hw) on this deployment. It was
observed silent during working hardware transcoding; the page demotes it rather than explaining it.
Stale 992 copies outside runbooks/ were deliberately left alone (out of scope here):
catalog/plex.yaml, catalog/render.py, schemas/renderer-contract.md,
platform/intel-gpu-plugin/README.md and platform/intel-gpu-plugin/kustomization.yaml all still
carry Groups: 44 100 992 1000 as if current, and render.py --selftest asserts around that
comment — so correcting it is a real change with gate consequences, not a doc edit. Open work.

Handoff: decisions/2026-10-02-plex-quicksync-verification.md. DECISIONS-LOG.md untouched.

CI on this PR is RED — and it is not this branch

Read from run 208's own gate log, not the tick (CLAUDE.md §2b). Green: decisions-log, lint,
l0, render-golden, l1, and approval-policy — which evaluated for real on the pull_request
event and passed, since this touches no sensitive path. The single failure is gitea-settings:

[fail]   required_approvals: DECLARED 1 but LIVE is 0
[fail] Gitea has DRIFTED from infra/gitea/expected-branch-protection.json.
[fail] MERGE GATE: BLOCKED

Every other declared setting matched (block_admin_merge_override, block_on_rejected_reviews,
dismiss_stale_approvals, enable_status_check, enable_push=false,
status_check_contexts = ["gate / gate (pull_request)"], runners online: 1). Live main
currently requires zero approvals while the declaration says 1, and REQUIRE_GITEA_SETTINGS: "true"
in .gitea/workflows/gate.yml makes that drift block every PR in the repo. Note the shape of it:
the check that would notice approvals are unenforced is the check that is failing.

Deliberately not fixed here — infra/gitea/** is a sensitive path needing a non-author approval, and
this PR is documentation only. Re-asserting the protection (infra/gitea/require-gate.sh) or changing
the declaration belongs in its own PR; fix/admin-approval-override (616a497, "let an administrator
merge their own green PR") looks like that work in flight.

🤖 Generated with Claude Code

## The good news: hardware transcoding works Confirmed on prod, **2026-10-02**, from `Plex Media Server.log` in the running pod: ``` Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD128' (Intel CoffeeLake-S GT2 [UHD Graphics 630]) hardware transcoding: using hardware decode accelerator vaapi hardware transcoding: zero-copy support present hardware transcoding: using zero-copy transcoding hardware transcoding: final decoder: vaapi, final encoder: vaapi ``` VAAPI for **both** decode and encode, with **zero-copy** — frames never round-trip through the CPU. The device-plugin mechanism from #37 works end to end. ## The bad news: the runbook's verification section was wrong `runbooks/plex-quicksync.md` said the Plex dashboard showing `Transcode (hw)` was *"the only proof available, and nothing else will ever tell you"*. **That is false.** The operator had hardware transcoding fully working while Dashboard → Activity showed nothing of the sort. A check that stays silent on a healthy system does not merely fail to prove anything — it sends someone hunting a fault that does not exist, and it did. This PR is **documentation only**. No change to `catalog/`, `schemas/`, `apps/` or `platform/`. ### Error 1 — the dashboard is not proof. The log is. The primary check is now: ```bash ./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \ grep -aiE "hardware transcoding: (final|using)" \ "/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log" \ | sed -E "s/(X-Plex-Token|token)=[^&[:space:]]*/\1=REDACTED/g" | tail -5' ``` `final decoder: vaapi, final encoder: vaapi` is the pass condition; both fields empty means it fell back to software; `using zero-copy transcoding` is the best case. Three things the page now insists on: - **The `sed` is not optional.** That log carries API URLs containing `X-Plex-Token`, a live plex.tv credential. CLAUDE.md §6, and a credential has been leaked into a transcript in this repo exactly this way. The redacting form is the only form shown. - **`Plex Transcoder Statistics.log` is the WRONG file** — XML `<Variant .../>` records, no ffmpeg output, no decoder/encoder verdict. Someone will reach for it because of the name, so the page says so outright. - **`grep -a` is required**: without it the occasional non-UTF-8 byte makes GNU grep print "binary file matches" and nothing else, which reads exactly like "no hardware transcoding". The dashboard is **kept but demoted** to a secondary, best-effort signal, marked unreliable, with its absence stated to prove nothing. The *other* half of that section was right and is unchanged: Plex declares no `metricsEndpoint`, renders no ServiceMonitor, and so no alert is possible — a fallback rule would evaluate against nothing, which is a failure mode this repo has already been burned by at scale. ### Error 2 — an fd snapshot is not evidence either A `/proc/*/fd` sweep inside the pod found **no** holder of `renderD128` while hardware transcoding was active. Transcode sessions are short; the scan landed in a gap. Recorded as a **documented false lead**, because it is an attractive wrong turn, with the reliable process-level check in its place — the transcoder's **argv**: ```bash ps -eo pid,user,comm | grep -i transcoder tr '\0' ' ' < /proc/<transcoder-pid>/cmdline | grep -oE '(-hwaccel[a-z_]*|vaapi|renderD[0-9]+)' ``` ### Error 3 — a missing preference key does not mean the setting is off The page told readers to treat an absent `HardwareAcceleratedCodecs` in `Preferences.xml` as "never enabled". With hardware acceleration demonstrably on and working, that key was **absent from all 27 attributes**, while `HardwareDevicePath` **was** written and the log said `hardware transcoding: enabled`. First-command #4 now reads both keys: `HardwareDevicePath` present is a positive signal, absence of `HardwareAcceleratedCodecs` means nothing either way, only `="0"` is conclusive, and the log settles it. The correct statements are kept — no `PLEX_PREFERENCE_*` env var can set the toggle, and the UI path is Settings → Transcoder → Show Advanced → "Use hardware acceleration when available". ### Also stale: GID 992 992 was the hostPath-era host `render` GID. Under the plugin the injected device arrives `root:root`, so the image's hook resolves GID **0** and adds `plex` to the **root group** — measured `Groups: 0 100 1000`, and `id plex` → `groups=1000(plex),0(root),100(users)`. The page now says 992 is irrelevant (a reader expecting it would think a working system was broken), marks the root-group membership as an inherent property of the plugin's `DeviceSpec` rather than a defect, and repeats that it is **not** a reason to reintroduce `supplementalGroups` — still a no-op, because `s6-setuidgid` rebuilds the group list from `/etc/group`. The stale `Groups: 100 1000` claim is gone. ### New: expect occasional legitimate per-title fallbacks Between two fully hardware sessions, one logged `enabled, but no hardware decode accelerator found` with empty final decoder/encoder. UHD 630 decodes H.264, HEVC 8/10-bit and VP9 but **not AV1**, and not 10-bit 4:2:2 HEVC — so if fallback happens on *particular files* rather than all of them, check the source codec before suspecting the device. **Written as a hypothesis from the hardware's capability set, not a confirmed diagnosis** — that title's codec was never read, and the page says so. ### Kept exactly as prominent The **"STOP HERE IF PLEX IS `Pending`"** section and the escape hatch (delete the `hardware:` block from `catalog/plex.yaml`, re-render) are untouched. They are correct and they are the 2am material. `runbooks/README.md`'s index entry was stale in the same two ways (still described a `/dev/dri` hostPath and the dashboard-only proof) and is corrected to match. ## Gate `DEV_KUBECONFIG=/dev/null ENGINE=<shim> ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'` → **rc 3 UNDETERMINED**, which is the calibrated worktree result and **is not green**: | stage | result | |---|---| | `spec-immutability` | SKIP — spec repo not checked out beside this one (R-TEST-06 not verified) | | `decisions-log` | **green** | | `lint` | **green** | | `l0` | **green** | | `render-golden` | **green** | | `l1` | **green** (touched requirements: `R-OBS-04`) | | `approval-policy` | **green** (`N/A` locally — `event='none'`; it evaluates for real on this PR) | | `gitea-settings` | SKIP — `SETTINGS_API_TOKEN` unset in the container | | `conformance` | SKIP — `CLUSTER_AVAILABLE!=true`, L2/L3 not run | Three skips, no failures — the expected set. `R-OBS-04` is reported `UNVERIFIED` for want of a cluster. Also checked, because it has bitten before: `runbooks/kustomization.yaml` still carries `- plex-quicksync=plex-quicksync.md` in the `configMapGenerator` `files:` list, so the page will actually serve rather than 404 after merge. ## What is NOT verified - **Nothing was re-measured on the live cluster in this session.** Every platform number on the page was transcribed from measurements taken earlier today, not re-run (CLAUDE.md §2b). - The per-title fallback's cause — that title's source codec was never read. - Whether Plex's dashboard *ever* labels a session `Transcode (hw)` on this deployment. It was observed silent during working hardware transcoding; the page demotes it rather than explaining it. - **Stale `992` copies outside `runbooks/` were deliberately left alone** (out of scope here): `catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md`, `platform/intel-gpu-plugin/README.md` and `platform/intel-gpu-plugin/kustomization.yaml` all still carry `Groups: 44 100 992 1000` as if current, and `render.py --selftest` asserts around that comment — so correcting it is a real change with gate consequences, not a doc edit. **Open work.** Handoff: `decisions/2026-10-02-plex-quicksync-verification.md`. `DECISIONS-LOG.md` untouched. ## CI on this PR is RED — and it is not this branch Read from run 208's own gate log, not the tick (CLAUDE.md §2b). Green: `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, and `approval-policy` — which evaluated for real on the `pull_request` event and passed, since this touches no sensitive path. The single failure is **`gitea-settings`**: ``` [fail] required_approvals: DECLARED 1 but LIVE is 0 [fail] Gitea has DRIFTED from infra/gitea/expected-branch-protection.json. [fail] MERGE GATE: BLOCKED ``` Every other declared setting matched (`block_admin_merge_override`, `block_on_rejected_reviews`, `dismiss_stale_approvals`, `enable_status_check`, `enable_push=false`, `status_check_contexts = ["gate / gate (pull_request)"]`, `runners online: 1`). **Live `main` currently requires zero approvals** while the declaration says 1, and `REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml` makes that drift block **every** PR in the repo. Note the shape of it: the check that would notice approvals are unenforced is the check that is failing. Deliberately not fixed here — `infra/gitea/**` is a sensitive path needing a non-author approval, and this PR is documentation only. Re-asserting the protection (`infra/gitea/require-gate.sh`) or changing the declaration belongs in its own PR; `fix/admin-approval-override` (`616a497`, "let an administrator merge their own green PR") looks like that work in flight. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-02 05:51:16 +00:00

fix(runbooks): Plex hw transcoding WORKS — the dashboard was never the proof

gate / gate (pull_request) Failing after 1m46s

Details

gate / conformance (pull_request) Has been skipped

Details

1677dff568

Hardware transcoding is confirmed working on prod (2026-10-02): VAAPI for both
decode and encode on /dev/dri/renderD128, with zero-copy, measured in
`Plex Media Server.log` in the running pod. The device-plugin mechanism from #37
works end to end.

`runbooks/plex-quicksync.md` said the Plex dashboard showing `Transcode (hw)` was
"the *only* proof available, and nothing else will ever tell you". That is false:
Dashboard -> Activity showed nothing while the feature was fully working, so the
page's authoritative check was silent on a healthy system and sent the operator
hunting a fault that did not exist.

Three inference errors corrected, two kept as named false leads:

1. The LOG is the primary check, not the dashboard. The grep ships in its
   REDACTING form -- that log carries API URLs with `X-Plex-Token` (CLAUDE.md §6).
   Also recorded: `Plex Transcoder Statistics.log` is the WRONG file (XML
   <Variant/> records, no ffmpeg output), and why `grep -a` is required. The
   dashboard is kept but demoted to best-effort, with its silence stated to prove
   nothing.
2. An fd snapshot is not evidence. A /proc/*/fd sweep found no holder of
   renderD128 while hardware transcoding was active -- sessions are short and the
   scan landed in a gap. The reliable process-level check is the transcoder's
   argv (-hwaccel, -hwaccel_device, vaapi, renderD128).
3. A missing preference key means nothing. `HardwareAcceleratedCodecs` was absent
   from all 27 attributes with the toggle on and working, while
   `HardwareDevicePath` was written and the log said "hardware transcoding:
   enabled". Only `="0"` is conclusive; the log settles it.

Also stale and now fixed: GID 992 is a hostPath-era number. Under the plugin the
injected device is root:root, so the image's hook resolves GID 0 and adds `plex`
to the root group (`Groups: 0 100 1000`, `groups=1000(plex),0(root),100(users)`).
Marked as an inherent property of the plugin's DeviceSpec, not a defect, and
explicitly not a reason to reintroduce supplementalGroups (still a no-op --
s6-setuidgid rebuilds the group list from /etc/group). Added a per-title fallback
entry (UHD 630 has no AV1 decode), labelled a HYPOTHESIS: the title that fell
back was never checked. `runbooks/README.md`'s index entry was stale the same two
ways and is corrected.

Unchanged on purpose: the "STOP HERE IF PLEX IS Pending" section, the escape
hatch, and the no-alert-is-possible reasoning (no metricsEndpoint, no
ServiceMonitor, nothing a rule could evaluate).

Documentation only -- no catalog/, schemas/, apps/ or platform/ change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

gitea-admin referenced this issue from a commit

2026-10-02 06:02:58 +00:00

docs(decisions): record the PR's measured CI result — gitea-settings has drifted

gitea-admin added 1 commit 2026-10-02 06:02:58 +00:00

docs(decisions): record the PR's measured CI result — gitea-settings has drifted

gate / gate (pull_request) Failing after 1m44s

Details

gate / conformance (pull_request) Has been skipped

Details

f0994c11a0

Read PR #39's own gate log rather than the tick (CLAUDE.md §2b). Every stage
this branch could affect is green: decisions-log, lint, l0, render-golden, l1,
and approval-policy (which evaluated for real on the pull_request event and
passed — no sensitive path touched). The one failure is `gitea-settings`:

    required_approvals: DECLARED 1 but LIVE is 0
    Gitea has DRIFTED from infra/gitea/expected-branch-protection.json

Every other declared setting matched, and one runner is online. So live `main`
currently requires ZERO approvals while the declaration says 1, and because
.gitea/workflows/gate.yml sets REQUIRE_GITEA_SETTINGS=true, that drift blocks
EVERY PR in the repo, not only this one. The check that would notice approvals
are unenforced is the check that is failing.

Not touched here: infra/gitea/** is a sensitive path (non-author approval
required) and outside this branch's documentation-only scope. Re-asserting the
protection or changing the declaration is an operator decision in its own PR;
local branch fix/admin-approval-override (616a497) looks like that work in
flight.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 approved these changes

2026-10-02 06:13:59 +00:00

jakman85 added 1 commit 2026-10-02 06:37:05 +00:00

Merge branch 'main' into fix/plex-quicksync-verification

gate / gate (pull_request) Successful in 1m46s

Details

gate / conformance (pull_request) Has been skipped

Details

0d7f94098b

jakman85 merged commit bcb8931b4f into main

2026-10-02 06:43:04 +00:00

jakman85 referenced this issue from a commit