Files
home-network-k3s-impl/decisions/2026-10-02-plex-quicksync-verification.md
T
jakman85andClaude Opus 5 1677dff568
gate / gate (pull_request) Failing after 1m46s
gate / conformance (pull_request) Has been skipped
fix(runbooks): Plex hw transcoding WORKS — the dashboard was never the proof
Hardware transcoding is confirmed working on prod (2026-10-02): VAAPI for both
decode and encode on /dev/dri/renderD128, with zero-copy, measured in
`Plex Media Server.log` in the running pod. The device-plugin mechanism from #37
works end to end.

`runbooks/plex-quicksync.md` said the Plex dashboard showing `Transcode (hw)` was
"the *only* proof available, and nothing else will ever tell you". That is false:
Dashboard -> Activity showed nothing while the feature was fully working, so the
page's authoritative check was silent on a healthy system and sent the operator
hunting a fault that did not exist.

Three inference errors corrected, two kept as named false leads:

1. The LOG is the primary check, not the dashboard. The grep ships in its
   REDACTING form -- that log carries API URLs with `X-Plex-Token` (CLAUDE.md §6).
   Also recorded: `Plex Transcoder Statistics.log` is the WRONG file (XML
   <Variant/> records, no ffmpeg output), and why `grep -a` is required. The
   dashboard is kept but demoted to best-effort, with its silence stated to prove
   nothing.
2. An fd snapshot is not evidence. A /proc/*/fd sweep found no holder of
   renderD128 while hardware transcoding was active -- sessions are short and the
   scan landed in a gap. The reliable process-level check is the transcoder's
   argv (-hwaccel, -hwaccel_device, vaapi, renderD128).
3. A missing preference key means nothing. `HardwareAcceleratedCodecs` was absent
   from all 27 attributes with the toggle on and working, while
   `HardwareDevicePath` was written and the log said "hardware transcoding:
   enabled". Only `="0"` is conclusive; the log settles it.

Also stale and now fixed: GID 992 is a hostPath-era number. Under the plugin the
injected device is root:root, so the image's hook resolves GID 0 and adds `plex`
to the root group (`Groups: 0 100 1000`, `groups=1000(plex),0(root),100(users)`).
Marked as an inherent property of the plugin's DeviceSpec, not a defect, and
explicitly not a reason to reintroduce supplementalGroups (still a no-op --
s6-setuidgid rebuilds the group list from /etc/group). Added a per-title fallback
entry (UHD 630 has no AV1 decode), labelled a HYPOTHESIS: the title that fell
back was never checked. `runbooks/README.md`'s index entry was stale the same two
ways and is corrected.

Unchanged on purpose: the "STOP HERE IF PLEX IS Pending" section, the escape
hatch, and the no-alert-is-possible reasoning (no metricsEndpoint, no
ServiceMonitor, nothing a rule could evaluate).

Documentation only -- no catalog/, schemas/, apps/ or platform/ change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-02 00:49:53 -05:00

9.7 KiB

date, branch, title
date branch title
2026-10-02 fix/plex-quicksync-verification Hardware transcoding works — and `runbooks/plex-quicksync.md`'s verification section was wrong

What changed

Hardware transcoding is confirmed working on prod. Measured 2026-10-02 from Plex Media Server.log in the running pod: vaapi on /dev/dri/renderD128 (Intel CoffeeLake-S GT2 [UHD Graphics 630]), using hardware decode accelerator vaapi, zero-copy support present, using zero-copy transcoding, and final decoder: vaapi, final encoder: vaapi. VAAPI for both directions, zero-copy, so frames never round-trip through the CPU. The device-plugin mechanism from PR #37 works end to end.

And runbooks/plex-quicksync.md's verification section was wrong, which is the correction this branch exists to publish. The page said the Plex dashboard showing Transcode (hw) was "the only proof available, and nothing else will ever tell you". The operator had hardware transcoding fully working while Dashboard → Activity showed nothing of the sort — so the page's authoritative check was silent on a working system, and sent someone looking for a fault that did not exist. Three inference errors are now corrected on the page, two of them recorded as named false leads so the next reader does not re-walk them:

  1. The dashboard is not proof. Replaced as the primary check by Plex's own log — the grep is in the page, in its redacting form, because Plex Media Server.log carries API URLs containing X-Plex-Token (CLAUDE.md §6; a credential has been leaked into a transcript here before). The page also now says explicitly that Plex Transcoder Statistics.log is the wrong file — it holds XML <Variant .../> records, not ffmpeg output, and the name invites the mistake — and why grep -a is required. The dashboard is kept but demoted to a secondary, best-effort signal, with its absence stated to prove nothing.
  2. An fd snapshot is not evidence. A /proc/*/fd sweep inside the pod found no holder of renderD128 while hardware transcoding was active; transcode sessions are short and the scan landed in a gap. Recorded as a documented false lead, with the reliable process-level check in its place: the transcoder's argv (-hwaccel, -hwaccel_device, vaapi, renderD128), found via ps -eo pid,user,comm | grep -i transcoder.
  3. A missing preference key does not mean the setting is off. The page told readers to treat an absent HardwareAcceleratedCodecs in Preferences.xml as "never enabled". With hardware acceleration demonstrably on and working, that key was absent from all 27 attributes while HardwareDevicePath was written and the log said hardware transcoding: enabled. First-command #4 now reads both keys and states plainly that HardwareDevicePath present is a positive signal, absence of HardwareAcceleratedCodecs means nothing either way, only ="0" is conclusive, and the log settles it.

Also on the page:

  • GID 992 is no longer presented as a number to expect. Under the plugin the injected device arrives root:root, so the image's cont-init.d hook resolves GID 0 and adds plex to the root group — measured Groups: 0 100 1000, id plex → groups=1000(plex),0(root),100(users). 992 was the hostPath-era host render GID. The page now says so, marks the root-group membership as an inherent property of the plugin's DeviceSpec rather than a defect, and repeats that it is not a reason to reintroduce supplementalGroups (still a no-op: s6-setuidgid rebuilds the group list from /etc/group). The stale Groups: 100 1000 claim in the escalation section is gone.
  • A new troubleshooting entry for legitimate per-title fallbacks, marked as a hypothesis. Between two fully hardware sessions, one logged enabled, but no hardware decode accelerator found and empty final decoder/encoder. UHD 630 decodes H.264, HEVC 8/10-bit and VP9 but not AV1 and not 10-bit 4:2:2 HEVC, so the likely cause is the source file, not the platform — explicitly written as a lead to check first, not a confirmed diagnosis of that item.
  • The status is stated at the top of the page and in a rewritten provenance footer that separates the 2026-10-02 confirmations from the superseded 2026-09-30/2026-10-01 hostPath-era readings.
  • runbooks/README.md's index entry for the page was stale in the same two ways (it still described a /dev/dri hostPath and the dashboard-only proof) and is corrected to match.

Unchanged on purpose: the "STOP HERE IF PLEX IS Pending" section and the escape hatch (delete the hardware: block from catalog/plex.yaml, re-render) keep exactly their existing prominence — they are correct and they are the 2am material. The "no alert is possible" reasoning is also kept verbatim in substance: Plex declares no metricsEndpoint, renders no ServiceMonitor, and a fallback alert would be a rule evaluating against nothing.

Documentation only. No change to catalog/, schemas/, apps/ or platform/.

What I verified

  • Gate, in the pinned toolbox from this worktree (DEV_KUBECONFIG=/dev/null, ENGINE shim bind-mounting the main checkout's .git): rc 3 UNDETERMINED, diff base origin/main @ 948875aad37d, with exactly the three expected skips and no failures. Green: decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped: spec-immutability (spec repo unavailable → R-TEST-06 not verified), gitea-settings (SETTINGS_API_TOKEN unset), conformance (CLUSTER_AVAILABLE!=true → L2/L3 not run). Touched requirements: R-OBS-04, reported UNVERIFIED for lack of a cluster. approval-policy passed as N/A locally (event='none', not a pull_request) — it evaluates for real on the PR. rc 3 is not green, it is the calibrated worktree result, and it is reported here as rc 3.
  • The gate was run three times, with the same rc 3 / three-skip result on each committed tip — including the final one. Worth recording from the first attempt: run before committing, with HEAD == origin/main, the touched-requirement set is NOT DETERMINABLE (on-base) and the R-TEST-03 coverage gate cannot be applied at all. Commit before gating from a fresh worktree branch.
  • runbooks/kustomization.yaml still carries - plex-quicksync=plex-quicksync.md in the configMapGenerator files: list (line 110) — checked explicitly, because without it the page merges and still 404s. README=README.md is likewise present, which is what serves the corrected index entry.
  • This change touches no sensitive_paths. infra/gitea/approval-policy.json lists ci/**, .gitea/**, conformance/**, clusters/**, bootstrap/**, infra/gitea/**, platform/kyverno/**, platform-config/kustomization.yaml, .sops.yaml, **/*.sops.yaml, **/rbac.yaml, CLAUDE.md. runbooks/** and decisions/** are not among them, and the approval-policy stage passed.
  • No YAML was added or edited, so indent-sequences: consistent (which blocked a sibling branch today) is not in play here; lint passed over the full tracked YAML set regardless.
  • DECISIONS-LOG.md is untouched — this handoff is its own file, per CLAUDE.md §7.
  • The repository-wide grep for 992 to find what else still asserts it: remaining occurrences are in catalog/plex.yaml, catalog/render.py, schemas/renderer-contract.md, platform/intel-gpu-plugin/README.md, platform/intel-gpu-plugin/kustomization.yaml, and three older decisions/ entries.

What I did NOT verify

  • I did not re-measure anything on the live cluster in this session. Every platform number on the page — the VAAPI/zero-copy log lines, RW-OPEN-OK as uid 1000, the gpu.intel.com/i915 limits and requests on the pod spec, Groups: 0 100 1000, id plex, the 27 Preferences.xml attributes, the empty /proc/*/fd sweep — came to me as measurements taken earlier on 2026-10-02 and was transcribed, not re-run. Per CLAUDE.md §2b, treat the page as reporting those readings rather than as proof of current state; re-run the primary check before relying on it.
  • The per-title fallback cause is unverified and labelled as such. The source codec of the title that logged no hardware decode accelerator found was never read. The AV1 / 10-bit-4:2:2-HEVC explanation is inferred from UHD 630's capability set alone.
  • Whether Plex's dashboard ever shows Transcode (hw) on this deployment is unknown. It was observed silent during working hardware transcoding; whether that is a client/version quirk, a reporting bug, or something about this setup was not investigated. The page therefore demotes it rather than explaining it.
  • The commands as written in the page were not executed from the page. They are the forms that were run, re-wrapped for dev.sh; in particular the exec -it deploy/plex -- sh route for the argv check is noted as possibly lacking a TTY under dev.sh run and was not exercised in that shape.
  • Stale 992 copies outside runbooks/ were left alone, deliberately — scope was documentation only, and catalog/plex.yaml, catalog/render.py, schemas/renderer-contract.md and platform/intel-gpu-plugin/ all still carry the hostPath-era Groups: 44 100 992 1000 reading as if current. render.py --selftest asserts around that comment, so correcting it is a real change with gate consequences, not a doc edit. That is open work. Older decisions/ entries also cite 992; those are historical records and should stay as written.
  • CI on the PR. The branch is pushed and the PR opened; I did not watch the Gitea run report, so nothing here should be read as "CI green". The only gate result I can quote is the local rc 3 above.