Merge remote-tracking branch 'origin/main' into feat/oauth2-proxy-auth-failure-visible

# Conflicts:
#	runbooks/kustomization.yaml

runbooks/README.md

@@ -135,6 +135,22 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
   plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
   token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
   Includes why a changed Secret does not restart anything, and why an expired token is harmless.
+- **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod
+  (`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware
+  acceleration when available"**, and no environment variable can do it (the image supports no
+  `PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`,
+  read out of the binary), says which widely-cited key **does not exist**, and is explicit that the
+  **only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no
+  alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin
+  fallback, and why a `:latest` pull can silently break it.
+- **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
+  re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
+  (the four lines that matter, including the upstream `-` that means the proxy answered a request the
+  app never saw). Covers the cookie collision between per-app sidecars sharing the default
+  `_oauth2_proxy` name on `.home.keithjackson.dev` — which deletes sessions belonging to apps in
+  *other* namespaces and is only legible with two logs side by side — the 5-minute access-token
+  session with no `--cookie-refresh`, and why an XHR answered with a 302 instead of a 401 loses a
+  POST body silently.
 - **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
   Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
   factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap

runbooks/kustomization.yaml

@@ -89,11 +89,25 @@ configMapGenerator:
       # that "is up but does not work" — the gate answers an unauthenticated XHR with a 302,
       # not a 401, so the failure arrives as an HTTP 200 carrying the wrong body.
       - oauth2-proxy-auth-failures=oauth2-proxy-auth-failures.md
+      # Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
+      # Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
+      # presents as random session loss in ONE app's log and is only legible with both side by side,
+      # so the page has to be reachable at 3am without going looking for it in git.
+      # SIBLING PAGE to oauth2-proxy-auth-failures above, and they are deliberately both here: this
+      # one is the SESSION-LIFECYCLE procedure (why a session died), that one is the DIAGNOSTIC
+      # TRIAGE (which of the three layers refused, and how to tell a 302-to-200 from a real success).
+      - oauth2-proxy-session=oauth2-proxy-session.md
       # Operator procedure, not an alert page (no alert links to it — the
       # alert-runbooks-resolvable check treats that as advisory). It is served
       # because the plex.tv claim window is short and is not a thing to go looking
       # for in git: /runbooks/plex-claim has to resolve the moment it is needed.
       - plex-claim=plex-claim.md
+      # Operator procedure, not an alert page, and NO alert is possible for it: plex renders no
+      # metricsEndpoint, so there is no ServiceMonitor, no transcode-mode metric and nothing a rule
+      # could evaluate. Served because the feature it completes is INERT until a human ticks a box in
+      # the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped
+      # feature stays switched off for months.
+      - plex-quicksync=plex-quicksync.md
       - pvc-capacity=pvc-capacity.md
       - targets-down=targets-down.md
       - update-rollback=update-rollback.md

runbooks/oauth2-proxy-session.md

@@ -0,0 +1,195 @@
+# oauth2-proxy-session — telling the auth layers apart from the sidecar log
+
+**Not alert-driven.** This is an operator procedure for the per-app oauth2-proxy sidecars
+(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`, `crafty`) and the shared forward-auth
+proxy in namespace `oauth2-proxy`. Use it when a gated UI behaves as though you are signed in and
+signed out at the same time: the page loads, but its buttons and saves quietly do nothing.
+
+> The sidecar log contains **email addresses, session UUIDs and full callback URLs including an
+> OAuth `code`**. Read it, do not paste it. (Repo rule §6: secrets never enter a transcript.)
+
+## What this means here
+
+Every gated app on `home.keithjackson.dev` has **at least two** independent authentications
+stacked on it, and the symptom rarely names the one that refused:
+
+1. **Keycloak** — the `homelab` realm issues the token. Failures here are visible: you see a login
+   form, or an error page from Keycloak's own hostname.
+2. **The oauth2-proxy sidecar** — validates the session cookie and the group claim
+   (`media-users` / `platform-admins`). **This is the layer that fails silently**, and the rest of
+   this page is about reading it.
+3. **The app's own login**, where it has one — Crafty and Radarr/Sonarr each present a local login
+   *behind* the Keycloak gate. See `runbooks/crafty-admin-credential.md` for that layer.
+
+The sidecar's access log distinguishes all three, if you know the four lines to look for. Get it:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
+      -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1'
+
+Substitute the namespace and label for the app you are chasing. Always pass `--context k3s-prod`:
+`dev.sh` merges both clusters and picks a current-context **per invocation**, and that default has
+already changed once. Do not run two `dev.sh` invocations at the same time.
+
+### The four lines, and what each one means
+
+| Log line | What actually happened | Which layer |
+|---|---|---|
+| `[AuthSuccess] Authenticated via OAuth2: Session{... groups:[...]}` | Keycloak issued a token and the proxy accepted it. **The IdP is fine.** | Keycloak ✔ |
+| `[oauthproxy.go:1026] No valid authentication in request. Initiating login.` | **No session cookie was presented at all.** Either you never had one, or something *deleted* it — see below. | sidecar |
+| `[stored_session.go:94] Error loading cookied session: cookie signature not valid, removing session` | A cookie **of the right name** arrived that this proxy **could not decrypt**. Almost always a *different* proxy's cookie. Note `removing session`: this proxy then **deletes** it. | sidecar, cookie collision |
+| an upstream path of `-` instead of `/` in an access-log line | the **proxy answered the request itself**; it never reached the app. Compare `POST - "/api/..."` against `POST / "/api/..."`. | sidecar |
+
+That last column is the one people miss. The upstream field sits between the method and the quoted
+path. `-` means the proxy handled it (a redirect, a 401, its own `/oauth2/*` endpoints); `/` means
+it was proxied to the app. Count them:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n jackett logs \
+      -l app.kubernetes.io/name=jackett -c oauth2-proxy --tail=-1 \
+      | grep -v "\"/ping\"" | grep -E " (POST|PUT|DELETE) " | cut -c1-140'
+
+If your writes show `-`, the app never saw them. On 2026-09-30, **4 of the 6 POSTs** in jackett's
+log had `-`.
+
+## First commands
+
+**1. Did Keycloak ever succeed?** No `AuthSuccess` at all means this is not a cookie problem —
+go to `runbooks/keycloak-down.md` or `runbooks/keycloak-client-secrets.md`.
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
+      -l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 | grep -c AuthSuccess'
+
+**2. How long does a session last?** Subtract `created` from `expires` on an `AuthSuccess` line.
+If it is exactly **5m**, the session's lifetime is Keycloak's access-token lifetime and the proxy
+is not refreshing it:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> logs \
+      -l app.kubernetes.io/name=<app> -c oauth2-proxy --tail=-1 \
+      | grep AuthSuccess | tail -1 | grep -o "created:[^ ]* [^ ]*\|expires:[^ ]* [^ ]*"'
+
+The same line says `refresh_token:true` — the proxy is *holding* a refresh token. It will only
+spend it if `--cookie-refresh` is set; the default is `0`, meaning disabled.
+
+**3. Is this a cookie collision?** Two checks, and you want both.
+
+*Names and scopes* — every instance that emits the default name `_oauth2_proxy` on the shared
+`.home.keithjackson.dev` domain is in the collision set with every other one:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pods -A -o json \
+      | grep -E "cookie-name|cookie-domain"'
+
+An instance with **no** `--cookie-name` argument is using the default. That is the failure — the
+flag being absent, not wrong.
+
+*The smoking gun* — search **every** proxy, not just the one you are debugging, for the removal
+line, and put the timestamps next to each other:
+
+    infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
+      echo "== $ns"; kubectl --context k3s-prod -n $ns logs -l app.kubernetes.io/name=$ns \
+      -c oauth2-proxy --tail=-1 2>/dev/null | grep stored_session; done'
+
+**4. Do the cookie-secrets actually differ?** They should — but confirm the collision is between
+instances that genuinely cannot read each other. Compare **hashes**, never values:
+
+    infra/devcontainer/dev.sh run bash -c 'for ns in jackett pihole qbittorrent radarr sonarr crafty; do \
+      printf "%s " "$ns"; kubectl --context k3s-prod -n $ns get secret $ns-oauth2-proxy \
+      -o jsonpath="{.data.cookie-secret}" | sha256sum | cut -c1-12; done'
+
+Never print `.data` or `.stringData` itself. A password has been leaked into a transcript here
+before.
+
+### Do not rank apps by their failure ratio
+
+Counting `No valid authentication` against `AuthSuccess` per app looks like a health metric and is
+not one. **It ranks chattiness, not breakage.** An app whose UI polls or holds a WebSocket
+manufactures one failure line per retry, so a single dead cookie in one idle browser tab becomes
+scores of log lines, while an app nobody is using scores zero no matter how broken it is. Measured
+2026-09-30: Radarr showed **156** unauthenticated requests against **1** `AuthSuccess`, but the top
+paths were `/api/v3/command` (62), `/api/v3/queue/details` (46) and `/signalr/messages/negotiate`
+(43) — polling, not 156 user-visible failures.
+
+**Bound the window to the current pod, not to wall-clock.** A ratio measured over 24 h across a pod
+that restarted 13 h ago is mostly about the *previous* configuration, and that is exactly how a
+deployed fix gets written off as ineffective. The pod's own log has an exact boundary — its process
+start — so read it there:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n <ns> get pod \
+      -l app.kubernetes.io/name=<app> -o jsonpath="{.items[*].status.startTime}{\"\n\"}"'
+
+then count within that same pod's log. On 2026-09-30 this took Crafty from an apparent 81:1 over 24 h
+to **3 failures in its fixed pod's entire 13 h 52 m life**, two of them `/ws` reconnects.
+
+## Likely causes, most likely first
+
+1. **Cookie collision between two per-app proxies.** Several instances emit the default
+   `_oauth2_proxy` on `.home.keithjackson.dev` with different `cookie-secret`s. Whoever
+   authenticates last overwrites the rest; the losers log `cookie signature not valid, removing
+   session` and — because `removing session` clears the cookie on *their* `--cookie-domain`, which
+   is the whole zone — **delete a session belonging to an app in another namespace**. The victim
+   then logs `No valid authentication` with no cookie at all, seconds after a clean login, while
+   its own `expires` is still minutes away. **Read both logs side by side; one app's log alone
+   makes this look like random session loss.** Fix: a unique `--cookie-name` and a host-scoped
+   `--cookie-domain` on the per-app sidecar. The shared forward-auth proxy keeps the zone-wide
+   scope — cross-host auth is its job.
+2. **Session pinned to a 5-minute access token with no refresh.** `--cookie-refresh` unset means
+   the proxy never spends the refresh token it holds, so the session expires with the access token.
+   Set it to about half the measured token lifetime.
+3. **An XHR got a 302 instead of a 401.** The app's JavaScript follows the redirect invisibly, the
+   live Keycloak SSO session satisfies it without a prompt, and the XHR resolves with **HTML where
+   it expected JSON**. Nothing is logged as an error and the operator is never re-prompted — this
+   is what "it just doesn't save, and reloading doesn't help" looks like. Worse on a write: a
+   browser following a 302 re-issues a POST as a GET and **drops the body**, so the change is
+   discarded silently. oauth2-proxy's built-in AJAX guard only fires on an exact
+   `Accept: application/json`, which most app JavaScript does not send; make it deterministic with
+   `--api-route=<regex>` (**singular** — `--api-routes` does not exist and the proxy refuses to
+   start). Keep browser-navigable paths *out* of the regex so a plain page load still signs you in.
+4. **Group membership.** An `AuthSuccess` followed by `403` is the `--allowed-group` gate, not a
+   session problem. Check the `groups:[...]` field on the `AuthSuccess` line.
+5. **The app's own second login.** Crafty, Radarr and Sonarr each have one. `AuthSuccess` followed
+   by the app's own `/login` page is layer 3, not layer 2.
+
+## When to escalate or stop
+
+- **Stop before you `kubectl edit`/`patch`/`apply` anything.** Flux reconciles `main`; a
+  hand-applied flag is drift that the next reconcile reverts or, worse, adopts. Argument changes
+  go through a branch and a PR. Reading logs is always safe; changing the Deployment is not an
+  agent action.
+- **Do not hand-edit `apps/**/<app>.yaml` — those files are GENERATED.** The five per-app sidecars
+  (jackett, pihole, qbittorrent, radarr, sonarr) get their argument list from one shared function,
+  `_oauth2_sidecar_container()` in `catalog/render.py`; `catalog/<app>.yaml` declares only
+  `auth.mode` and `auth.groups`, and the `auth` schema is `additionalProperties: false`, so there
+  is **no per-app override hook**. A hand-edit to the rendered file is caught by the merge gate's
+  `render-golden` stage ("GOLDEN DRIFT ... does not match the canonical render") and would be
+  erased by the next render anyway. Change the renderer, then regenerate:
+
+      python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
+
+  **Budget for the blast radius**: because the function is shared, any change to it rewrites the
+  sidecar block in **all five** rendered apps, and the gate requires all five be regenerated in
+  the same commit. There is no way to change one app's sidecar arguments alone. Measured
+  2026-09-30. Coordinate before starting — this is a single change, not five.
+- **Crafty is the exception** and is not rendered from the catalog: its sidecar is hand-written in
+  `platform/crafty/deployment.yaml`, so it can be, and was, fixed independently.
+- **You cannot currently alert on any of this**, and that is worth knowing before you promise
+  monitoring. Measured 2026-09-30: no per-app sidecar sets `--metrics-address` and none declares a
+  `metrics` container port, so oauth2-proxy's metrics listener never starts — the only oauth2-proxy
+  series in Prometheus come from the shared proxy in namespace `oauth2-proxy`, which does set both.
+  Until the renderer emits them for the per-app sidecars, every diagnosis on this page is
+  log-reading only.
+- **Validate an argument change before you open the PR**, not after. The pinned image will tell
+  you what it understood, without touching the cluster:
+
+      podman run --rm quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 --help
+
+  Running it with your full proposed argument list prints a `Cookie settings:` line and an
+  `API route - Path:` line back at you, and rejects mutually exclusive flags outright
+  (`--session-cookie-minimal` with `--cookie-refresh`, for one). A flag typo here is a CrashLoop
+  on a service you have just told someone is fixed.
+- **Do not rotate a `cookie-secret` to "clear" a bad session.** It signs every live session out of
+  that app and does nothing about the collision, which is caused by the *names* matching, not the
+  secrets differing.
+- **Escalate to Keycloak** (`runbooks/keycloak-down.md`, `runbooks/keycloak-client-secrets.md`) if
+  there is no `AuthSuccess` in any proxy's log — that is an IdP or client-secret problem and no
+  cookie flag will help.
+- **Hand off rather than guess** if you see `cookie signature not valid` on a proxy that already
+  has a unique `--cookie-name`. That combination should not occur; it means something is writing
+  that name which is not in this inventory. Record what you measured.

runbooks/plex-quicksync.md

@@ -0,0 +1,283 @@
+# plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did
+
+**Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**,
+not an incident page. It is indexed here because the feature it completes is **inert until a human
+clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
+to prevent.
+**Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the
+rendered golden — the `dri` hostPath volume and the `/dev/dri` mount), `catalog/render.py` (the
+mechanism), `schemas/renderer-contract.md` `### hardware` (the contract).
+**Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile.
+
+## What this means here
+
+Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync
+moves that work off the CPU — which matters on a three-node cluster where the CPU is also running
+everything else. Two independent things have to be true before that happens, and **only one of them
+is in git**:
+
+| Half | Who does it | State |
+|---|---|---|
+| The render device is inside the container at `/dev/dri` | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders a `dri` hostPath volume (`/dev/dri`, `type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container. Flux applies it. | Done by merging that change. |
+| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
+
+**There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
+grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`,
+`/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin` — found **zero** `PLEX_PREFERENCE*` support.
+Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentifier`,
+`ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`,
+`TranscoderTempDirectory`), none of them the transcode toggle. No environment variable, no ConfigMap
+and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this;
+there isn't one, and the search has already been done.
+
+What the device mount does **not** do, so nobody reads more into a green gate than is there: it does
+not enable hardware transcoding, it does not prove the device can be *opened* from inside a
+non-privileged pod (see "Likely causes" #2), and it renders no `nodeSelector` — Plex can land on any
+of the three nodes and all three carry the device.
+
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation; the toolchain lives only in the pinned dev
+container. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
+picks a current-context **per invocation**, and that default has already changed once. Do not run two
+`dev.sh` invocations at once.
+
+**Never `cat` `Preferences.xml`.** It contains `PlexOnlineToken`, a plex.tv account credential. Every
+read of that file below greps a single named key — do not widen one.
+
+## First commands
+
+**1. Is the device mounted into the pod at all?** (Proves Flux applied the change, not that it works.)
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
+     -o jsonpath="{.items[0].spec.containers[0].volumeMounts[*].mountPath}"; echo'
+```
+
+`/dev/dri` in that list ⇒ the GitOps half landed. Absent ⇒ the running pod predates the merge; compare
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 \
+     -o jsonpath="{.status.lastAppliedRevision}"; echo'
+```
+
+against the merge commit before blaming anything else.
+
+**2. Is the device actually visible inside the container?**
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'
+```
+
+Expect `renderD128` (usually `card0` too), mode `crw-rw----`, group `render`. An **empty directory**
+here is the `DirectoryOrCreate` degradation: the kubelet created an empty `/dev/dri` because the node
+had none. That is the designed-safe outcome — the app tier still reconciles — but it means software
+transcoding, silently. Find which node the pod is on and ask why that node lost its iGPU:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
+```
+
+**3. Can Plex *open* it?** This is the question the manifest cannot answer — see "Likely causes" #2.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
+     bash -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
+```
+
+`RW-OPEN-DENIED` (or `Operation not permitted` / `Permission denied`) with the device plainly visible
+in step 2 is the **device-cgroup** failure, not a file-permission failure. The two look identical from
+the outside and are not the same problem.
+
+**4. Is the toggle on?** Named key only — never the whole file:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
+     grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
+     "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'
+```
+
+`ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on,
+and the remaining question is whether it *works* — go to "How to verify".
+
+## The one manual step
+
+Plex web UI → **Settings → Transcoder → "Use hardware acceleration when available"** → tick → Save.
+(It sits under *Show Advanced*.) A transcode already in flight is unaffected; the next one picks it up.
+
+The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by
+extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from
+recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise
+them and not invent them:
+
+| Key | What it is |
+|---|---|
+| `HardwareAcceleratedCodecs` | the toggle this page is about |
+| `HardwareAcceleratedEncoders` | hardware **encode**, as distinct from decode |
+| `HardwareDevicePath` | how Plex is pointed at a *specific* render node — relevant only if a host ever exposes more than one |
+| `TranscoderToneMapping`, `TranscoderToneMappingAgorithm` | HDR→SDR tone mapping. **That misspelling of "Algorithm" is upstream's, in the binary. It is not a typo in this page — do not "fix" it** here or anywhere else; the corrected spelling is not a key Plex reads. |
+| `TranscoderHEVCEncoding` | HEVC output |
+
+**`HardwareAcceleratedCodecsEnableHEVC` does not exist.** It is widely cited and it is not in the
+binary. Recorded explicitly so nobody spends an evening hunting for it or, worse, writes it into
+`Preferences.xml` and concludes hardware transcoding is broken.
+
+Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn
+tier, it is in no git commit, and the UI writes the key correctly. Use the UI.
+
+## How to verify — and why there is no alert
+
+Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a
+lower quality explicitly), then read Plex's own **Dashboard → Activity**:
+
+- **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.**
+- plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says.
+
+**That dashboard is the *only* proof available, and nothing else will ever tell you.**
+`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
+there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work
+to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently
+inert, and this repo has already been burned at scale by rules that evaluated against nothing while
+looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that
+gap is ever worth closing, the honest order is a real exporter first and a rule second.
+
+Corroborating, not a substitute — the server log on the config PVC:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
+     sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
+     \"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'
+```
+
+Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions.
+Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence
+of hardware. The dashboard remains the verdict.
+
+## Likely causes, most likely first
+
+When the dashboard still says plain `Transcode`:
+
+1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists.
+   First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
+   investigating anything technical.
+
+2. **containerd's device cgroup is refusing the `open()` — UNVERIFIED, and the largest remaining
+   technical risk in this feature.** Whether a non-privileged pod may `open()` a bind-mounted
+   character device on this k3s/containerd build **has not been established**; it can only be settled
+   by deploying and trying. The container inits as uid 0, which settles *file* permissions, but
+   **uid 0 does not bypass the cgroup device gate.** Symptom: step 2 shows `renderD128` present and
+   correctly owned, step 3 returns `RW-OPEN-DENIED`.
+   **The documented fallback is the Intel Device Plugin for Kubernetes** (`gpu.intel.com/i915` as a
+   resource request, no hostPath). Because the ServiceDefinition field expresses *intent* rather than
+   a device path, that migration is a **`catalog/render.py` change plus a re-render — with no
+   ServiceDefinition edit anywhere**; `catalog/plex.yaml` keeps saying exactly what it says today.
+   **`privileged: true` is not an available escape hatch:** Kyverno `restrict-host-namespaces` is in
+   **Enforce** mode and rejects it in app namespaces. Do not reach for it.
+
+3. **The pod landed on a node without the device.** Step 2 shows an empty `/dev/dri`. All three prod
+   nodes carried `renderD128` with `i915` bound when this was measured (2026-09-30), so this means a
+   node changed — a rebuild, a BIOS reset, or a driver regression. `DirectoryOrCreate` is why the app
+   tier still came up; the cost is that the loss is silent. **Fix the node, do not tighten the mount:**
+   `Directory` would stall `apps-wave-0` (`wait: true`, with `apps-wave-1` depending on it) for a
+   feature that is not load-bearing.
+
+4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
+   mechanism this feature leans on with no commit in this repo.
+
+5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so
+   `Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as
+   a hardware failure.
+
+6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which
+   Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
+   sessions show `(hw)` and others do not. Not a fault.
+
+## The floating-image caveat
+
+Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
+user to the device's group, the absence of `PLEX_PREFERENCE*`, the six `setPref` keys, the
+`HardwareAcceleratedCodecs` string — was read from digest **`sha256:e0ab27395614…`**
+(`plexmediaserver 1.43.4.10903`).
+
+The deployment runs **`plexinc/pms-docker:latest` with `imagePullPolicy: Always`**. A silent pull can
+therefore change the behaviour this feature depends on **with no commit in this repo**, while the
+manifest still looks identical. Symptom to watch for: hardware transcoding that worked and quietly
+stopped, with no git change to point at.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
+     -o jsonpath="{.items[0].status.containerStatuses[0].imageID}"; echo'
+```
+
+If that digest is not the one above, **re-verify the hook before trusting this page**:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- ls /etc/cont-init.d/'
+```
+
+`45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one
+described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups`
+needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that
+hook exists.
+
+## When to escalate or stop
+
+- **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be
+  **inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's
+  `cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its
+  group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces**
+  the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with
+  `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
+  discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
+- **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
+  reject it. If the cgroup gate is the blocker, the answer is the Intel device plugin (cause #2).
+- **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles
+  `main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating
+  `kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change
+  through a branch, a PR and a green gate.
+- **Do not edit `Preferences.xml` on the PVC**, never `cat` it, and never `sops -d` anything into a
+  terminal. It holds `PlexOnlineToken`.
+- **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and
+  watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
+  lose the DB.
+- **Stop and get the operator** if the device cannot be opened from the pod (cause #2 confirmed). The
+  next step is a platform change — the Intel device plugin — with its own PR, not more poking at this
+  workload.
+
+## Known gaps, deliberately not fixed here
+
+Recorded so they are findable, and so nobody assumes they were overlooked:
+
+- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
+  `TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
+  overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
+  transcode scratch performance changes silently whenever the pod is rescheduled. Both candidate fixes
+  need a **schema** change and are therefore out of scope here: pin Plex to prod-2 (needs a
+  `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory` emptyDir and
+  transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not have). The RAM option
+  is the better one, because it removes the node asymmetry outright instead of working around it.
+- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
+  quality only — decode, and the feature on this page, are unaffected. Changing it needs a kernel
+  parameter and a **serialized rolling reboot**, which is an operator action (CLAUDE.md §5), and this
+  cluster has a documented history of that going badly (a prod node hung on a warm reboot on
+  2026-09-03). Not worth coupling to this change.
+
+---
+Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
+The node measurements (all three prod nodes) and the in-container image measurements cited here were
+taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
+(`plexmediaserver 1.43.4.10903`). Hardware transcoding had **not** been observed working at the time
+of writing, and the Plex-side toggle was still unset.