feat(gpu): Intel GPU device plugin v0.37.0, in its own Flux Kustomization

Makes the three prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the Plex hardware-transcoding blocker, and it only makes the resource EXIST — no consumer is switched over here. WHY. The `/dev/dri` hostPath merged on main @ cdf3e14 makes the device node visible and not openable. Measured in the running pod on k3s-prod-2: uid 0, O_RDWR -> EPERM "Operation not permitted" uid 1000, O_RDWR -> EPERM "Operation not permitted" EPERM from the device cgroup, not EACCES from file permissions — root on the HOST opens the same device rc=0, the node is unified cgroup2fs, and the container's runc config.json carries one deny-all rule with `linux.devices: None`. A hostPath is a filesystem operation: it makes the device node appear and adds no cgroup allow rule, because Kubernetes cannot infer one from a volume. A device plugin's Allocate response is what makes kubelet populate `linux.devices` AND the matching cgroup allow rule — verified in the v0.37.0 source, which emits `DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node. It is the only non-privileged way to get there; `privileged: true` is blocked by Kyverno restrict-host-namespaces under Enforce in app namespaces, and `plex` is labelled app-namespace=true. PLACEMENT: its own Kustomization (dependsOn cilium, wait:true), not a line in platform/kustomization.yaml. The shared tier was cheaper and needed no human reviewer on approval-policy.json's terms, but it is wait:true over eleven-plus components and the repo already argued this when Cilium was split out: its Ready condition cannot tell "the plugin is wedged" from "longhorn is slow", the only emergency valve is freezing the whole tier, and the tier's 10m/5m interval/timeout are called out there as wrong for a 3-node DaemonSet rollout — which is exactly what this is. Worse here than average: platform gates platform-config gates apps-wave-0 (wait:true) gates apps-wave-1, so a plugin that never went healthy inside it would stall every app — a component added to fix a silent no-op would have gained the power to cause a real outage. Touching clusters/** costs nothing extra: required_approvals:1 with block_admin_merge_override:true applies to every PR here regardless of path. wait:true is a health SIGNAL, not a gate — nothing in the rendered graph dependsOn this, so it gates nothing, while making `flux get kustomization intel-gpu-plugin` answer "did it come up on all three nodes". NON-CHOICES, each argued in the README: standalone DaemonSet over the operator (no CRD, controller-manager, webhook or second namespace to keep healthy); `--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler believes — no GPU-side partitioning backs it, so it converts a loud Unschedulable into a quiet "transcodes are slow"); `--health-management` left OFF, the one flag that would open a device and reintroduce this EPERM; no NFD overlay (all three nodes identical); no supplementalGroups (the pms-docker cont-init hook already adds `plex` to each device's group, measured live as Groups: 44 100 992 1000). The DaemonSet is vendored as `kubectl kustomize` output of upstream v0.37.0's gpu_plugin/base, rather than referenced as a remote kustomize base, for the reason external-snapshotter records: a remote base is git-fetched on every reconcile and those fetches have flaked a whole tier. It is generated rather than byte-copied because upstream mixes YAML sequence-indentation styles within the one document and ci/yamllint.yaml sets `indent-sequences: consistent`, so a verbatim copy is a hard lint ERROR — measured, not guessed: the first gate run of this branch returned rc 1 BLOCKED on exactly that. The transform is proven formatting-only (`yq -P 'sort_keys(..)'` of both diffs clean). Image pinned to docker.io/intel/intel-gpu-plugin:0.37.0 (digest verified present: sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1). ORDERING — DO NOT REORDER. catalog/render.py, catalog/plex.yaml and apps/wave-0/plex.yaml are untouched. The renderer change that emits `gpu.intel.com/i915: 1` instead of the hostPath must NOT land until this plugin is merged AND confirmed advertising the resource on all three nodes: a request for a resource no node advertises makes the Plex pod Unschedulable, and apps-wave-0 is wait:true with apps-wave-1 depending on it. VERIFIED: manifests render offline (component, clusters/base, both overlays); no node currently advertises gpu.intel.com/* (queried — "NONE"); the plugin only lists/stats devices and never opens one (no os.Open/OpenFile/ioctl/syscall anywhere in its 908-line source); all four app-scoped Kyverno policies are gated on app-namespace=true and the new namespace is labelled "false". NOT VERIFIED: anything runtime. The plugin has never executed, the resource has never been advertised, and the open() test has not been re-run. Gate returned rc 3 (UNDETERMINED) with exactly the three expected skips — not green. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Merge pull request 'feat(plex): hardware transcoding via hardware.gpu: intel-quicksync, plus the runbook for the manual last mile' (#32 ) from feat/plex-quicksync into main
2026-10-01 22:25:33 -05:00 · 2026-10-02 02:17:48 +00:00 · 2026-10-02 02:17:23 +00:00 · 2026-10-01 21:06:34 -05:00 · 2026-10-01 21:05:17 -05:00
13 changed files with 1315 additions and 0 deletions
@@ -143,6 +143,8 @@ spec:
        - name: media
          mountPath: /media
          readOnly: true
        - name: dri
          mountPath: /dev/dri
      volumes:
      - name: config
        persistentVolumeClaim:
@@ -150,6 +152,10 @@ spec:
      - name: media
        persistentVolumeClaim:
          claimName: plex-media
      - name: dri
        hostPath:
          path: /dev/dri
          type: DirectoryOrCreate
 ---
 # Service
 apiVersion: v1
@@ -80,6 +80,58 @@ env:
  # remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
  # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
  # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
 # HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
 # INTENT ONLY. This says "this workload needs an Intel Quick Sync render device"; it does NOT name a
 # device path. The renderer currently satisfies it with exactly two things — a `/dev/dri` hostPath
 # volume (`type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container —
 # and nothing else: no nodeSelector, no privileged, no resources (schemas/renderer-contract.md
 # `### hardware`). If that mechanism is later replaced by the Intel GPU device plugin
 # (`gpu.intel.com/i915`), it is a `catalog/render.py` change and THIS BLOCK DOES NOT MOVE. Do not
 # "simplify" it into a raw `/dev/...` path; that is the whole point of the enum.
 #
 # THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
 # anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
 # GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
 # `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Plex happens to run on k3s-prod-2 today, but
 # that is scheduler history rather than a requirement — this field deliberately renders no
 # nodeSelector, because placement is a separate concern with its own justification.
 #
 # NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
 # The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
 # the host `render` GID in as a supplemental group. It cannot work here. MEASURED IN THE RUNNING
 # CONTAINER, 2026-09-30: the image's own
 # `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device under
 # `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
 # `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
 # GID renumbering self-healing (no GID is pinned anywhere in this repo). The service then execs via
 # `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from `/etc/group` rather
 # than appending to it: `Plex Media Server` runs with `Groups: 100 1000`, and root's gid `0` — which
 # PID 1 (`s6-svscan`) does hold — is GONE. Anything Kubernetes supplies therefore reaches s6-svscan
 # and is discarded before Plex ever starts. Mounting the device is sufficient; the group is the
 # image's job and it does it correctly. `render.py --selftest` asserts the ABSENCE of a pod
 # securityContext for this field precisely so nobody adds an inert one back later.
 #
 # `DirectoryOrCreate` IS THE SAFE FAILURE MODE, not laziness. A node that comes back without the
 # device (rebuild, BIOS reset, driver regression, hardware swap) gets an empty directory and Plex
 # degrades to SOFTWARE transcoding. `Directory` would fail the mount instead — and because
 # `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that would stall the WHOLE app
 # tier for a feature that is not load-bearing. The price, stated plainly: a missing device is
 # SILENT. This SD renders no `metricsEndpoint`, so there is no ServiceMonitor, no transcode-mode
 # metric, and no alert that could ever fire. The only proof is an operator reading the Plex
 # dashboard.
 #
 # THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Mounting the device does
 # not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
 # is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
 # set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
 # `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin`) found ZERO `PLEX_PREFERENCE*` support,
 # and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
 # `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
 # unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
 # `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took and the
 # device-cgroup failure mode to expect first.
 hardware:
  gpu: intel-quicksync
 exposure: public              # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
@@ -0,0 +1,104 @@
 ---
 # Flux Kustomization: Intel GPU device plugin — reconciles ./platform/intel-gpu-plugin.
 # Makes the three prod nodes advertise `gpu.intel.com/i915`, which is the resource a later PR will
 # have `hardware.gpu: intel-quicksync` request in place of today's non-working /dev/dri hostPath.
 #
 # WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
 #
 # The cheap option was a line in the shared `platform` tier plus a new platform/ directory, and
 # neither path is sensitive in infra/gitea/approval-policy.json, so it would have needed no human
 # reviewer on the policy's terms. It is still the wrong place, for the reason the repo already
 # wrote down in platform/kustomization.yaml when Cilium was split out of that tier:
 #
 #   * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot tell
 #     you "the GPU plugin is wedged" from "longhorn is slow".
 #   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
 #     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device
 #     plugin.
 #   * The tier's 10m interval / 5m timeout are explicitly called out there as WRONG for a 3-node
 #     DaemonSet rollout. This is a 3-node DaemonSet rollout.
 #
 # And the consequence is worse here than for an average component, because of what sits downstream:
 # `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates apps-wave-1.
 # A plugin DaemonSet that never went healthy inside that tier would stall every app on "dependency
 # not ready" — i.e. a component added to fix a SILENT NO-OP (software transcoding) would have
 # acquired the ability to cause a REAL OUTAGE. That trade is the whole argument.
 #
 # The usual objection to this placement is the approval cost, and it does not apply: Gitea enforces
 # `required_approvals: 1` with `block_admin_merge_override: true` on EVERY PR here regardless of
 # path, and authors cannot self-approve. So this change needed the operator's approval either way;
 # touching clusters/** costs nothing extra and buys a blast radius of exactly one component.
 #
 # dependsOn: [cilium], WHICH IS THE MINIMUM AND ALSO THE EARLIEST
 #
 # Not platform-config (what headlamp/crafty/homepage/oauth2-proxy use) — those are leaf UIs that
 # genuinely need Keycloak + wildcard TLS + split-horizon DNS + Gateway. This plugin needs none of
 # it. It never uses the network and never talks to the API server; it talks to the local kubelet
 # over a unix socket. Its only real prerequisite is a working CNI, so its pod can get a sandbox at
 # all, and `cilium` already depends on `flux-system` (the root Kustomization that applies the
 # cluster-vars ConfigMap this one substitutes from), so that edge is transitively satisfied too.
 #
 # Depending on `cilium` rather than `platform` also gets the resource advertised EARLIEST — this
 # reconciles in parallel with the `platform` tier instead of behind all eleven of its components.
 # That matters on a cold start once the consumer PR lands: a Plex pod requesting a resource no node
 # advertises is Unschedulable, and apps-wave-0 is wait:true. Reconciling early shrinks that window
 # instead of widening it. It is not a GUARANTEE — see the ordering note below, which is a real
 # constraint on the next PR and not a formality.
 #
 # Kyverno (in `platform`) is deliberately NOT a dependency: every policy that could match this
 # workload is scoped to namespaces labelled app-namespace=true, and this namespace is labelled
 # "false" (platform/intel-gpu-plugin/namespace.yaml). So there is no admission race to lose.
 #
 # wait: true — A HEALTH SIGNAL, NOT A GATE. This is the one place this component departs from the
 # headlamp/crafty leaf template, and the departure is deliberate. Those run wait:false because a
 # leaf UI can be legitimately not-Ready for a long time during a cold start while it waits on
 # Keycloak/DNS/TLS, so its health is not a useful signal. This DaemonSet has no such dependency: it
 # reads sysfs and opens a socket, so on healthy nodes it should be Ready in seconds, and
 # not-Ready genuinely means BROKEN. wait:true turns `flux get kustomization intel-gpu-plugin` into
 # a one-line answer to "did the plugin come up on all three nodes", which is exactly the question
 # the operator must answer before the consumer PR may merge.
 #
 # NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
 # plugin is this Kustomization's own Ready condition and nothing else. Anyone later adding
 # `dependsOn: [intel-gpu-plugin]` is choosing to make plugin health gate their thing; for the app
 # waves that would hand a device plugin the power to stall every app, which is the thing this file
 # exists to prevent. Think before adding that edge.
 #
 # ORDERING — DO NOT REORDER THE STACK. The renderer change (catalog/render.py emitting
 # `gpu.intel.com/i915: 1` for `hardware.gpu: intel-quicksync` instead of the hostPath) must NOT
 # land until this plugin is merged AND confirmed advertising the resource on all three nodes. If
 # the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is
 # wait:true and apps-wave-1 dependsOn it, so that stalls the whole app tier — turning today's silent
 # no-op into a real outage. This PR's only job is to make the resource exist.
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
  name: intel-gpu-plugin
  namespace: flux-system
 spec:
  # Static vendored manifests with nothing to re-render; kustomize-controller is source-triggered,
  # so this is only the drift-correction floor.
  interval: 10m
  retryInterval: 2m
  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
  # exceed 5m on a cold start (image pull on three nodes), and the Cilium split-out comment calls
  # out that exact mismatch as one of its reasons for leaving the shared tier.
  timeout: 10m
  prune: true
  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
  wait: true
  dependsOn:
    - name: cilium   # the CNI — the only real prerequisite; transitively gives us flux-system too
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./platform/intel-gpu-plugin
  # Retained for parity with every other tier / future overlay knobs. This component references no
  # ${VARS} today, and its manifests contain no `$` literals for envsubst to eat (unlike
  # clusters/base/runbooks.yaml, which omits substitution for exactly that reason).
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
  # No SOPS decryption patch: this component carries no Secret, so clusters/dev adds nothing for it
  # (same as homepage/runbooks, unlike headlamp/crafty/oauth2-proxy/backup).
@@ -12,6 +12,15 @@ resources:
  # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
  # freeze the other eleven platform components. See clusters/base/cilium.yaml.
  - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
  # Intel GPU device plugin — its OWN Kustomization (dependsOn cilium, wait:true), NOT a line in
  # platform/kustomization.yaml. Same reasoning as the Cilium split-out, applied to a leaf: inside
  # that wait:true tier a wedged 3-node DaemonSet would be indistinguishable from eleven unrelated
  # components AND would stall platform-config + every apps wave on "dependency not ready" — so a
  # component added to fix a silent no-op (Plex software transcoding) could cause a real outage.
  # wait:true here is a HEALTH SIGNAL, not a gate: nothing in the graph dependsOn it.
  # It advertises `gpu.intel.com/i915`; the renderer change that REQUESTS that resource must land
  # strictly later. See clusters/base/intel-gpu-plugin.yaml.
  - intel-gpu-plugin.yaml # Flux Kustomization -> ./platform/intel-gpu-plugin (dependsOn cilium)
  - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
  - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
  # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
@@ -0,0 +1,174 @@
 ---
 date: 2026-09-30
 branch: feat/plex-quicksync
 title: Plex opts into `hardware.gpu: intel-quicksync` — the device mount is GitOps, the Plex-side toggle is a documented manual step, and hardware transcoding has NOT been observed working
 ---
 ## What changed
 Three things, and the third is the point of the branch.
 1. **`catalog/plex.yaml`** — added the four lines the whole feature reduces to:
   ```yaml
   hardware:
     gpu: intel-quicksync
   ```
   placed between `env` and `exposure` to match the schema's own property order, with ~48 lines of
   comment in the file's established style. The comment records: the field is **intent, not
   mechanism** (a later move to the Intel device plugin is a `render.py` change and this block does
   not move); that `/dev/dri/renderD128` is present on **all three** prod nodes with `i915` bound;
   **why a `supplementalGroups` entry would be inert** (the image's cont-init hook does the group
   work inside the container and `s6-setuidgid` replaces the process group list from `/etc/group`);
   why `DirectoryOrCreate` is the load-bearing safe failure mode (`apps-wave-0` is `wait: true` with
   `apps-wave-1` depending on it, so a failed mount would stall the whole app tier for a feature
   that is not load-bearing); and that the Plex-side toggle is a **separate manual operator step**,
   pointing at the new runbook.
 2. **`apps/wave-0/plex.yaml`** — re-rendered. Six added lines, nothing else, in the whole `apps/`
   tree:
   ```
           - name: dri
             mountPath: /dev/dri
         - name: dri
           hostPath:
             path: /dev/dri
             type: DirectoryOrCreate
   ```
 3. **`runbooks/plex-quicksync.md`** (new), registered in `runbooks/kustomization.yaml`'s
   `configMapGenerator` as `- plex-quicksync=plex-quicksync.md` and indexed in `runbooks/README.md`
   under "Operator procedures". **The file alone would have been a 404** — the pod serves a
   ConfigMap, not the git tree, and `conformance/tests/l0-static/alert-runbooks-resolvable` exists
   precisely because "the file is in git" and "the URL resolves" are different claims. The page
   covers: the GitOps/manual boundary as a table; the toggle (Settings → Transcoder → "Use hardware
   acceleration when available", preference key `HardwareAcceleratedCodecs`); the real neighbouring
   keys and the widely-cited one that **does not exist**; that the only proof is `Transcode (hw)` on
   the Plex dashboard **and why no alert is possible**; the device-cgroup failure to expect first
   with a concrete open() diagnostic and the Intel-device-plugin fallback; and the floating-`:latest`
   caveat with a digest check.
 **Stacked on PR #27 (`feat/sd-hardware-capability`), which is open and unmerged.** This branch was
 cut from it rather than from `main`, so its one commit (`d9baa2a`, the schema + renderer + contract)
 is in this PR's range. That is deliberate: merging #27 is not an agent's action (CLAUDE.md §1), and
 duplicating its changes here would have been worse than showing them. **This must merge after #27.**
 Not done, deliberately: no `--emit-waves` (it writes into `clusters/base/`, a sensitive path that
 would force a human reviewer, and it is a no-op unless the *set* of files under `apps/` changes — it
 does not here). No `podSecurityContext`. No `nodeSelector`. No alert rule. No `kubectl apply`.
 ### Two follow-ups left unclaimed, with the reasoning
 - **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
  `TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
  overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
  transcode scratch performance changes silently on reschedule, with nothing reporting it. Both
  candidate fixes need a **schema** change and so neither belongs in this PR: pin Plex to prod-2
  (needs a `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory`
  emptyDir and transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not
  have). **The RAM option is the better one** — it removes the node asymmetry outright instead of
  working around it — which is exactly why it deserves its own PR and its own review rather than
  being smuggled in behind a four-line catalog change. Documented in the runbook's "Known gaps".
 - **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
  quality only; decode and this feature are unaffected. Changing it needs a kernel parameter plus a
  **serialized rolling reboot** — an operator action under CLAUDE.md §5, on a cluster with a
  documented history of that going badly (a prod node hung on a warm reboot on 2026-09-03). Kept
  well clear of this PR.
 ### Rebase note (2026-10-01, Ruth Okonkwo) — resolution only, no redesign
 Rebased onto `main` @ `08deb20` (was cut from `0fbf544`), restacked on the rebased
 `feat/sd-hardware-capability` (PR #27) so the stack stays coherent — this branch still carries
 exactly PR #27's commit plus its own, with no duplicate of anything already in `main`. The branch
 ref was force-updated with `--force-with-lease`, so PR #32 is the same PR with a new head.
 **The `catalog/render.py` conflict disappeared** once PR #27 was rebased; it was never this
 branch's own conflict. **One conflict remained, in `runbooks/README.md`, purely textual.** PR #30
 (`fix/jackett-oauth2-proxy-session`, merged as `08deb20`) appended an `oauth2-proxy-session` index
 entry under *Operator procedures* at the same point this branch appended its `plex-quicksync` entry.
 **Both entries were kept, in full.** Ordering follows this file's existing convention, which is
 grouped/narrative rather than alphabetical (the *Nodes* section runs down/io/memory/disk/cpu):
 `plex-quicksync` sits next to `plex-claim` as the other Plex page, leaving `oauth2-proxy-session`
 adjacent to `crafty-admin-credential`, which cross-references the same sidecar gate. The change to
 `runbooks/README.md` is an 8-line pure insertion, so PR #30's entry is untouched.
 `runbooks/kustomization.yaml` auto-merged and **both** `configMapGenerator` entries are present.
 Re-verified after the resolution, on the rebased stack: `--selftest` PASS (rc 0); a full re-render
 reproduces every committed golden byte-identically; and the **only** `apps/` diff against the new
 `main` is `apps/wave-0/plex.yaml`, confined to the expected six lines (the `dri` mount and its
 `DirectoryOrCreate` hostPath volume). No other service's golden moved — PR #28 added ~176 lines of
 oauth2-proxy sidecar rendering, and this confirms by measurement rather than assumption that Plex
 (`auth: native`, no sidecar) is unaffected by it.
 ## What I verified
 - **The render is exactly the intended change, and nothing else moved.** After
  `python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` (exit 0,
  silent): `git diff --stat apps/` = `apps/wave-0/plex.yaml | 6 ++++++`, and
  `git status --porcelain apps/` = a single ` M apps/wave-0/plex.yaml`. Read line by line, the diff
  is the `dri` volumeMount and the `dri` hostPath volume, and nothing more. **Specifically checked:**
  the same invocation also rewrites `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, which
  are git-tracked but excluded from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in
  `ci/render-golden.sh` — a spurious diff there would have landed **ungated**. Neither file shows a
  diff; whole-repo `git status --porcelain` lists only `apps/wave-0/plex.yaml` and `catalog/plex.yaml`
  plus my new files.
 - **The rendered output matches what the contract promises** — `schemas/renderer-contract.md`
  `### hardware` says exactly two things (a `/dev/dri` hostPath with `type: DirectoryOrCreate` and a
  read-write mount on the app container) and exactly two things appeared. No `securityContext`, no
  `nodeSelector`, no resources in the diff.
 - **The runbook is projected into the server, not merely committed.** `runbooks/kustomization.yaml`
  carries `- plex-quicksync=plex-quicksync.md`, whose shape matches the generator-entry regex the
  L0 check parses; the extensionless key is what makes `/runbooks/plex-quicksync` a plain static
  lookup. Also indexed in `runbooks/README.md`.
 - **The runbook clears the L0 quality floor** that `alert-runbooks-resolvable` applies to
  alert-linked pages, even though nothing links to this one (it is advisory for an unreferenced
  page): all four required sections (`## What this means here`, `## First commands`,
  `## Likely causes, most likely first`, `## When to escalate or stop`), well over the 1500-byte
  floor, `dev.sh` used throughout, and `--context k3s-prod` on every command.
 - **The gate.** See the run recorded in the PR description for the per-stage table and the real rc.
 - **No secret was printed.** Every `Preferences.xml` read in the runbook greps one named key, and the
  page states explicitly why: the file holds `PlexOnlineToken`. Nothing was `sops -d`'d.
 ## What I did NOT verify
 **Hardware transcoding has NOT been observed working.** Nothing in this branch is evidence that it
 does. A green gate here proves the manifest renders and the golden matches — it proves nothing about
 runtime behaviour, and `vars.CLUSTER_AVAILABLE` is unset so L2/L3 conformance did not run at all.
 - **The device-cgroup question is OPEN, and it is the largest remaining technical risk.** Whether
  containerd on this k3s build permits a **non-privileged** pod to `open()` a bind-mounted character
  device is **UNVERIFIED**. It can only be settled by deploying and trying. The container inits as
  uid 0, which settles *file* permissions, but uid 0 does **not** bypass the cgroup device gate — and
  the two failures look identical from the outside. If it is refused, the fallback is the Intel
  Device Plugin (`gpu.intel.com/i915`), which is a `catalog/render.py` change plus a re-render with
  **no ServiceDefinition edit**; `privileged: true` is not available (Kyverno
  `restrict-host-namespaces` is Enforce). The runbook has the diagnostic sequence.
 - **The Plex-side toggle was still unset at the time of writing.** `HardwareAcceleratedCodecs` was
  **absent** from `Preferences.xml` — the toggle has never been enabled on this server. Until an
  operator ticks it in the UI, this branch changes the pod spec and **nothing about transcoding**.
  That is the whole reason the runbook exists.
 - **No metric, so no alert, so no automated detection — permanently.** `catalog/plex.yaml` declares
  no `metricsEndpoint`, so Plex renders no ServiceMonitor; there is no transcode-mode signal for a
  rule to evaluate. A silent degradation to software transcoding (a node losing its iGPU, the
  `DirectoryOrCreate` path) will **not** be reported by anything. I deliberately did not add an
  alert: with no metric behind it, it could never fire, and this repo has been burned at scale by
  exactly that. Closing the gap honestly means an exporter first.
 - **I did not re-derive this session's node or in-container measurements.** The three-node
  `renderD128`/`i915`/GID-992 facts, the cont-init hook behaviour, the `s6-setuidgid` group
  replacement, the `PLEX_PREFERENCE*` absence, the six `setPref` keys and the
  `HardwareAcceleratedCodecs` string were established earlier in this session and are cited, not
  re-measured, by me. Per CLAUDE.md §2b, treat this file as a record of belief and re-measure against
  the cluster before relying on any of it.
 - **The image is a floating tag.** Everything known about the hook came from digest
  `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), while the Deployment runs
  `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. A silent pull can invalidate the
  "no `supplementalGroups` needed" conclusion **with no commit in this repo**. Not guarded against —
  the runbook tells the reader to re-check the hook after an image bump, which is documentation, not
  enforcement.
 - **I did not verify the live Gitea gate state** (branch protection, required contexts, runner
  online) for this PR. Per §2b that must be queried, not read out of `CLAUDE.md`'s table; I did not
  query it.
 - **`/runbooks/plex-quicksync` was not fetched.** The L0 check asserts every static precondition for
  an HTTP 200; it does not make the request, and I did not either.
@@ -0,0 +1,198 @@
 ---
 date: 2026-10-01
 branch: feat/intel-gpu-device-plugin
 title: Intel GPU device plugin, in its own Kustomization — the resource exists; nothing has run yet
 ---
 ## What changed
 Deployed the **Intel Device Plugin for Kubernetes (GPU)**, standalone DaemonSet, pinned at
 **v0.37.0**, so the prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the
 Plex hardware-transcoding blocker: the `/dev/dri` hostPath merged on `main` @ `cdf3e14` makes the
 device node *visible* and not *openable* (`EPERM` from the device cgroup at both uid 0 and uid
 1000; the container's runc `config.json` carries one deny-all rule and `linux.devices: None`). A
 hostPath is a filesystem operation and adds no cgroup allow rule. A device plugin's `Allocate`
 response is what makes kubelet populate `linux.devices` *and* the matching cgroup allow rule, and
 it is the only non-privileged way to get there — `privileged: true` is blocked by Kyverno
 `restrict-host-namespaces` under Enforce in app namespaces, and `plex` is labelled
 `app-namespace=true`.
 New files:
 - `platform/intel-gpu-plugin/intel-gpu-plugin.yaml` — the DaemonSet, **vendored** as
  `kubectl kustomize` output of upstream `deployments/gpu_plugin/base` at `v0.37.0`, under a
  provenance header carrying the exact re-vendor command and both sha256s (generated body
  `80ebd070…`, raw upstream source `c8773918…`). Generated rather than byte-copied because
  upstream mixes YAML sequence-indentation styles within the one document and `ci/yamllint.yaml`
  sets `indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate.
  That is not a guess: the first gate run of this branch returned **rc 1 BLOCKED** on exactly that
  error, and regenerating through kustomize is the fix (and is also the re-vendor idiom
  `platform/external-snapshotter` documents). Vendored at all, rather than referenced as a remote
  kustomize base, for the other reason `external-snapshotter` records: a remote base is
  `git fetch`ed on every reconcile and those fetches have flaked a whole tier.
 - `platform/intel-gpu-plugin/namespace.yaml` — a PLATFORM namespace, explicitly
  `homelab.keithjackson.dev/app-namespace: "false"` (following `vpn-egress`/`kyverno`).
 - `platform/intel-gpu-plugin/kustomization.yaml` — namespace scoping plus the image pin.
 - `platform/intel-gpu-plugin/README.md` — the measurement, the placement argument, the
  non-choices, and the post-merge verification sequence.
 - `clusters/base/intel-gpu-plugin.yaml` — **its own Flux Kustomization**: `dependsOn: [cilium]`,
  `wait: true`, `interval: 10m`, `timeout: 10m`, `prune: true`.
 - One wired line (plus its rationale comment) in `clusters/base/kustomization.yaml`.
 **Placement: its own Kustomization, not a line in the shared `platform` tier.** The shared tier was
 cheaper and needed no human reviewer on `approval-policy.json`'s terms, but `platform` is
 `wait: true` over eleven-plus components and the repo already argued this case when Cilium was
 split out: its Ready condition cannot distinguish "the plugin is wedged" from "longhorn is slow",
 the only emergency valve is freezing the entire tier, and the tier's 10m/5m interval/timeout are
 called out there as *wrong for a 3-node DaemonSet rollout* — which is exactly what this is. The
 consequence is worse than average here, because `platform` gates `platform-config` gates
 `apps-wave-0` (`wait: true`) gates `apps-wave-1`: a plugin that never went healthy inside that tier
 would stall every app, so a component added to fix a **silent no-op** would have gained the ability
 to cause a **real outage**. The approval cost of touching `clusters/**` is nil, because
 `required_approvals: 1` with `block_admin_merge_override: true` applies to every PR here regardless
 of path and authors cannot self-approve.
 `dependsOn: [cilium]` is the minimum *and* the earliest: the plugin never uses the network and
 never talks to the API server (it talks to the local kubelet over a unix socket), so a working CNI
 is its only real prerequisite, and `cilium` already depends on `flux-system`, which applies the
 `cluster-vars` ConfigMap this Kustomization substitutes from. Depending on `cilium` rather than
 `platform` also means it reconciles *in parallel with* the platform tier rather than behind all
 eleven of its components, which shrinks the cold-start window in which a node has not yet
 advertised the resource.
 `wait: true` is a deliberate departure from the `headlamp`/`crafty` leaf template. Those use
 `wait: false` because a leaf UI can be legitimately not-Ready for a long time during a cold start,
 making its health a useless signal. This DaemonSet reads sysfs and opens a socket, so not-Ready
 genuinely means broken — and since **nothing in the graph `dependsOn` it** (verified against the
 rendered graph), `wait: true` gates nothing while turning
 `flux get kustomization intel-gpu-plugin` into a one-line answer to the question that must be
 answered before the consumer PR may merge.
 Deliberate non-choices, each argued in the README: standalone DaemonSet rather than the operator
 (no CRD, no controller-manager, no cert-manager webhook, no second namespace to keep healthy);
 `--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler
 believes — there is no GPU-side partitioning behind it, so it converts a loud "Unschedulable" into
 a quiet "transcodes are slow"); `--health-management` left OFF, because it is the one flag that
 *would* open a device and would reintroduce the exact `EPERM` this fixes; no NFD overlay (all three
 nodes are identical); no `supplementalGroups` anywhere, because the `pms-docker` cont-init hook
 already adds the `plex` user to each device's group and that works identically on a
 plugin-injected device.
 **Scope held deliberately:** `catalog/render.py`, `catalog/plex.yaml` and `apps/wave-0/plex.yaml`
 are untouched. Switching Plex from the hostPath to a `gpu.intel.com/i915: 1` request is a strictly
 later PR, and **it must not land until this plugin is merged AND confirmed advertising the resource
 on all three nodes** — a request for a resource no node advertises makes the Plex pod
 Unschedulable, and `apps-wave-0` is `wait: true` with `apps-wave-1` depending on it, so that would
 stall the whole app tier and turn today's silent no-op into a real outage.
 ## What I verified
 Mechanism, read from the v0.37.0 source rather than assumed:
 - **The plugin only lists devices; it never opens one.** The scan path uses `os.ReadDir`,
  `os.ReadFile`, `os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. `grep` for
  `os.Open|OpenFile|ioctl|syscall|unix\.` across `cmd/gpu_plugin/gpu_plugin.go` (908 lines) returns
  **nothing**. Listing and `stat()` are what demonstrably work today, so the plugin needs no cgroup
  grant and no extra privilege — it does not need more than the upstream manifest gives it.
 - **The fix mechanism is real:** `createDeviceSpecsFromDrmFiles` emits
  `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, which is the
  thing kubelet turns into `linux.devices` plus a cgroup allow rule.
 - **The resource name is `gpu.intel.com/i915` by derivation, not by hope:** `scan()` calls
  `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the kernel driver name, and these
  nodes have `i915` bound.
 - `--shared-dev-num` default is `1` (`flag.IntVar(..., 1, ...)`); `--health-management` default is
  `false` and `setupLevelZeroService` returns early unless it is set.
 - Missing `/dev/dri/by-path` is non-fatal (`newDevicePlugin` warns, sets `bypathFound = false`).
 Image pin, verified against the registry:
 - `docker.io/intel/intel-gpu-plugin:0.37.0` exists; manifest digest
  `sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1` (single amd64
  manifest, not an index), fetched from `registry-1.docker.io` on 2026-10-01. `v0.37.0` is the
  latest upstream release (2026-09-16) and is not a prerelease.
 - The Docker Hub tag list carries `0.35.0 / 0.36.0 / 0.37.0` — **image tags have no `v` prefix**
  while GitHub release tags do.
 Policy scope, read from the policies rather than assumed:
 - All four app-scoped Kyverno policies (`restrict-host-namespaces`, `require-egress-class-label`,
  `require-auth-mode-label`, `require-default-deny-baseline`) are gated on
  `namespaceSelector`/`selector` `matchLabels: homelab.keithjackson.dev/app-namespace: "true"`.
  The new namespace is labelled `"false"`, so none of them match it.
 - Measured on prod: `plex` **is** `app-namespace=true`, confirming `privileged: true` is genuinely
  unavailable there; `kyverno` and `vpn-egress` are the two namespaces carrying the label as
  `"false"`; the other platform namespaces leave it unset. No `intel*` namespace exists, so no
  collision.
 Cluster before-state (read-only, `kubectl` in the pinned toolbox with the prod kubeconfig
 bind-mounted read-only — never copied into the worktree):
 - Three nodes `k3s-prod-{1,2,3}`, each `v1.36.4+k3s1`, Debian 13 trixie, `containerd://2.3.4-k3s1.36`.
 - Every extended resource currently advertised is `devices.kubevirt.io/{kvm,sev,tun,vhost-net}` on
  all three nodes. **No node advertises `gpu.intel.com/*`** — queried explicitly, returned
  "NONE". That is the before-state this change is meant to move.
 - Those `devices.kubevirt.io/*` resources are themselves advertised by a device plugin, which is
  measured evidence that the device-plugin API and the default kubelet socket path
  (`/var/lib/kubelet/device-plugins`) work on these nodes.
 Offline rendering (the main proof available to a PR):
 - `kubectl kustomize platform/intel-gpu-plugin` renders two objects: `Namespace/intel-gpu-plugin`
  carrying `app-namespace=false`, and `DaemonSet/intel-gpu-plugin` in namespace
  `intel-gpu-plugin`, with the image resolved to `docker.io/intel/intel-gpu-plugin:0.37.0`.
 - `kubectl kustomize clusters/base` renders, and the new Flux Kustomization appears as
  `intel-gpu-plugin  ./platform/intel-gpu-plugin  wait=true  dependsOn=cilium`. Both overlays
  (`clusters/prod`, `clusters/dev`) also render and carry it.
 - Across the whole rendered graph, **no Kustomization lists `intel-gpu-plugin` in `dependsOn`** —
  so `wait: true` gates nothing.
 - The vendored DaemonSet is a **formatting-only** transform of upstream, proven rather than
  asserted: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file `diff`
  clean, so passing it through kustomize changed no content. The committed body's sha256 matches
  the one recorded in its own provenance header (`80ebd070…`), so the file has not drifted from
  what was generated.
 Gate (pinned toolbox, from this worktree, with the `ENGINE` shim bind-mounting the main checkout's
 gitdir and `DEV_KUBECONFIG=/dev/null`): see the PR description for the per-stage table. **`rc 3`
 (UNDETERMINED), with exactly the three expected skips** — `spec-immutability` (sibling spec repo is
 not beside a worktree), `gitea-settings` (no CI token), `conformance` (no cluster in CI). Every
 other stage passed. **rc 3 is not green** and is not reported as such.
 ## What I did NOT verify
 **Nothing about this component's runtime behaviour. It has never run.**
 - **The plugin has never executed.** No pod has been scheduled, no image has been pulled, no scan
  has happened. It cannot run until the operator merges this and Flux reconciles — `kubectl apply`
  and `flux reconcile` are operator actions (CLAUDE.md §5), so this is expected, not an oversight.
 - **The resource has never been advertised.** `gpu.intel.com/i915` does not exist on any node and
  I have not seen it exist. The claim that it *will* be `i915` rather than `xe` is derived from
  source plus the established fact that `i915` is bound — it is a derivation, not an observation.
 - **The DaemonSet has never been admitted.** That the Kyverno policies do not match this namespace
  is read off the policy manifests; no admission request has actually been evaluated. Nor has the
  `kubectl kustomize` output ever been server-side dry-run against the API — rendering is not
  admission.
 - **The `open()` test has NOT been re-run.** The `EPERM` is exactly as it was. Nothing in this PR
  touches Plex, so Plex is still software-transcoding and will continue to until the later renderer
  PR lands. The decisive evidence is an `open()` as **uid 1000 through `s6-setuidgid plex`**;
  passing as uid 0 is weaker and must not be accepted as proof.
 - **A green gate proves the manifests render, nothing more.** And this gate was not green: it was
  `rc 3`. Note also that the Gitea `gate` job runs `./ci/gate.sh || [ "$?" -eq 3 ]`
  (`.gitea/workflows/gate.yml:138`), so a green tick on this PR cannot distinguish rc 0 from rc 3,
  and `vars.CLUSTER_AVAILABLE` is unset so no L2/L3 conformance runs in CI.
 - **`seLinuxOptions: type: container_device_plugin_t`** is carried verbatim from upstream. These
  nodes are Debian 13 and I did not measure whether SELinux is enabled or whether that type exists
  there. Expected to be inert (runc skips labelling when SELinux is disabled), but unverified — if
  the pods fail to start with an SELinux error, that line is the first suspect.
 - **Renovate has not been run** against this pin. The inline-marker idiom is copied from
  `platform/kubevirt` / `platform/external-snapshotter`, but whether Renovate normalises the
  `v`-prefix mismatch (release `v0.37.0` vs image `0.37.0`) for this dep is unknown. If it proposes
  `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in `renovate.json`.
 - **Resource limits unexamined under load.** The upstream 115Mi/100m limits are taken as-is; I have
  no measurement of this plugin's footprint on these nodes. Worth a glance given
  `keycloak-operator` is OOMing at 91% of its limit elsewhere in this cluster.
 - **`clusters/dev` renders it, and there is no dev cluster** (the operator confirmed the
  `k3s-dev-*` nodes are gone). That is inert rather than wrong — the unified graph (R-ENV-02) has
  both overlays render the same base — but it is not a thing I could test.
 - **Branch protection / required-checks state not re-measured** in this session. CLAUDE.md §2b
  says re-measure before relying on it; I did not, so treat the approval/gating claims in §2 as
  unconfirmed-by-me.
@@ -0,0 +1,225 @@
 # platform/intel-gpu-plugin
 **Intel Device Plugin for Kubernetes (GPU)** — the standalone GPU plugin DaemonSet from
 [`intel/intel-device-plugins-for-kubernetes`](https://github.com/intel/intel-device-plugins-for-kubernetes),
 pinned at **v0.37.0**. It makes every node advertise **`gpu.intel.com/i915`** so a workload can
 *request* a GPU instead of hoping a `hostPath` is enough.
 - **Serves:** hardware transcoding for Plex (`hardware.gpu: intel-quicksync`), and any later
  consumer of an Intel iGPU.
 - **Tier:** its **own** Flux Kustomization
  ([`clusters/base/intel-gpu-plugin.yaml`](../../clusters/base/intel-gpu-plugin.yaml),
  `dependsOn: cilium`, `wait: true`) — **not** a line in `platform/kustomization.yaml`. See
  *Placement* below.
 - **Namespace:** `intel-gpu-plugin` — a PLATFORM namespace, labelled
  `homelab.keithjackson.dev/app-namespace: "false"`, so the app-scoped Kyverno policies
  (default-deny generation, auth-mode, egress-class, host-namespaces) do not apply.
 - **No RBAC, no ServiceAccount, no CRDs, no webhook, no API-server access.** A device plugin talks
  to the *local kubelet* over a unix socket.
 ## What this component owns
 | File | Purpose |
 |---|---|
 | `namespace.yaml` | `intel-gpu-plugin` ns, explicitly `app-namespace=false` |
 | `intel-gpu-plugin.yaml` | the DaemonSet, **vendored** (machine-generated) from upstream v0.37.0 |
 | `kustomization.yaml` | namespace scoping + the pinned image (`images:` transformer) |
 ## The problem this fixes
 Plex was given a `/dev/dri` hostPath (merged on `main` @ `cdf3e14`). The device is **visible** in
 the container and **cannot be opened**. Measured in the running pod on `k3s-prod-2`:
 ```
 uid 0,    O_RDWR -> EPERM "Operation not permitted"
 uid 1000, O_RDWR -> EPERM "Operation not permitted"
 ```
 `EPERM` from the **device cgroup**, not `EACCES` from file permissions — proven three ways:
 1. root on the **host** opens the same device with `rc=0`;
 2. the node is unified `cgroup2fs`;
 3. the container's runc `config.json` reads
   `linux.resources.devices: [{'allow': False, 'access': 'rwm'}]` and `linux.devices: None` —
   one deny-all rule, no allow entry for major 226, and the runtime was never asked to grant a
   device.
 A hostPath is a **filesystem** operation. It makes the device *node* appear and adds no cgroup
 allow rule, because Kubernetes cannot infer one from a volume.
 **A device plugin is the mechanism that closes exactly this gap.** Verified in the v0.37.0 source
 (`cmd/gpu_plugin/gpu_plugin.go`, `createDeviceSpecsFromDrmFiles`), the `Allocate` response carries
 per DRM node:
 ```go
 specs = append(specs, pluginapi.DeviceSpec{
    HostPath: devPath, ContainerPath: devPath, Permissions: "rw",
 })
 ```
 and it is that `DeviceSpec` which makes kubelet populate `linux.devices` **and** the matching
 cgroup allow rule. It is the only **non-privileged** way to get there. `privileged: true` would
 also work and is not available: `platform/kyverno/policies/restrict-host-namespaces.yaml` blocks it
 under Enforce in app namespaces, and `plex` is labelled `app-namespace=true` (measured
 2026-10-01). Correctly so.
 ## Placement — why its own Kustomization
 The cheap option was one line in the shared `platform` tier. Neither that file nor a new
 `platform/<name>/` directory is a sensitive path in `infra/gitea/approval-policy.json`, so it would
 have needed no human reviewer *on the policy's terms*. It is still the wrong place, and the repo
 already argued the case when Cilium was split out (read the comment in
 `platform/kustomization.yaml`):
 * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot distinguish
  "the GPU plugin is wedged" from "longhorn is slow".
 * The only emergency valve would be `flux suspend kustomization platform` — freezing storage, TLS,
  DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device plugin.
 * The tier's 10m/5m interval/timeout are called out there as **wrong for a 3-node DaemonSet
  rollout**. This is a 3-node DaemonSet rollout.
 The consequence is worse than average here because of what sits downstream: `platform` gates
 `platform-config`, which gates `apps-wave-0` (`wait: true`), which gates `apps-wave-1`. A plugin
 that never went healthy inside that tier would stall **every app** on "dependency not ready" — so a
 component added to fix a *silent no-op* (software transcoding) would have acquired the ability to
 cause a *real outage*.
 The usual objection — that `clusters/**` needs a human approval — costs nothing: Gitea enforces
 `required_approvals: 1` with `block_admin_merge_override: true` on **every** PR here regardless of
 path, and authors cannot self-approve. The approval was required either way.
 ### `wait: true` is a health signal, not a gate
 This departs from the `headlamp`/`crafty` leaf template, deliberately. Those run `wait: false`
 because a leaf UI can be legitimately not-Ready for a long time during a cold start while it waits
 on Keycloak/DNS/TLS, making its health useless as a signal. This DaemonSet has no such dependency —
 it reads sysfs and opens a socket, so on healthy nodes it is Ready in seconds and **not-Ready
 genuinely means broken**. `wait: true` makes `flux get kustomization intel-gpu-plugin` a one-line
 answer to "did it come up on all three nodes", which is precisely the question that must be
 answered before the consumer PR may merge.
 Nothing in the graph `dependsOn` this, so `wait: true` gates nothing. **Think hard before adding
 `dependsOn: [intel-gpu-plugin]`** anywhere — for the app waves that would hand a device plugin the
 power to stall every app, which is the thing this placement exists to prevent.
 ## Deliberate non-choices
 **Standalone DaemonSet, not the operator.** The operator adds a CRD (`GpuDevicePlugin`), a
 controller-manager Deployment, cert-manager-backed admission webhooks and a second namespace —
 three more things that can be unhealthy — to buy declarative management of a DaemonSet that is
 already declared in git. For three identical nodes and one consumer that is strictly more blast
 radius for no gain.
 **It lists devices; it never opens them** — worth being certain about, since *opening* is exactly
 what fails today. Verified in v0.37.0 source: the scan path uses only `os.ReadDir`, `os.ReadFile`,
 `os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. There is no `os.Open`, no
 `OpenFile`, no `ioctl` and no raw syscall against a DRM node anywhere in
 `cmd/gpu_plugin/gpu_plugin.go`. Listing and `stat()` demonstrably work today — that is how the Plex
 probe found the device before failing to open it — so the plugin needs no cgroup grant of its own
 and no privilege. Accordingly `/dev/dri` and `/sys/class/drm` are mounted **readOnly**.
 * **`--health-management` stays OFF** (upstream default). It is the one flag that *would* need to
  open a device: it starts a Level-Zero sidecar (`setupLevelZeroService` returns early unless the
  flag is set). Enabling it would reintroduce the exact `EPERM` this component exists to fix.
 * `/dev/dri/by-path` being absent is non-fatal — `newDevicePlugin` warns and sets
  `bypathFound = false`.
 **`--shared-dev-num` left at the upstream default of 1** (i.e. not set). The flag multiplies each
 card into N schedulable device IDs, letting N containers share one GPU. Not raised, because:
 * Plex is the only consumer today and wants one device;
 * the count is a **lie the scheduler believes** — there is no GPU-side partitioning or quota behind
  it, so N containers get the same `renderD128` and contend for the same two media engines
  (`vcs0`/`vecs0`). Raising it converts "Unschedulable", which is loud and obvious, into
  "transcodes are mysteriously slow", which is neither;
 * it is a one-line, zero-migration change on the day a second consumer exists — and *then* the
  right number will be known rather than guessed.
 **No NFD overlay.** All three nodes are identical (HP EliteDesk 800 G5 Mini, i7-9700T, UHD Graphics
 630, PCI `8086:3e98`, `i915` bound), so a nodeSelector on a GPU feature label would gate on
 something true everywhere. If a GPU-less node ever joins, add upstream's `nfd` overlay *then*
 rather than relying on the plugin's behaviour on such a node.
 **No `supplementalGroups`, anywhere.** The `pms-docker` image's own cont-init hook stats each
 `/dev/dri` char device and adds the `plex` user to its group — measured live as
 `Groups: 44 100 992 1000` on the running Plex process, where GID 992 is the host `render` group.
 That mechanism is image-side and works identically on a plugin-injected device.
 ## Resource name
 `gpu.intel.com/i915`. Not a guess: `scan()` calls `devTree.AddDevice(devProps.driver(), ...)`, so
 the suffix is the **kernel driver name**. These nodes have `i915` bound (not `xe`), so the resource
 is `i915`. A future kernel moving CoffeeLake to `xe` would silently rename it to `gpu.intel.com/xe`
 and a consumer's request would stop matching — worth knowing before that kernel arrives.
 ## Versioning
 Pinned to **v0.37.0** via the `images:` transformer in `kustomization.yaml`; the DaemonSet itself is
 vendored as `kubectl kustomize` output of the upstream base (see its header for the exact re-vendor
 command and both sha256s). It is *generated* rather than a byte copy because upstream mixes YAML
 sequence-indentation styles within the one document and `ci/yamllint.yaml` sets
 `indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate
 (measured: it did). `kubectl kustomize` normalises the formatting, and the result was proven to be
 a formatting-only change: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file
 diff clean. This is also the re-vendor idiom `platform/external-snapshotter` documents.
 Vendored rather than referenced as a remote kustomize base for the same reason as
 `platform/external-snapshotter`: a remote base is `git fetch`ed on **every** reconcile, those
 fetches intermittently exceeded kustomize-controller's 27s timeout, and transient GitHub slowness
 then rendered as "kustomize build failed". A local file makes the build offline and deterministic —
 which also means the CI gate's `kustomize build` proves the manifests rather than proving GitHub
 was up.
 Renovate tracks it through the inline-marker idiom (`# renovate: datasource=github-releases
 depName=...` immediately above the version literal), because `renovate.json` sets
 `"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}` — an image reference in a
 plain manifest is tracked by nothing. **Caveat:** upstream *release* tags carry a `v` (`v0.37.0`)
 while *image* tags do not (`0.37.0`). Whether Renovate normalises that for this dep is unverified;
 if it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
 `renovate.json`, not a hand-edit.
 ## Ordering — do not reorder the stack
 The renderer change (`catalog/render.py` emitting `gpu.intel.com/i915: 1` for
 `hardware.gpu: intel-quicksync` instead of the hostPath) **must not land until this plugin is
 merged AND confirmed advertising the resource on all three nodes.** If the renderer requests a
 resource no node advertises, the Plex pod is **Unschedulable**; `apps-wave-0` is `wait: true` and
 `apps-wave-1` `dependsOn` it, so that stalls the whole app tier — turning today's silent no-op into
 a real outage.
 ## Post-merge verification
 Nothing about this component's runtime behaviour can be verified from a PR: it does not exist until
 the operator merges and Flux reconciles. Afterwards:
 ```bash
 D=./infra/devcontainer/dev.sh
 # 1. Flux applied it, and (wait:true) the DaemonSet went healthy.
 $D run bash -c 'flux get kustomization intel-gpu-plugin'
 $D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin get ds intel-gpu-plugin'
 #    want: DESIRED=3 CURRENT=3 READY=3 AVAILABLE=3
 # 2. The resource is advertised on ALL THREE nodes. This is the gate on the renderer PR.
 $D run bash -c 'kubectl --context k3s-prod get nodes -o json \
  | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
 #    want: three lines, each i915=1   (before this change: ABSENT on all three)
 # 3. If a node shows ABSENT, read that node's plugin logs.
 $D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin logs -l app=intel-gpu-plugin --tail=50 --prefix'
 ```
 ### The decisive test
 Resource-advertised is necessary but **not** sufficient — it proves the plugin registered, not that
 the cgroup allow rule reaches Plex. The decisive test is re-running the `open()` probe **as uid 1000
 through `s6-setuidgid plex`** inside the Plex pod, *after* the renderer PR has switched Plex to
 request the resource:
 ```bash
 $D run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
  s6-setuidgid plex python3 -c "import os; os.close(os.open(\"/dev/dri/renderD128\", os.O_RDWR)); print(\"rc=0 OK\")"'
 ```
 **Passing as uid 0 is weaker evidence and must not be accepted as proof** — Plex transcodes as uid
 1000, and the two differ in group membership, which is the half the image's cont-init hook supplies.
@@ -0,0 +1,117 @@
 ---
 # VENDORED, MACHINE-GENERATED, from intel/intel-device-plugins-for-kubernetes @ v0.37.0:
 #   deployments/gpu_plugin/base
 #
 # RE-VENDOR on a version bump (and bump `newTag` in kustomization.yaml to match):
 #   kubectl kustomize \
 #     "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/base?ref=vX.Y.Z" \
 #     > platform/intel-gpu-plugin/intel-gpu-plugin.yaml
 #   # then re-add this header, and READ THE DIFF — upstream has changed the securityContext
 #   # and the volume set between minor releases before.
 #
 # Generated 2026-10-01 at v0.37.0; sha256 of the body below (header excluded):
 #   80ebd070f983f3116f4004f719d9c872220997ec766b4ab7e7290585ac8ef842
 # The raw upstream source file reviewed to produce it
 # (deployments/gpu_plugin/base/intel-gpu-plugin.yaml) was sha256
 #   c8773918c2370e711d4ba84120a9971a290bf75ecf81dc520782bcccd7ef24b6
 # and the two were proven EQUIVALENT, not merely similar: `yq -P 'sort_keys(..)'` of each diffs
 # clean, so passing it through kustomize changed formatting only and no content.
 #
 # WHY GENERATED RATHER THAN A BYTE COPY of the upstream file: upstream mixes YAML sequence
 # indentation styles within the one document (`containers:` non-indented, `env:` indented), and
 # ci/yamllint.yaml sets `indentation: indent-sequences: consistent`, so a verbatim copy is a hard
 # lint ERROR and blocks the merge gate — measured, this file failed exactly that way first time
 # round. `kubectl kustomize` normalises the output, which is also precisely the re-vendor idiom
 # platform/external-snapshotter documents for the same reason.
 #
 # WHY VENDORED AT ALL rather than a remote kustomize base (`...gpu_plugin/base?ref=v0.37.0` left in
 # `resources:`): the same reason platform/external-snapshotter is vendored. A remote base is
 # `git fetch`ed by Flux's kustomize-controller on EVERY reconcile, those fetches intermittently
 # exceeded the controller's 27s timeout, and a transient GitHub slowness then rendered as
 # "kustomize build failed" on the whole tier. A local file makes the build OFFLINE and
 # deterministic — which also means `kustomize build` in the CI gate proves the real thing instead
 # of proving GitHub was up. (platform/kubevirt still carries the remote-base pattern and its latent
 # fragility; not this component's problem.)
 #
 # DO NOT HAND-EDIT THE BODY. Every local change belongs in kustomization.yaml as a transformer or
 # patch, so the next re-vendor stays a mechanical regenerate-and-diff instead of a three-way merge.
 # The image tag below is re-pinned (and fully qualified) by the `images:` transformer there. The
 # rationale for each thing deliberately NOT changed — no privileged, no supplementalGroups, no
 # -shared-dev-num, no NFD overlay, no health-management — is in README.md.
 apiVersion: apps/v1
 kind: DaemonSet
 metadata:
  labels:
    app: intel-gpu-plugin
  name: intel-gpu-plugin
 spec:
  selector:
    matchLabels:
      app: intel-gpu-plugin
  template:
    metadata:
      labels:
        app: intel-gpu-plugin
    spec:
      containers:
      - env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        image: intel/intel-gpu-plugin:0.37.0
        imagePullPolicy: IfNotPresent
        name: intel-gpu-plugin
        resources:
          limits:
            cpu: 100m
            memory: 115Mi
          requests:
            cpu: 40m
            memory: 45Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          seLinuxOptions:
            type: container_device_plugin_t
          seccompProfile:
            type: RuntimeDefault
        volumeMounts:
        - mountPath: /dev/dri
          name: devfs
          readOnly: true
        - mountPath: /sys/class/drm
          name: sysfsdrm
          readOnly: true
        - mountPath: /var/lib/kubelet/device-plugins
          name: kubeletsockets
        - mountPath: /var/run/cdi
          name: cdipath
      nodeSelector:
        kubernetes.io/arch: amd64
      volumes:
      - hostPath:
          path: /dev/dri
        name: devfs
      - hostPath:
          path: /sys/class/drm
        name: sysfsdrm
      - hostPath:
          path: /var/lib/kubelet/device-plugins
        name: kubeletsockets
      - hostPath:
          path: /var/run/cdi
          type: DirectoryOrCreate
        name: cdipath
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
@@ -0,0 +1,99 @@
 ---
 # Intel GPU device plugin for Kubernetes — makes the nodes advertise `gpu.intel.com/i915`.
 #
 # WHAT THIS FIXES, and why nothing cheaper works. Plex already has a `/dev/dri` hostPath mount
 # (merged on main @ cdf3e14). The device node is VISIBLE in the container and cannot be OPENED.
 # Measured in the running pod on k3s-prod-2:
 #     uid 0,    O_RDWR -> EPERM "Operation not permitted"
 #     uid 1000, O_RDWR -> EPERM "Operation not permitted"
 # EPERM from the DEVICE CGROUP, not EACCES from file permissions — proven three ways: root on the
 # HOST opens the same device rc=0; the node is unified cgroup2fs; and the container's runc
 # config.json carries one deny-all rule and nothing else:
 #     linux.resources.devices: [{'allow': False, 'access': 'rwm'}]
 #     linux.devices: None
 # No allow entry for major 226, and the runtime was never asked to grant a device. A hostPath is a
 # FILESYSTEM operation: it makes the device node appear, and adds no cgroup allow rule, because
 # Kubernetes cannot infer one from a volume.
 #
 # A device plugin is the mechanism that closes exactly that gap. Verified by reading the v0.37.0
 # source (cmd/gpu_plugin/gpu_plugin.go, createDeviceSpecsFromDrmFiles): the Allocate response
 # carries `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, and it
 # is that DeviceSpec which makes kubelet populate `linux.devices` AND the matching cgroup allow
 # rule. It is the only NON-PRIVILEGED way to get there. `privileged: true` would also work and is
 # not available: platform/kyverno/policies/restrict-host-namespaces.yaml blocks it under Enforce in
 # app namespaces, and `plex` is labelled app-namespace=true (measured 2026-10-01). Correctly so.
 #
 # STANDALONE DAEMONSET, NOT THE OPERATOR. Upstream ships both. The operator adds a CRD
 # (GpuDevicePlugin), a controller-manager Deployment, cert-manager-backed admission webhooks and a
 # second namespace — i.e. three more things that can be unhealthy — to buy declarative management
 # of a DaemonSet we are declaring in git anyway. For three identical nodes and one consumer that is
 # strictly more blast radius for no gain. The DaemonSet has no CRDs, no controller, no webhook, and
 # no API-server access at all: a device plugin talks to the LOCAL kubelet over a unix socket, which
 # is why there is no ServiceAccount or RBAC here (and therefore no `**/rbac.yaml` sensitive path).
 #
 # LISTING, NOT OPENING — the thing worth being sure about, since opening is precisely what fails
 # today. Verified in v0.37.0 source: the scan path uses only os.ReadDir / os.ReadFile / os.Stat /
 # os.Readlink over /sys/class/drm and /dev/dri. There is no os.Open, no OpenFile, no ioctl and no
 # syscall of any kind against a DRM node anywhere in cmd/gpu_plugin/gpu_plugin.go. Listing and
 # stat() demonstrably work today (that is how the Plex probe found the device before failing to
 # open it), so the plugin needs no cgroup grant of its own and no privilege.
 #   * `--health-management` is left OFF (upstream default). It is the one flag that WOULD need to
 #     open a device: it starts a Level-Zero sidecar (setupLevelZeroService returns early unless the
 #     flag is set). Turning it on would reintroduce the exact EPERM this component exists to fix.
 #   * /dev/dri and /sys/class/drm are mounted readOnly, consistent with list-and-stat only.
 #   * /dev/dri/by-path absence is non-fatal (newDevicePlugin warns and sets bypathFound=false).
 #
 # RESOURCE NAME: `gpu.intel.com/i915`. Not a guess — `scan()` does
 # `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the KERNEL DRIVER name. These nodes
 # have i915 bound (not xe) on UHD Graphics 630 / PCI 8086:3e98, so the resource is i915. A future
 # kernel moving CoffeeLake to xe would silently rename it to `gpu.intel.com/xe` and the consumer's
 # request would stop matching — worth knowing before that kernel arrives.
 #
 # `--shared-dev-num` LEFT AT THE UPSTREAM DEFAULT OF 1 (so: not set). The flag multiplies each
 # card into N schedulable device IDs (`for i := 0; i < sharedDevNum; i++`), letting N containers
 # share one GPU. Deliberately not raised:
 #   * Plex is the only consumer and wants one device.
 #   * The count is a LIE the scheduler believes. There is no GPU-side partitioning or quota behind
 #     it — N containers get the same renderD128 and contend for the same two media engines
 #     (vcs0/vecs0). Raising it converts "Unschedulable", which is loud and obvious, into
 #     "transcodes are mysteriously slow", which is neither.
 #   * It is a one-line, zero-migration change on the day a second consumer actually exists, and at
 #     that point the right number will be known rather than guessed.
 # No NFD overlay either: all three nodes are identical (established cluster fact), so a nodeSelector
 # on a GPU feature label would gate on something that is true everywhere. If a GPU-less node ever
 # joins, add upstream's nfd overlay THEN rather than relying on the plugin's behaviour there.
 #
 # NO supplementalGroups, anywhere, on purpose. The pms-docker image's own cont-init hook stats each
 # /dev/dri char device and adds the `plex` user to its group — measured live as `Groups: 44 100 992
 # 1000` on the running Plex process, where GID 992 is the host `render` group. That mechanism is
 # image-side and works identically on a plugin-injected device, so adding a supplementalGroup here
 # or on the consumer would duplicate a thing that already works.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: intel-gpu-plugin
 resources:
  - namespace.yaml
  - intel-gpu-plugin.yaml
 images:
  # PINNED, never `latest`. Upstream release v0.37.0 (2026-09-16); verified present on Docker Hub
  # 2026-10-01, manifest digest sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1
  # (single amd64 manifest, not an index). Recorded here rather than pinned by digest because this
  # repo pins upstream platform components by VERSION TAG (platform/kubevirt, external-snapshotter);
  # digest pinning is reserved for the conformance runner, where it is a security boundary.
  # `newName` fully qualifies the registry: upstream writes the bare `intel/intel-gpu-plugin`, which
  # relies on containerd's docker.io default rather than saying so.
  #
  # RENOVATE: the inline-marker idiom (customManagers in renovate.json), not the docker datasource —
  # renovate.json sets `"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}`, so an
  # image reference in a plain manifest is tracked by NOTHING. The marker must sit on the line
  # IMMEDIATELY above the version literal for the regex to match, which is why it is down here on
  # `newTag` and not up in this header.
  # CAVEAT, stated because it will bite someone: the GitHub release tags carry a `v`
  # (`v0.37.0`) and the IMAGE tags do not (`0.37.0` — confirmed against the Docker Hub tag list).
  # Whether Renovate normalises that for this dep is UNVERIFIED here (no Renovate run available).
  # If it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
  # renovate.json, not a hand-edit of this line.
  - name: intel/intel-gpu-plugin
    newName: docker.io/intel/intel-gpu-plugin
    # renovate: datasource=github-releases depName=intel/intel-device-plugins-for-kubernetes
    newTag: "0.37.0"
@@ -0,0 +1,34 @@
 ---
 # intel-gpu-plugin — namespace for the Intel GPU device plugin DaemonSet.
 #
 # PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and that distinction is load-bearing here rather
 # than cosmetic. Every Kyverno policy that could object to this DaemonSet is scoped by a
 # `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"` — verified by
 # reading the policies, not assumed:
 #   * platform/kyverno/policies/restrict-host-namespaces.yaml  (hostNetwork/PID/IPC + privileged)
 #   * platform/kyverno/policies/require-egress-class-label.yaml
 #   * platform/kyverno/policies/require-auth-mode-label.yaml
 #   * platform/kyverno/policies/require-default-deny-baseline.yaml (a *generate* rule, so an
 #     app-namespace also gets a default-deny NetworkPolicy minted into it)
 # Labelling this namespace "false" therefore puts it out of scope for all four. That is the same
 # mechanism restrict-host-namespaces' own header describes: "platform components that do [need host
 # access] live in platform namespaces that do NOT carry the app-namespace label".
 #
 # Measured on prod 2026-10-01: `kyverno` and `vpn-egress` are the two namespaces that carry the
 # label explicitly as "false"; the rest of the platform namespaces leave it unset. Both are out of
 # scope identically — an unset label does not match `"true"`. We set it EXPLICITLY, following
 # vpn-egress, because "someone chose this" and "nobody thought about it" should not look the same
 # in git, and because this component's correctness depends on the answer.
 #
 # NOTE this plugin does NOT actually need any of the things those policies forbid: it sets no
 # hostNetwork/hostPID/hostIPC and is not privileged (see intel-gpu-plugin.yaml). The namespace
 # placement is not a workaround for a policy — there is nothing to work around. It is here because
 # a node-level platform DaemonSet is not an application, and because the app-namespace fabric
 # (default-deny NetworkPolicy, egress-class, auth-mode) describes obligations that are meaningless
 # for a process that talks only to the local kubelet over a unix socket and never uses the network.
 apiVersion: v1
 kind: Namespace
 metadata:
  name: intel-gpu-plugin
  labels:
    homelab.keithjackson.dev/app-namespace: "false"
@@ -115,6 +115,14 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
  plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
  token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
  Includes why a changed Secret does not restart anything, and why an expired token is harmless.
 - **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod
  (`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware
  acceleration when available"**, and no environment variable can do it (the image supports no
  `PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`,
  read out of the binary), says which widely-cited key **does not exist**, and is explicit that the
  **only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no
  alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin
  fallback, and why a `:latest` pull can silently break it.
 - **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
  re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
  (the four lines that matter, including the upstream `-` that means the proxy answered a request the
@@ -95,6 +95,12 @@ configMapGenerator:
      # because the plex.tv claim window is short and is not a thing to go looking
      # for in git: /runbooks/plex-claim has to resolve the moment it is needed.
      - plex-claim=plex-claim.md
      # Operator procedure, not an alert page, and NO alert is possible for it: plex renders no
      # metricsEndpoint, so there is no ServiceMonitor, no transcode-mode metric and nothing a rule
      # could evaluate. Served because the feature it completes is INERT until a human ticks a box in
      # the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped
      # feature stays switched off for months.
      - plex-quicksync=plex-quicksync.md
      - pvc-capacity=pvc-capacity.md
      - targets-down=targets-down.md
      - update-rollback=update-rollback.md
@@ -0,0 +1,283 @@
 # plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did
 **Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**,
 not an incident page. It is indexed here because the feature it completes is **inert until a human
 clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
 to prevent.
 **Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the
 rendered golden — the `dri` hostPath volume and the `/dev/dri` mount), `catalog/render.py` (the
 mechanism), `schemas/renderer-contract.md` `### hardware` (the contract).
 **Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile.
 ## What this means here
 Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync
 moves that work off the CPU — which matters on a three-node cluster where the CPU is also running
 everything else. Two independent things have to be true before that happens, and **only one of them
 is in git**:
 | Half | Who does it | State |
 |---|---|---|
 | The render device is inside the container at `/dev/dri` | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders a `dri` hostPath volume (`/dev/dri`, `type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container. Flux applies it. | Done by merging that change. |
 | Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
 **There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
 grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`,
 `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin` — found **zero** `PLEX_PREFERENCE*` support.
 Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentifier`,
 `ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`,
 `TranscoderTempDirectory`), none of them the transcode toggle. No environment variable, no ConfigMap
 and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this;
 there isn't one, and the search has already been done.
 What the device mount does **not** do, so nobody reads more into a green gate than is there: it does
 not enable hardware transcoding, it does not prove the device can be *opened* from inside a
 non-privileged pod (see "Likely causes" #2), and it renders no `nodeSelector` — Plex can land on any
 of the three nodes and all three carry the device.
 ## Running the commands in this runbook
 `kubectl` does not work on the operator workstation; the toolchain lives only in the pinned dev
 container. Every command below runs as:
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
 Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
 picks a current-context **per invocation**, and that default has already changed once. Do not run two
 `dev.sh` invocations at once.
 **Never `cat` `Preferences.xml`.** It contains `PlexOnlineToken`, a plex.tv account credential. Every
 read of that file below greps a single named key — do not widen one.
 ## First commands
 **1. Is the device mounted into the pod at all?** (Proves Flux applied the change, not that it works.)
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
     -o jsonpath="{.items[0].spec.containers[0].volumeMounts[*].mountPath}"; echo'
 ```
 `/dev/dri` in that list ⇒ the GitOps half landed. Absent ⇒ the running pod predates the merge; compare
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 \
     -o jsonpath="{.status.lastAppliedRevision}"; echo'
 ```
 against the merge commit before blaming anything else.
 **2. Is the device actually visible inside the container?**
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'
 ```
 Expect `renderD128` (usually `card0` too), mode `crw-rw----`, group `render`. An **empty directory**
 here is the `DirectoryOrCreate` degradation: the kubelet created an empty `/dev/dri` because the node
 had none. That is the designed-safe outcome — the app tier still reconciles — but it means software
 transcoding, silently. Find which node the pod is on and ask why that node lost its iGPU:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
 ```
 **3. Can Plex *open* it?** This is the question the manifest cannot answer — see "Likely causes" #2.
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
     bash -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
 ```
 `RW-OPEN-DENIED` (or `Operation not permitted` / `Permission denied`) with the device plainly visible
 in step 2 is the **device-cgroup** failure, not a file-permission failure. The two look identical from
 the outside and are not the same problem.
 **4. Is the toggle on?** Named key only — never the whole file:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
     grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
     "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'
 ```
 `ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on,
 and the remaining question is whether it *works* — go to "How to verify".
 ## The one manual step
 Plex web UI → **Settings → Transcoder → "Use hardware acceleration when available"** → tick → Save.
 (It sits under *Show Advanced*.) A transcode already in flight is unaffected; the next one picks it up.
 The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by
 extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from
 recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise
 them and not invent them:
 | Key | What it is |
 |---|---|
 | `HardwareAcceleratedCodecs` | the toggle this page is about |
 | `HardwareAcceleratedEncoders` | hardware **encode**, as distinct from decode |
 | `HardwareDevicePath` | how Plex is pointed at a *specific* render node — relevant only if a host ever exposes more than one |
 | `TranscoderToneMapping`, `TranscoderToneMappingAgorithm` | HDR→SDR tone mapping. **That misspelling of "Algorithm" is upstream's, in the binary. It is not a typo in this page — do not "fix" it** here or anywhere else; the corrected spelling is not a key Plex reads. |
 | `TranscoderHEVCEncoding` | HEVC output |
 **`HardwareAcceleratedCodecsEnableHEVC` does not exist.** It is widely cited and it is not in the
 binary. Recorded explicitly so nobody spends an evening hunting for it or, worse, writes it into
 `Preferences.xml` and concludes hardware transcoding is broken.
 Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn
 tier, it is in no git commit, and the UI writes the key correctly. Use the UI.
 ## How to verify — and why there is no alert
 Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a
 lower quality explicitly), then read Plex's own **Dashboard → Activity**:
 - **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.**
 - plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says.
 **That dashboard is the *only* proof available, and nothing else will ever tell you.**
 `catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
 there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work
 to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently
 inert, and this repo has already been burned at scale by rules that evaluated against nothing while
 looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that
 gap is ever worth closing, the honest order is a real exporter first and a rule second.
 Corroborating, not a substitute — the server log on the config PVC:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
     sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
     \"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'
 ```
 Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions.
 Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence
 of hardware. The dashboard remains the verdict.
 ## Likely causes, most likely first
 When the dashboard still says plain `Transcode`:
 1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists.
   First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
   investigating anything technical.
 2. **containerd's device cgroup is refusing the `open()` — UNVERIFIED, and the largest remaining
   technical risk in this feature.** Whether a non-privileged pod may `open()` a bind-mounted
   character device on this k3s/containerd build **has not been established**; it can only be settled
   by deploying and trying. The container inits as uid 0, which settles *file* permissions, but
   **uid 0 does not bypass the cgroup device gate.** Symptom: step 2 shows `renderD128` present and
   correctly owned, step 3 returns `RW-OPEN-DENIED`.
   **The documented fallback is the Intel Device Plugin for Kubernetes** (`gpu.intel.com/i915` as a
   resource request, no hostPath). Because the ServiceDefinition field expresses *intent* rather than
   a device path, that migration is a **`catalog/render.py` change plus a re-render — with no
   ServiceDefinition edit anywhere**; `catalog/plex.yaml` keeps saying exactly what it says today.
   **`privileged: true` is not an available escape hatch:** Kyverno `restrict-host-namespaces` is in
   **Enforce** mode and rejects it in app namespaces. Do not reach for it.
 3. **The pod landed on a node without the device.** Step 2 shows an empty `/dev/dri`. All three prod
   nodes carried `renderD128` with `i915` bound when this was measured (2026-09-30), so this means a
   node changed — a rebuild, a BIOS reset, or a driver regression. `DirectoryOrCreate` is why the app
   tier still came up; the cost is that the loss is silent. **Fix the node, do not tighten the mount:**
   `Directory` would stall `apps-wave-0` (`wait: true`, with `apps-wave-1` depending on it) for a
   feature that is not load-bearing.
 4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
   mechanism this feature leans on with no commit in this repo.
 5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so
   `Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as
   a hardware failure.
 6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which
   Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
   sessions show `(hw)` and others do not. Not a fault.
 ## The floating-image caveat
 Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
 user to the device's group, the absence of `PLEX_PREFERENCE*`, the six `setPref` keys, the
 `HardwareAcceleratedCodecs` string — was read from digest **`sha256:e0ab27395614…`**
 (`plexmediaserver 1.43.4.10903`).
 The deployment runs **`plexinc/pms-docker:latest` with `imagePullPolicy: Always`**. A silent pull can
 therefore change the behaviour this feature depends on **with no commit in this repo**, while the
 manifest still looks identical. Symptom to watch for: hardware transcoding that worked and quietly
 stopped, with no git change to point at.
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
     -o jsonpath="{.items[0].status.containerStatuses[0].imageID}"; echo'
 ```
 If that digest is not the one above, **re-verify the hook before trusting this page**:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- ls /etc/cont-init.d/'
 ```
 `45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one
 described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups`
 needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that
 hook exists.
 ## When to escalate or stop
 - **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be
  **inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's
  `cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its
  group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces**
  the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with
  `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
  discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
 - **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
  reject it. If the cgroup gate is the blocker, the answer is the Intel device plugin (cause #2).
 - **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles
  `main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating
  `kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change
  through a branch, a PR and a green gate.
 - **Do not edit `Preferences.xml` on the PVC**, never `cat` it, and never `sops -d` anything into a
  terminal. It holds `PlexOnlineToken`.
 - **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and
  watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
  lose the DB.
 - **Stop and get the operator** if the device cannot be opened from the pod (cause #2 confirmed). The
  next step is a platform change — the Intel device plugin — with its own PR, not more poking at this
  workload.
 ## Known gaps, deliberately not fixed here
 Recorded so they are findable, and so nobody assumes they were overlooked:
 - **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
  `TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
  overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
  transcode scratch performance changes silently whenever the pod is rescheduled. Both candidate fixes
  need a **schema** change and are therefore out of scope here: pin Plex to prod-2 (needs a
  `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory` emptyDir and
  transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not have). The RAM option
  is the better one, because it removes the node asymmetry outright instead of working around it.
 - **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
  quality only — decode, and the feature on this page, are unaffected. Changing it needs a kernel
  parameter and a **serialized rolling reboot**, which is an operator action (CLAUDE.md §5), and this
  cluster has a documented history of that going badly (a prod node hung on a warm reboot on
  2026-09-03). Not worth coupling to this change.
 ---
 Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
 The node measurements (all three prod nodes) and the in-container image measurements cited here were
 taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
 (`plexmediaserver 1.43.4.10903`). Hardware transcoding had **not** been observed working at the time
 of writing, and the Plex-side toggle was still unset.
Author	SHA1	Message	Date
jakman85andClaude Opus 5	4e1b169256	feat(gpu): Intel GPU device plugin v0.37.0, in its own Flux Kustomization gate / gate (pull_request) Successful in 1m41s Details gate / conformance (pull_request) Has been skipped Details Makes the three prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the Plex hardware-transcoding blocker, and it only makes the resource EXIST — no consumer is switched over here. WHY. The `/dev/dri` hostPath merged on main @ `cdf3e14` makes the device node visible and not openable. Measured in the running pod on k3s-prod-2: uid 0, O_RDWR -> EPERM "Operation not permitted" uid 1000, O_RDWR -> EPERM "Operation not permitted" EPERM from the device cgroup, not EACCES from file permissions — root on the HOST opens the same device rc=0, the node is unified cgroup2fs, and the container's runc config.json carries one deny-all rule with `linux.devices: None`. A hostPath is a filesystem operation: it makes the device node appear and adds no cgroup allow rule, because Kubernetes cannot infer one from a volume. A device plugin's Allocate response is what makes kubelet populate `linux.devices` AND the matching cgroup allow rule — verified in the v0.37.0 source, which emits `DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node. It is the only non-privileged way to get there; `privileged: true` is blocked by Kyverno restrict-host-namespaces under Enforce in app namespaces, and `plex` is labelled app-namespace=true. PLACEMENT: its own Kustomization (dependsOn cilium, wait:true), not a line in platform/kustomization.yaml. The shared tier was cheaper and needed no human reviewer on approval-policy.json's terms, but it is wait:true over eleven-plus components and the repo already argued this when Cilium was split out: its Ready condition cannot tell "the plugin is wedged" from "longhorn is slow", the only emergency valve is freezing the whole tier, and the tier's 10m/5m interval/timeout are called out there as wrong for a 3-node DaemonSet rollout — which is exactly what this is. Worse here than average: platform gates platform-config gates apps-wave-0 (wait:true) gates apps-wave-1, so a plugin that never went healthy inside it would stall every app — a component added to fix a silent no-op would have gained the power to cause a real outage. Touching clusters/** costs nothing extra: required_approvals:1 with block_admin_merge_override:true applies to every PR here regardless of path. wait:true is a health SIGNAL, not a gate — nothing in the rendered graph dependsOn this, so it gates nothing, while making `flux get kustomization intel-gpu-plugin` answer "did it come up on all three nodes". NON-CHOICES, each argued in the README: standalone DaemonSet over the operator (no CRD, controller-manager, webhook or second namespace to keep healthy); `--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler believes — no GPU-side partitioning backs it, so it converts a loud Unschedulable into a quiet "transcodes are slow"); `--health-management` left OFF, the one flag that would open a device and reintroduce this EPERM; no NFD overlay (all three nodes identical); no supplementalGroups (the pms-docker cont-init hook already adds `plex` to each device's group, measured live as Groups: 44 100 992 1000). The DaemonSet is vendored as `kubectl kustomize` output of upstream v0.37.0's gpu_plugin/base, rather than referenced as a remote kustomize base, for the reason external-snapshotter records: a remote base is git-fetched on every reconcile and those fetches have flaked a whole tier. It is generated rather than byte-copied because upstream mixes YAML sequence-indentation styles within the one document and ci/yamllint.yaml sets `indent-sequences: consistent`, so a verbatim copy is a hard lint ERROR — measured, not guessed: the first gate run of this branch returned rc 1 BLOCKED on exactly that. The transform is proven formatting-only (`yq -P 'sort_keys(..)'` of both diffs clean). Image pinned to docker.io/intel/intel-gpu-plugin:0.37.0 (digest verified present: sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1). ORDERING — DO NOT REORDER. catalog/render.py, catalog/plex.yaml and apps/wave-0/plex.yaml are untouched. The renderer change that emits `gpu.intel.com/i915: 1` instead of the hostPath must NOT land until this plugin is merged AND confirmed advertising the resource on all three nodes: a request for a resource no node advertises makes the Plex pod Unschedulable, and apps-wave-0 is wait:true with apps-wave-1 depending on it. VERIFIED: manifests render offline (component, clusters/base, both overlays); no node currently advertises gpu.intel.com/* (queried — "NONE"); the plugin only lists/stats devices and never opens one (no os.Open/OpenFile/ioctl/syscall anywhere in its 908-line source); all four app-scoped Kyverno policies are gated on app-namespace=true and the new namespace is labelled "false". NOT VERIFIED: anything runtime. The plugin has never executed, the resource has never been advertised, and the open() test has not been re-run. Gate returned rc 3 (UNDETERMINED) with exactly the three expected skips — not green. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 22:25:33 -05:00
jakman85	cdf3e14e27	Merge pull request 'feat(plex): hardware transcoding via `hardware.gpu: intel-quicksync`, plus the runbook for the manual last mile' (#32 ) from feat/plex-quicksync into main gate / gate (push) Successful in 1m42s Details gate / conformance (push) Has been skipped Details Reviewed-on: #32 Reviewed-by: Keith Jackson <[email protected]>	2026-10-02 02:17:48 +00:00
jakman85	41a20dfd20	Merge pull request 'feat(schema): declare a host hardware capability, `hardware.gpu: intel-quicksync`' (#27 ) from feat/sd-hardware-capability into main gate / conformance (push) Has been cancelled Details gate / gate (push) Has been cancelled Details Reviewed-on: #27 Reviewed-by: Keith Jackson <[email protected]>	2026-10-02 02:17:23 +00:00
jakman85andClaude Opus 5	df1fd1b3d1	chore(decisions): record the rebase onto main @ `08deb20` gate / gate (pull_request) Successful in 1m43s Details gate / conformance (pull_request) Has been skipped Details The runbooks/README.md conflict was two index entries appended at the same point, not a semantic one. Both kept, in the file's grouped order. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 21:06:34 -05:00
jakman85andClaude Opus 5	0fee584e3a	feat(plex): opt into `hardware.gpu: intel-quicksync`, and write the runbook for the step GitOps cannot reach Plex has been transcoding in software on hardware that can do it in the iGPU. This opts its ServiceDefinition into the `hardware` capability from #27 and re-renders. catalog/plex.yaml gains four lines of YAML and the comment that explains them: hardware: gpu: intel-quicksync which renders exactly two things into apps/wave-0/plex.yaml and nothing else — a `dri` hostPath volume (/dev/dri, type: DirectoryOrCreate) and a read-write /dev/dri volumeMount on the app container. Whole-tree render diff: 6 added lines, one file. The comment records what is not obvious from the four lines: the device exists on ALL THREE prod nodes with i915 bound (so no nodeSelector is rendered and none is wanted); why `supplementalGroups` would be INERT rather than a hardening (the image's cont-init hook adds the plex user to the device's group inside the container, and s6-setuidgid then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied); and why `DirectoryOrCreate` is load-bearing (apps-wave-0 is wait:true with apps-wave-1 depending on it, so a failed mount would stall the whole app tier for a feature that is not). THE FEATURE IS INERT UNTIL A HUMAN CLICKS A BOX, and that is the part this commit is really about. Mounting the device does not enable hardware transcoding; Plex's `HardwareAcceleratedCodecs` preference does, and it is ABSENT from Preferences.xml — the toggle has never been on. It cannot be automated: a bounded grep across the running image's entire init surface (/etc/cont-init.d, /etc/services.d, /etc/fix-attrs.d, /plex-common.sh, /usr/local/bin) found ZERO PLEX_PREFERENCE* support, and the six setPref keys the image does write are all unrelated. So runbooks/plex-quicksync.md is the procedure, registered in the configMapGenerator (a runbook in git with no generator entry is reviewed, merged, and still a 404) and indexed in runbooks/README.md. It covers the GitOps/manual boundary, the toggle and the real preference keys (including the widely-cited one that does not exist and the upstream misspelling nobody should "fix"), verification via `Transcode (hw)` on the Plex dashboard — stated plainly as the ONLY proof, because Plex renders no ServiceMonitor and an alert could never fire — the device-cgroup failure to expect first with its Intel-device-plugin fallback, and the floating-`:latest` caveat. Stacked on #27 (feat/sd-hardware-capability), which is open and unmerged; that branch's commit is in this range by design and this must merge after it. NOT VERIFIED, and the handoff says so at length: hardware transcoding has not been observed working. Whether containerd's device cgroup lets a non-privileged pod open() a bind-mounted char device is an open question that only deploying settles, and the Plex-side toggle was still unset at the time of writing. A green gate proves the manifest renders; it proves nothing about transcoding. Gate (pinned toolbox, worktree): rc 3 — UNDETERMINED, not green. Passed: decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped (the known local ceiling): spec-immutability (no sibling spec repo), gitea-settings (no SETTINGS_API_TOKEN), conformance (CLUSTER_AVAILABLE unset, so L2/L3 did not run). No fourth skip, nothing failed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 21:05:17 -05:00