--- date: 2026-10-01 branch: feat/intel-gpu-device-plugin title: Intel GPU device plugin, in its own Kustomization — the resource exists; nothing has run yet --- ## What changed Deployed the **Intel Device Plugin for Kubernetes (GPU)**, standalone DaemonSet, pinned at **v0.37.0**, so the prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the Plex hardware-transcoding blocker: the `/dev/dri` hostPath merged on `main` @ `cdf3e14` makes the device node *visible* and not *openable* (`EPERM` from the device cgroup at both uid 0 and uid 1000; the container's runc `config.json` carries one deny-all rule and `linux.devices: None`). A hostPath is a filesystem operation and adds no cgroup allow rule. A device plugin's `Allocate` response is what makes kubelet populate `linux.devices` *and* the matching cgroup allow rule, and it is the only non-privileged way to get there — `privileged: true` is blocked by Kyverno `restrict-host-namespaces` under Enforce in app namespaces, and `plex` is labelled `app-namespace=true`. New files: - `platform/intel-gpu-plugin/intel-gpu-plugin.yaml` — the DaemonSet, **vendored** as `kubectl kustomize` output of upstream `deployments/gpu_plugin/base` at `v0.37.0`, under a provenance header carrying the exact re-vendor command and both sha256s (generated body `80ebd070…`, raw upstream source `c8773918…`). Generated rather than byte-copied because upstream mixes YAML sequence-indentation styles within the one document and `ci/yamllint.yaml` sets `indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate. That is not a guess: the first gate run of this branch returned **rc 1 BLOCKED** on exactly that error, and regenerating through kustomize is the fix (and is also the re-vendor idiom `platform/external-snapshotter` documents). Vendored at all, rather than referenced as a remote kustomize base, for the other reason `external-snapshotter` records: a remote base is `git fetch`ed on every reconcile and those fetches have flaked a whole tier. - `platform/intel-gpu-plugin/namespace.yaml` — a PLATFORM namespace, explicitly `homelab.keithjackson.dev/app-namespace: "false"` (following `vpn-egress`/`kyverno`). - `platform/intel-gpu-plugin/kustomization.yaml` — namespace scoping plus the image pin. - `platform/intel-gpu-plugin/README.md` — the measurement, the placement argument, the non-choices, and the post-merge verification sequence. - `clusters/base/intel-gpu-plugin.yaml` — **its own Flux Kustomization**: `dependsOn: [cilium]`, `wait: true`, `interval: 10m`, `timeout: 10m`, `prune: true`. - One wired line (plus its rationale comment) in `clusters/base/kustomization.yaml`. **Placement: its own Kustomization, not a line in the shared `platform` tier.** The shared tier was cheaper and needed no human reviewer on `approval-policy.json`'s terms, but `platform` is `wait: true` over eleven-plus components and the repo already argued this case when Cilium was split out: its Ready condition cannot distinguish "the plugin is wedged" from "longhorn is slow", the only emergency valve is freezing the entire tier, and the tier's 10m/5m interval/timeout are called out there as *wrong for a 3-node DaemonSet rollout* — which is exactly what this is. The consequence is worse than average here, because `platform` gates `platform-config` gates `apps-wave-0` (`wait: true`) gates `apps-wave-1`: a plugin that never went healthy inside that tier would stall every app, so a component added to fix a **silent no-op** would have gained the ability to cause a **real outage**. The approval cost of touching `clusters/**` is nil, because `required_approvals: 1` with `block_admin_merge_override: true` applies to every PR here regardless of path and authors cannot self-approve. `dependsOn: [cilium]` is the minimum *and* the earliest: the plugin never uses the network and never talks to the API server (it talks to the local kubelet over a unix socket), so a working CNI is its only real prerequisite, and `cilium` already depends on `flux-system`, which applies the `cluster-vars` ConfigMap this Kustomization substitutes from. Depending on `cilium` rather than `platform` also means it reconciles *in parallel with* the platform tier rather than behind all eleven of its components, which shrinks the cold-start window in which a node has not yet advertised the resource. `wait: true` is a deliberate departure from the `headlamp`/`crafty` leaf template. Those use `wait: false` because a leaf UI can be legitimately not-Ready for a long time during a cold start, making its health a useless signal. This DaemonSet reads sysfs and opens a socket, so not-Ready genuinely means broken — and since **nothing in the graph `dependsOn` it** (verified against the rendered graph), `wait: true` gates nothing while turning `flux get kustomization intel-gpu-plugin` into a one-line answer to the question that must be answered before the consumer PR may merge. Deliberate non-choices, each argued in the README: standalone DaemonSet rather than the operator (no CRD, no controller-manager, no cert-manager webhook, no second namespace to keep healthy); `--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler believes — there is no GPU-side partitioning behind it, so it converts a loud "Unschedulable" into a quiet "transcodes are slow"); `--health-management` left OFF, because it is the one flag that *would* open a device and would reintroduce the exact `EPERM` this fixes; no NFD overlay (all three nodes are identical); no `supplementalGroups` anywhere, because the `pms-docker` cont-init hook already adds the `plex` user to each device's group and that works identically on a plugin-injected device. **Scope held deliberately:** `catalog/render.py`, `catalog/plex.yaml` and `apps/wave-0/plex.yaml` are untouched. Switching Plex from the hostPath to a `gpu.intel.com/i915: 1` request is a strictly later PR, and **it must not land until this plugin is merged AND confirmed advertising the resource on all three nodes** — a request for a resource no node advertises makes the Plex pod Unschedulable, and `apps-wave-0` is `wait: true` with `apps-wave-1` depending on it, so that would stall the whole app tier and turn today's silent no-op into a real outage. ## What I verified Mechanism, read from the v0.37.0 source rather than assumed: - **The plugin only lists devices; it never opens one.** The scan path uses `os.ReadDir`, `os.ReadFile`, `os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. `grep` for `os.Open|OpenFile|ioctl|syscall|unix\.` across `cmd/gpu_plugin/gpu_plugin.go` (908 lines) returns **nothing**. Listing and `stat()` are what demonstrably work today, so the plugin needs no cgroup grant and no extra privilege — it does not need more than the upstream manifest gives it. - **The fix mechanism is real:** `createDeviceSpecsFromDrmFiles` emits `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, which is the thing kubelet turns into `linux.devices` plus a cgroup allow rule. - **The resource name is `gpu.intel.com/i915` by derivation, not by hope:** `scan()` calls `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the kernel driver name, and these nodes have `i915` bound. - `--shared-dev-num` default is `1` (`flag.IntVar(..., 1, ...)`); `--health-management` default is `false` and `setupLevelZeroService` returns early unless it is set. - Missing `/dev/dri/by-path` is non-fatal (`newDevicePlugin` warns, sets `bypathFound = false`). Image pin, verified against the registry: - `docker.io/intel/intel-gpu-plugin:0.37.0` exists; manifest digest `sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1` (single amd64 manifest, not an index), fetched from `registry-1.docker.io` on 2026-10-01. `v0.37.0` is the latest upstream release (2026-09-16) and is not a prerelease. - The Docker Hub tag list carries `0.35.0 / 0.36.0 / 0.37.0` — **image tags have no `v` prefix** while GitHub release tags do. Policy scope, read from the policies rather than assumed: - All four app-scoped Kyverno policies (`restrict-host-namespaces`, `require-egress-class-label`, `require-auth-mode-label`, `require-default-deny-baseline`) are gated on `namespaceSelector`/`selector` `matchLabels: homelab.keithjackson.dev/app-namespace: "true"`. The new namespace is labelled `"false"`, so none of them match it. - Measured on prod: `plex` **is** `app-namespace=true`, confirming `privileged: true` is genuinely unavailable there; `kyverno` and `vpn-egress` are the two namespaces carrying the label as `"false"`; the other platform namespaces leave it unset. No `intel*` namespace exists, so no collision. Cluster before-state (read-only, `kubectl` in the pinned toolbox with the prod kubeconfig bind-mounted read-only — never copied into the worktree): - Three nodes `k3s-prod-{1,2,3}`, each `v1.36.4+k3s1`, Debian 13 trixie, `containerd://2.3.4-k3s1.36`. - Every extended resource currently advertised is `devices.kubevirt.io/{kvm,sev,tun,vhost-net}` on all three nodes. **No node advertises `gpu.intel.com/*`** — queried explicitly, returned "NONE". That is the before-state this change is meant to move. - Those `devices.kubevirt.io/*` resources are themselves advertised by a device plugin, which is measured evidence that the device-plugin API and the default kubelet socket path (`/var/lib/kubelet/device-plugins`) work on these nodes. Offline rendering (the main proof available to a PR): - `kubectl kustomize platform/intel-gpu-plugin` renders two objects: `Namespace/intel-gpu-plugin` carrying `app-namespace=false`, and `DaemonSet/intel-gpu-plugin` in namespace `intel-gpu-plugin`, with the image resolved to `docker.io/intel/intel-gpu-plugin:0.37.0`. - `kubectl kustomize clusters/base` renders, and the new Flux Kustomization appears as `intel-gpu-plugin ./platform/intel-gpu-plugin wait=true dependsOn=cilium`. Both overlays (`clusters/prod`, `clusters/dev`) also render and carry it. - Across the whole rendered graph, **no Kustomization lists `intel-gpu-plugin` in `dependsOn`** — so `wait: true` gates nothing. - The vendored DaemonSet is a **formatting-only** transform of upstream, proven rather than asserted: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file `diff` clean, so passing it through kustomize changed no content. The committed body's sha256 matches the one recorded in its own provenance header (`80ebd070…`), so the file has not drifted from what was generated. Gate (pinned toolbox, from this worktree, with the `ENGINE` shim bind-mounting the main checkout's gitdir and `DEV_KUBECONFIG=/dev/null`): see the PR description for the per-stage table. **`rc 3` (UNDETERMINED), with exactly the three expected skips** — `spec-immutability` (sibling spec repo is not beside a worktree), `gitea-settings` (no CI token), `conformance` (no cluster in CI). Every other stage passed. **rc 3 is not green** and is not reported as such. ## What I did NOT verify **Nothing about this component's runtime behaviour. It has never run.** - **The plugin has never executed.** No pod has been scheduled, no image has been pulled, no scan has happened. It cannot run until the operator merges this and Flux reconciles — `kubectl apply` and `flux reconcile` are operator actions (CLAUDE.md §5), so this is expected, not an oversight. - **The resource has never been advertised.** `gpu.intel.com/i915` does not exist on any node and I have not seen it exist. The claim that it *will* be `i915` rather than `xe` is derived from source plus the established fact that `i915` is bound — it is a derivation, not an observation. - **The DaemonSet has never been admitted.** That the Kyverno policies do not match this namespace is read off the policy manifests; no admission request has actually been evaluated. Nor has the `kubectl kustomize` output ever been server-side dry-run against the API — rendering is not admission. - **The `open()` test has NOT been re-run.** The `EPERM` is exactly as it was. Nothing in this PR touches Plex, so Plex is still software-transcoding and will continue to until the later renderer PR lands. The decisive evidence is an `open()` as **uid 1000 through `s6-setuidgid plex`**; passing as uid 0 is weaker and must not be accepted as proof. - **A green gate proves the manifests render, nothing more.** And this gate was not green: it was `rc 3`. Note also that the Gitea `gate` job runs `./ci/gate.sh || [ "$?" -eq 3 ]` (`.gitea/workflows/gate.yml:138`), so a green tick on this PR cannot distinguish rc 0 from rc 3, and `vars.CLUSTER_AVAILABLE` is unset so no L2/L3 conformance runs in CI. - **`seLinuxOptions: type: container_device_plugin_t`** is carried verbatim from upstream. These nodes are Debian 13 and I did not measure whether SELinux is enabled or whether that type exists there. Expected to be inert (runc skips labelling when SELinux is disabled), but unverified — if the pods fail to start with an SELinux error, that line is the first suspect. - **Renovate has not been run** against this pin. The inline-marker idiom is copied from `platform/kubevirt` / `platform/external-snapshotter`, but whether Renovate normalises the `v`-prefix mismatch (release `v0.37.0` vs image `0.37.0`) for this dep is unknown. If it proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in `renovate.json`. - **Resource limits unexamined under load.** The upstream 115Mi/100m limits are taken as-is; I have no measurement of this plugin's footprint on these nodes. Worth a glance given `keycloak-operator` is OOMing at 91% of its limit elsewhere in this cluster. - **`clusters/dev` renders it, and there is no dev cluster** (the operator confirmed the `k3s-dev-*` nodes are gone). That is inert rather than wrong — the unified graph (R-ENV-02) has both overlays render the same base — but it is not a thing I could test. - **Branch protection / required-checks state not re-measured** in this session. CLAUDE.md §2b says re-measure before relying on it; I did not, so treat the approval/gating claims in §2 as unconfirmed-by-me.