home-network-k3s-impl/platform/intel-gpu-plugin/README.md

platform/intel-gpu-plugin

Intel Device Plugin for Kubernetes (GPU) — the standalone GPU plugin DaemonSet from intel/intel-device-plugins-for-kubernetes, pinned at v0.37.0. It makes every node advertise gpu.intel.com/i915 so a workload can request a GPU instead of hoping a hostPath is enough.

• Serves: hardware transcoding for Plex (hardware.gpu: intel-quicksync), and any later consumer of an Intel iGPU.
• Tier: its own Flux Kustomization (clusters/base/intel-gpu-plugin.yaml, dependsOn: cilium, wait: true) — not a line in platform/kustomization.yaml. See Placement below.
• Namespace: intel-gpu-plugin — a PLATFORM namespace, labelled homelab.keithjackson.dev/app-namespace: "false", so the app-scoped Kyverno policies (default-deny generation, auth-mode, egress-class, host-namespaces) do not apply.
• No RBAC, no ServiceAccount, no CRDs, no webhook, no API-server access. A device plugin talks to the local kubelet over a unix socket.

What this component owns

File	Purpose
namespace.yaml	intel-gpu-plugin ns, explicitly app-namespace=false
intel-gpu-plugin.yaml	the DaemonSet, vendored (machine-generated) from upstream v0.37.0
kustomization.yaml	namespace scoping + the pinned image (images: transformer)

The problem this fixes

Plex was given a /dev/dri hostPath (merged on main @ cdf3e14). The device is visible in the container and cannot be opened. Measured in the running pod on k3s-prod-2:

uid 0,    O_RDWR -> EPERM "Operation not permitted"
uid 1000, O_RDWR -> EPERM "Operation not permitted"

EPERM from the device cgroup, not EACCES from file permissions — proven three ways:

1. root on the host opens the same device with rc=0;
2. the node is unified cgroup2fs;
3. the container's runc config.json reads linux.resources.devices: [{'allow': False, 'access': 'rwm'}] and linux.devices: None — one deny-all rule, no allow entry for major 226, and the runtime was never asked to grant a device.

A hostPath is a filesystem operation. It makes the device node appear and adds no cgroup allow rule, because Kubernetes cannot infer one from a volume.

A device plugin is the mechanism that closes exactly this gap. Verified in the v0.37.0 source (cmd/gpu_plugin/gpu_plugin.go, createDeviceSpecsFromDrmFiles), the Allocate response carries per DRM node:

specs = append(specs, pluginapi.DeviceSpec{
    HostPath: devPath, ContainerPath: devPath, Permissions: "rw",
})

and it is that DeviceSpec which makes kubelet populate linux.devices and the matching cgroup allow rule. It is the only non-privileged way to get there. privileged: true would also work and is not available: platform/kyverno/policies/restrict-host-namespaces.yaml blocks it under Enforce in app namespaces, and plex is labelled app-namespace=true (measured 2026-10-01). Correctly so.

Placement — why its own Kustomization

The cheap option was one line in the shared platform tier. Neither that file nor a new platform/<name>/ directory is a sensitive path in infra/gitea/approval-policy.json, so it would have needed no human reviewer on the policy's terms. It is still the wrong place, and the repo already argued the case when Cilium was split out (read the comment in platform/kustomization.yaml):

• platform is wait: true over eleven-plus components, so its Ready condition cannot distinguish "the GPU plugin is wedged" from "longhorn is slow".
• The only emergency valve would be flux suspend kustomization platform — freezing storage, TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device plugin.
• The tier's 10m/5m interval/timeout are called out there as wrong for a 3-node DaemonSet rollout. This is a 3-node DaemonSet rollout.

The consequence is worse than average here because of what sits downstream: platform gates platform-config, which gates apps-wave-0 (wait: true), which gates apps-wave-1. A plugin that never went healthy inside that tier would stall every app on "dependency not ready" — so a component added to fix a silent no-op (software transcoding) would have acquired the ability to cause a real outage.

The usual objection — that clusters/** needs a human approval — costs nothing: Gitea enforces required_approvals: 1 with block_admin_merge_override: true on every PR here regardless of path, and authors cannot self-approve. The approval was required either way.

wait: true is a health signal, not a gate

This departs from the headlamp/crafty leaf template, deliberately. Those run wait: false because a leaf UI can be legitimately not-Ready for a long time during a cold start while it waits on Keycloak/DNS/TLS, making its health useless as a signal. This DaemonSet has no such dependency — it reads sysfs and opens a socket, so on healthy nodes it is Ready in seconds and not-Ready genuinely means broken. wait: true makes flux get kustomization intel-gpu-plugin a one-line answer to "did it come up on all three nodes", which is precisely the question that must be answered before the consumer PR may merge.

Nothing in the graph dependsOn this, so wait: true gates nothing. Think hard before adding dependsOn: [intel-gpu-plugin] anywhere — for the app waves that would hand a device plugin the power to stall every app, which is the thing this placement exists to prevent.

Deliberate non-choices

Standalone DaemonSet, not the operator. The operator adds a CRD (GpuDevicePlugin), a controller-manager Deployment, cert-manager-backed admission webhooks and a second namespace — three more things that can be unhealthy — to buy declarative management of a DaemonSet that is already declared in git. For three identical nodes and one consumer that is strictly more blast radius for no gain.

It lists devices; it never opens them — worth being certain about, since opening is exactly what fails today. Verified in v0.37.0 source: the scan path uses only os.ReadDir, os.ReadFile, os.Stat and os.Readlink over /sys/class/drm and /dev/dri. There is no os.Open, no OpenFile, no ioctl and no raw syscall against a DRM node anywhere in cmd/gpu_plugin/gpu_plugin.go. Listing and stat() demonstrably work today — that is how the Plex probe found the device before failing to open it — so the plugin needs no cgroup grant of its own and no privilege. Accordingly /dev/dri and /sys/class/drm are mounted readOnly.

• --health-management stays OFF (upstream default). It is the one flag that would need to open a device: it starts a Level-Zero sidecar (setupLevelZeroService returns early unless the flag is set). Enabling it would reintroduce the exact EPERM this component exists to fix.
• /dev/dri/by-path being absent is non-fatal — newDevicePlugin warns and sets bypathFound = false.

--shared-dev-num left at the upstream default of 1 (i.e. not set). The flag multiplies each card into N schedulable device IDs, letting N containers share one GPU. Not raised, because:

• Plex is the only consumer today and wants one device;
• the count is a lie the scheduler believes — there is no GPU-side partitioning or quota behind it, so N containers get the same renderD128 and contend for the same two media engines (vcs0/vecs0). Raising it converts "Unschedulable", which is loud and obvious, into "transcodes are mysteriously slow", which is neither;
• it is a one-line, zero-migration change on the day a second consumer exists — and then the right number will be known rather than guessed.

No NFD overlay. All three nodes are identical (HP EliteDesk 800 G5 Mini, i7-9700T, UHD Graphics 630, PCI 8086:3e98, i915 bound), so a nodeSelector on a GPU feature label would gate on something true everywhere. If a GPU-less node ever joins, add upstream's nfd overlay then rather than relying on the plugin's behaviour on such a node.

No supplementalGroups, anywhere. The pms-docker image's own cont-init hook stats each /dev/dri char device and adds the plex user to its group — measured live as Groups: 44 100 992 1000 on the running Plex process, where GID 992 is the host render group. That mechanism is image-side and works identically on a plugin-injected device.

Resource name

gpu.intel.com/i915. Not a guess: scan() calls devTree.AddDevice(devProps.driver(), ...), so the suffix is the kernel driver name. These nodes have i915 bound (not xe), so the resource is i915. A future kernel moving CoffeeLake to xe would silently rename it to gpu.intel.com/xe and a consumer's request would stop matching — worth knowing before that kernel arrives.

Versioning

Pinned to v0.37.0 via the images: transformer in kustomization.yaml; the DaemonSet itself is vendored as kubectl kustomize output of the upstream base (see its header for the exact re-vendor command and both sha256s). It is generated rather than a byte copy because upstream mixes YAML sequence-indentation styles within the one document and ci/yamllint.yaml sets indent-sequences: consistent — a verbatim copy is a hard lint error that blocks the gate (measured: it did). kubectl kustomize normalises the formatting, and the result was proven to be a formatting-only change: yq -P 'sort_keys(..)' of the generated file and of the raw upstream file diff clean. This is also the re-vendor idiom platform/external-snapshotter documents.

Vendored rather than referenced as a remote kustomize base for the same reason as platform/external-snapshotter: a remote base is git fetched on every reconcile, those fetches intermittently exceeded kustomize-controller's 27s timeout, and transient GitHub slowness then rendered as "kustomize build failed". A local file makes the build offline and deterministic — which also means the CI gate's kustomize build proves the manifests rather than proving GitHub was up.

Renovate tracks it through the inline-marker idiom (# renovate: datasource=github-releases depName=... immediately above the version literal), because renovate.json sets "docker": {"enabled": false} and "kubernetes": {"enabled": false} — an image reference in a plain manifest is tracked by nothing. Caveat: upstream release tags carry a v (v0.37.0) while image tags do not (0.37.0). Whether Renovate normalises that for this dep is unverified; if it ever proposes newTag: v0.38.0, the fix is an extractVersion packageRule in renovate.json, not a hand-edit.

Ordering — do not reorder the stack

The renderer change (catalog/render.py emitting gpu.intel.com/i915: 1 for hardware.gpu: intel-quicksync instead of the hostPath) must not land until this plugin is merged AND confirmed advertising the resource on all three nodes. If the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is wait: true and apps-wave-1 dependsOn it, so that stalls the whole app tier — turning today's silent no-op into a real outage.

Post-merge verification

Nothing about this component's runtime behaviour can be verified from a PR: it does not exist until the operator merges and Flux reconciles. Afterwards:

D=./infra/devcontainer/dev.sh

# 1. Flux applied it, and (wait:true) the DaemonSet went healthy.
$D run bash -c 'flux get kustomization intel-gpu-plugin'
$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin get ds intel-gpu-plugin'
#    want: DESIRED=3 CURRENT=3 READY=3 AVAILABLE=3

# 2. The resource is advertised on ALL THREE nodes. This is the gate on the renderer PR.
$D run bash -c 'kubectl --context k3s-prod get nodes -o json \
  | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
#    want: three lines, each i915=1   (before this change: ABSENT on all three)

# 3. If a node shows ABSENT, read that node's plugin logs.
$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin logs -l app=intel-gpu-plugin --tail=50 --prefix'

The decisive test

Resource-advertised is necessary but not sufficient — it proves the plugin registered, not that the cgroup allow rule reaches Plex. The decisive test is re-running the open() probe as uid 1000 through s6-setuidgid plex inside the Plex pod, after the renderer PR has switched Plex to request the resource:

$D run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
  s6-setuidgid plex python3 -c "import os; os.close(os.open(\"/dev/dri/renderD128\", os.O_RDWR)); print(\"rc=0 OK\")"'

Passing as uid 0 is weaker evidence and must not be accepted as proof — Plex transcodes as uid 1000, and the two differ in group membership, which is the half the image's cont-init hook supplies.