home-network-k3s-impl/decisions/2026-10-01-intel-gpu-device-plugin.md

date, branch, title

date	branch	title
2026-10-01	feat/intel-gpu-device-plugin	Intel GPU device plugin, in its own Kustomization — the resource exists; nothing has run yet

What changed

Deployed the Intel Device Plugin for Kubernetes (GPU), standalone DaemonSet, pinned at v0.37.0, so the prod nodes advertise gpu.intel.com/i915. This is the measured fix for the Plex hardware-transcoding blocker: the /dev/dri hostPath merged on main @ cdf3e14 makes the device node visible and not openable (EPERM from the device cgroup at both uid 0 and uid 1000; the container's runc config.json carries one deny-all rule and linux.devices: None). A hostPath is a filesystem operation and adds no cgroup allow rule. A device plugin's Allocate response is what makes kubelet populate linux.devices and the matching cgroup allow rule, and it is the only non-privileged way to get there — privileged: true is blocked by Kyverno restrict-host-namespaces under Enforce in app namespaces, and plex is labelled app-namespace=true.

New files:

• platform/intel-gpu-plugin/intel-gpu-plugin.yaml — the DaemonSet, vendored as kubectl kustomize output of upstream deployments/gpu_plugin/base at v0.37.0, under a provenance header carrying the exact re-vendor command and both sha256s (generated body 80ebd070…, raw upstream source c8773918…). Generated rather than byte-copied because upstream mixes YAML sequence-indentation styles within the one document and ci/yamllint.yaml sets indent-sequences: consistent — a verbatim copy is a hard lint error that blocks the gate. That is not a guess: the first gate run of this branch returned rc 1 BLOCKED on exactly that error, and regenerating through kustomize is the fix (and is also the re-vendor idiom platform/external-snapshotter documents). Vendored at all, rather than referenced as a remote kustomize base, for the other reason external-snapshotter records: a remote base is git fetched on every reconcile and those fetches have flaked a whole tier.
• platform/intel-gpu-plugin/namespace.yaml — a PLATFORM namespace, explicitly homelab.keithjackson.dev/app-namespace: "false" (following vpn-egress/kyverno).
• platform/intel-gpu-plugin/kustomization.yaml — namespace scoping plus the image pin.
• platform/intel-gpu-plugin/README.md — the measurement, the placement argument, the non-choices, and the post-merge verification sequence.
• clusters/base/intel-gpu-plugin.yaml — its own Flux Kustomization: dependsOn: [cilium], wait: true, interval: 10m, timeout: 10m, prune: true.
• One wired line (plus its rationale comment) in clusters/base/kustomization.yaml.

Placement: its own Kustomization, not a line in the shared platform tier. The shared tier was cheaper and needed no human reviewer on approval-policy.json's terms, but platform is wait: true over eleven-plus components and the repo already argued this case when Cilium was split out: its Ready condition cannot distinguish "the plugin is wedged" from "longhorn is slow", the only emergency valve is freezing the entire tier, and the tier's 10m/5m interval/timeout are called out there as wrong for a 3-node DaemonSet rollout — which is exactly what this is. The consequence is worse than average here, because platform gates platform-config gates apps-wave-0 (wait: true) gates apps-wave-1: a plugin that never went healthy inside that tier would stall every app, so a component added to fix a silent no-op would have gained the ability to cause a real outage. The approval cost of touching clusters/** is nil, because required_approvals: 1 with block_admin_merge_override: true applies to every PR here regardless of path and authors cannot self-approve.

dependsOn: [cilium] is the minimum and the earliest: the plugin never uses the network and never talks to the API server (it talks to the local kubelet over a unix socket), so a working CNI is its only real prerequisite, and cilium already depends on flux-system, which applies the cluster-vars ConfigMap this Kustomization substitutes from. Depending on cilium rather than platform also means it reconciles in parallel with the platform tier rather than behind all eleven of its components, which shrinks the cold-start window in which a node has not yet advertised the resource.

wait: true is a deliberate departure from the headlamp/crafty leaf template. Those use wait: false because a leaf UI can be legitimately not-Ready for a long time during a cold start, making its health a useless signal. This DaemonSet reads sysfs and opens a socket, so not-Ready genuinely means broken — and since nothing in the graph dependsOn it (verified against the rendered graph), wait: true gates nothing while turning flux get kustomization intel-gpu-plugin into a one-line answer to the question that must be answered before the consumer PR may merge.

Deliberate non-choices, each argued in the README: standalone DaemonSet rather than the operator (no CRD, no controller-manager, no cert-manager webhook, no second namespace to keep healthy); --shared-dev-num left at the upstream default of 1 (a higher count is a lie the scheduler believes — there is no GPU-side partitioning behind it, so it converts a loud "Unschedulable" into a quiet "transcodes are slow"); --health-management left OFF, because it is the one flag that would open a device and would reintroduce the exact EPERM this fixes; no NFD overlay (all three nodes are identical); no supplementalGroups anywhere, because the pms-docker cont-init hook already adds the plex user to each device's group and that works identically on a plugin-injected device.

Scope held deliberately: catalog/render.py, catalog/plex.yaml and apps/wave-0/plex.yaml are untouched. Switching Plex from the hostPath to a gpu.intel.com/i915: 1 request is a strictly later PR, and it must not land until this plugin is merged AND confirmed advertising the resource on all three nodes — a request for a resource no node advertises makes the Plex pod Unschedulable, and apps-wave-0 is wait: true with apps-wave-1 depending on it, so that would stall the whole app tier and turn today's silent no-op into a real outage.

What I verified

Mechanism, read from the v0.37.0 source rather than assumed:

• The plugin only lists devices; it never opens one. The scan path uses os.ReadDir, os.ReadFile, os.Stat and os.Readlink over /sys/class/drm and /dev/dri. grep for os.Open|OpenFile|ioctl|syscall|unix\. across cmd/gpu_plugin/gpu_plugin.go (908 lines) returns nothing. Listing and stat() are what demonstrably work today, so the plugin needs no cgroup grant and no extra privilege — it does not need more than the upstream manifest gives it.
• The fix mechanism is real: createDeviceSpecsFromDrmFiles emits pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"} per DRM node, which is the thing kubelet turns into linux.devices plus a cgroup allow rule.
• The resource name is gpu.intel.com/i915 by derivation, not by hope: scan() calls devTree.AddDevice(devProps.driver(), ...), so the suffix is the kernel driver name, and these nodes have i915 bound.
• --shared-dev-num default is 1 (flag.IntVar(..., 1, ...)); --health-management default is false and setupLevelZeroService returns early unless it is set.
• Missing /dev/dri/by-path is non-fatal (newDevicePlugin warns, sets bypathFound = false).

Image pin, verified against the registry:

• docker.io/intel/intel-gpu-plugin:0.37.0 exists; manifest digest sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1 (single amd64 manifest, not an index), fetched from registry-1.docker.io on 2026-10-01. v0.37.0 is the latest upstream release (2026-09-16) and is not a prerelease.
• The Docker Hub tag list carries 0.35.0 / 0.36.0 / 0.37.0 — image tags have no v prefix while GitHub release tags do.

Policy scope, read from the policies rather than assumed:

• All four app-scoped Kyverno policies (restrict-host-namespaces, require-egress-class-label, require-auth-mode-label, require-default-deny-baseline) are gated on namespaceSelector/selector matchLabels: homelab.keithjackson.dev/app-namespace: "true". The new namespace is labelled "false", so none of them match it.
• Measured on prod: plex is app-namespace=true, confirming privileged: true is genuinely unavailable there; kyverno and vpn-egress are the two namespaces carrying the label as "false"; the other platform namespaces leave it unset. No intel* namespace exists, so no collision.

Cluster before-state (read-only, kubectl in the pinned toolbox with the prod kubeconfig bind-mounted read-only — never copied into the worktree):

• Three nodes k3s-prod-{1,2,3}, each v1.36.4+k3s1, Debian 13 trixie, containerd://2.3.4-k3s1.36.
• Every extended resource currently advertised is devices.kubevirt.io/{kvm,sev,tun,vhost-net} on all three nodes. No node advertises gpu.intel.com/* — queried explicitly, returned "NONE". That is the before-state this change is meant to move.
• Those devices.kubevirt.io/* resources are themselves advertised by a device plugin, which is measured evidence that the device-plugin API and the default kubelet socket path (/var/lib/kubelet/device-plugins) work on these nodes.

Offline rendering (the main proof available to a PR):

• kubectl kustomize platform/intel-gpu-plugin renders two objects: Namespace/intel-gpu-plugin carrying app-namespace=false, and DaemonSet/intel-gpu-plugin in namespace intel-gpu-plugin, with the image resolved to docker.io/intel/intel-gpu-plugin:0.37.0.
• kubectl kustomize clusters/base renders, and the new Flux Kustomization appears as intel-gpu-plugin ./platform/intel-gpu-plugin wait=true dependsOn=cilium. Both overlays (clusters/prod, clusters/dev) also render and carry it.
• Across the whole rendered graph, no Kustomization lists intel-gpu-plugin in dependsOn — so wait: true gates nothing.
• The vendored DaemonSet is a formatting-only transform of upstream, proven rather than asserted: yq -P 'sort_keys(..)' of the generated file and of the raw upstream file diff clean, so passing it through kustomize changed no content. The committed body's sha256 matches the one recorded in its own provenance header (80ebd070…), so the file has not drifted from what was generated.

Gate (pinned toolbox, from this worktree, with the ENGINE shim bind-mounting the main checkout's gitdir and DEV_KUBECONFIG=/dev/null): see the PR description for the per-stage table. rc 3 (UNDETERMINED), with exactly the three expected skips — spec-immutability (sibling spec repo is not beside a worktree), gitea-settings (no CI token), conformance (no cluster in CI). Every other stage passed. rc 3 is not green and is not reported as such.

What I did NOT verify

Nothing about this component's runtime behaviour. It has never run.

• The plugin has never executed. No pod has been scheduled, no image has been pulled, no scan has happened. It cannot run until the operator merges this and Flux reconciles — kubectl apply and flux reconcile are operator actions (CLAUDE.md §5), so this is expected, not an oversight.
• The resource has never been advertised. gpu.intel.com/i915 does not exist on any node and I have not seen it exist. The claim that it will be i915 rather than xe is derived from source plus the established fact that i915 is bound — it is a derivation, not an observation.
• The DaemonSet has never been admitted. That the Kyverno policies do not match this namespace is read off the policy manifests; no admission request has actually been evaluated. Nor has the kubectl kustomize output ever been server-side dry-run against the API — rendering is not admission.
• The open() test has NOT been re-run. The EPERM is exactly as it was. Nothing in this PR touches Plex, so Plex is still software-transcoding and will continue to until the later renderer PR lands. The decisive evidence is an open() as uid 1000 through s6-setuidgid plex; passing as uid 0 is weaker and must not be accepted as proof.
• A green gate proves the manifests render, nothing more. And this gate was not green: it was rc 3. Note also that the Gitea gate job runs ./ci/gate.sh || [ "$?" -eq 3 ] (.gitea/workflows/gate.yml:138), so a green tick on this PR cannot distinguish rc 0 from rc 3, and vars.CLUSTER_AVAILABLE is unset so no L2/L3 conformance runs in CI.
• seLinuxOptions: type: container_device_plugin_t is carried verbatim from upstream. These nodes are Debian 13 and I did not measure whether SELinux is enabled or whether that type exists there. Expected to be inert (runc skips labelling when SELinux is disabled), but unverified — if the pods fail to start with an SELinux error, that line is the first suspect.
• Renovate has not been run against this pin. The inline-marker idiom is copied from platform/kubevirt / platform/external-snapshotter, but whether Renovate normalises the v-prefix mismatch (release v0.37.0 vs image 0.37.0) for this dep is unknown. If it proposes newTag: v0.38.0, the fix is an extractVersion packageRule in renovate.json.
• Resource limits unexamined under load. The upstream 115Mi/100m limits are taken as-is; I have no measurement of this plugin's footprint on these nodes. Worth a glance given keycloak-operator is OOMing at 91% of its limit elsewhere in this cluster.
• clusters/dev renders it, and there is no dev cluster (the operator confirmed the k3s-dev-* nodes are gone). That is inert rather than wrong — the unified graph (R-ENV-02) has both overlays render the same base — but it is not a thing I could test.
• Branch protection / required-checks state not re-measured in this session. CLAUDE.md §2b says re-measure before relying on it; I did not, so treat the approval/gating claims in §2 as unconfirmed-by-me.