Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(gpu): Intel GPU device plugin v0.37.0 — nodes advertise gpu.intel.com/i915 #36

Merged
jakman85 merged 1 commits from feat/intel-gpu-device-plugin into main 2026-10-02 03:53:50 +00:00
Conversation 1 Commits 1 Files Changed 7
+786
gitea-admin commented 2026-10-02 03:29:18 +00:00
Owner
Copy link
Copy source

Deploys the Intel Device Plugin for Kubernetes (GPU), standalone DaemonSet, pinned at v0.37.0, so the three prod nodes advertise gpu.intel.com/i915.

This PR's only job is to make the resource exist. No consumer is switched over here — see Ordering below, which is a real constraint on the next PR and not a formality.

Why — measured, not theoretical

The /dev/dri hostPath merged on main @ cdf3e14 makes the device node visible and not openable. Measured in the running Plex pod on k3s-prod-2:

uid 0,    O_RDWR -> EPERM "Operation not permitted"
uid 1000, O_RDWR -> EPERM "Operation not permitted"

EPERM from the device cgroup, not EACCES from file permissions — proven three ways: root on the host opens the same device rc=0; the node is unified cgroup2fs; and the container's runc config.json reads

linux.resources.devices: [{'allow': False, 'access': 'rwm'}]
linux.devices: None

One deny-all rule, no allow entry for major 226, and the runtime was never asked to grant a device. A hostPath is a filesystem operation: it makes the device node appear and adds no cgroup allow rule, because Kubernetes cannot infer one from a volume.

A device plugin is the mechanism that closes exactly that gap. Verified by reading the v0.37.0 source (cmd/gpu_plugin/gpu_plugin.go, createDeviceSpecsFromDrmFiles): the Allocate response carries pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"} per DRM node, and it is that DeviceSpec which makes kubelet populate linux.devices and the matching cgroup allow rule. It is the only non-privileged way to get there — privileged: true is blocked by Kyverno restrict-host-namespaces under Enforce in app namespaces, and plex is labelled app-namespace=true (measured). Correctly so.

Placement: its own Flux Kustomization, not a line in platform/

The cheap option was one line in the shared platform tier; neither that file nor a new platform/<name>/ directory is a sensitive path in approval-policy.json, so it would have needed no human reviewer on the policy's terms. It is still the wrong place, and the repo already argued the case when Cilium was split out:

  • platform is wait: true over eleven-plus components, so its Ready condition cannot distinguish "the plugin is wedged" from "longhorn is slow".
  • The only emergency valve would be flux suspend kustomization platform — freezing storage, TLS, DNS, ingress, monitoring, Keycloak and KubeVirt to stop touching a device plugin.
  • The tier's 10m/5m interval/timeout are called out there as wrong for a 3-node DaemonSet rollout. This is a 3-node DaemonSet rollout.

The consequence is worse than average here because of what sits downstream: platform → platform-config → apps-wave-0 (wait: true) → apps-wave-1. A plugin that never went healthy inside that tier would stall every app on "dependency not ready" — so a component added to fix a silent no-op would have acquired the ability to cause a real outage.

The approval cost of touching clusters/** is nil: required_approvals: 1 with block_admin_merge_override: true applies to every PR here regardless of path, and authors cannot self-approve. This needed your approval either way.

dependsOn: [cilium] is the minimum and the earliest — the plugin never uses the network and never talks to the API server (it talks to the local kubelet over a unix socket), so a working CNI is its only real prerequisite, and cilium already depends on flux-system (which applies the cluster-vars ConfigMap this substitutes from). Depending on cilium rather than platform also means it reconciles in parallel with the platform tier instead of behind all eleven of its components, shrinking the cold-start window in which a node has not yet advertised the resource.

wait: true is a health signal, not a gate. A deliberate departure from the headlamp/crafty leaf template: those use wait: false because a leaf UI can be legitimately not-Ready for a long time during a cold start, making its health useless as a signal. This DaemonSet reads sysfs and opens a socket, so not-Ready genuinely means broken. Nothing in the rendered graph dependsOn it (verified), so wait: true gates nothing, while making flux get kustomization intel-gpu-plugin a one-line answer to the question that must be answered before the consumer PR may merge.

Deliberate non-choices

  • Standalone DaemonSet, not the operator. The operator adds a CRD, a controller-manager Deployment, cert-manager-backed webhooks and a second namespace — three more things that can be unhealthy — to buy declarative management of a DaemonSet already declared in git.
  • --shared-dev-num left at the upstream default of 1. A higher count is a lie the scheduler believes: there is no GPU-side partitioning behind it, so N containers get the same renderD128 and contend for the same two media engines. Raising it converts a loud "Unschedulable" into a quiet "transcodes are mysteriously slow". One-line change the day a second consumer exists, when the right number will be known.
  • --health-management left OFF. It is the one flag that would open a device (it starts a Level-Zero sidecar) and would reintroduce the exact EPERM this fixes.
  • No NFD overlay — all three nodes are identical, so a GPU feature-label selector would gate on something true everywhere.
  • No supplementalGroups, anywhere. The pms-docker cont-init hook already stats each /dev/dri device and adds plex to its group (measured live: Groups: 44 100 992 1000, GID 992 = host render). That works identically on a plugin-injected device.

Pinned image

docker.io/intel/intel-gpu-plugin:0.37.0 — verified present on Docker Hub 2026-10-01, manifest digest sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1 (single amd64 manifest). v0.37.0 is the latest upstream release (2026-09-16), not a prerelease. Pinned by tag via the images: transformer, matching how this repo pins upstream platform components (kubevirt, external-snapshotter); digest pinning here is reserved for the conformance runner, where it is a security boundary.

Renovate tracks it through the inline-marker idiom, because renovate.json sets "docker": {"enabled": false} and "kubernetes": {"enabled": false} — an image in a plain manifest is tracked by nothing. Caveat, flagged rather than hidden: upstream release tags carry a v (v0.37.0), image tags do not (0.37.0). Whether Renovate normalises that for this dep is unverified; if it proposes newTag: v0.38.0, the fix is an extractVersion packageRule in renovate.json.

The DaemonSet is vendored as kubectl kustomize output of upstream's gpu_plugin/base rather than left as a remote base (a remote base is git fetched on every reconcile and those fetches have flaked a whole tier — see external-snapshotter). Generated rather than byte-copied because upstream mixes YAML sequence-indentation styles within the one document and ci/yamllint.yaml sets indent-sequences: consistent: a verbatim copy is a hard lint error. That is measured, not guessed — the first gate run of this branch returned rc 1 BLOCKED on exactly that, and regenerating through kustomize is the fix. The transform is proven formatting-only (yq -P 'sort_keys(..)' of both files diffs clean).

Ordering — DO NOT REORDER THE STACK

catalog/render.py, catalog/plex.yaml and apps/wave-0/plex.yaml are untouched.

The renderer change that makes hardware.gpu: intel-quicksync emit gpu.intel.com/i915: 1 instead of the hostPath must not land until this plugin is merged AND confirmed advertising the resource on all three nodes. If the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is wait: true and apps-wave-1 dependsOn it, so that would stall the whole app tier — turning today's silent no-op into a real outage.

Verification

Before-state, measured on prod (read-only): three nodes k3s-prod-{1,2,3}, each v1.36.4+k3s1, Debian 13 trixie, containerd://2.3.4-k3s1.36. Every extended resource currently advertised is devices.kubevirt.io/{kvm,sev,tun,vhost-net}. Queried explicitly for gpu.intel.com/* → "NONE — no node advertises gpu.intel.com/*". (Those devices.kubevirt.io/* resources are themselves advertised by a device plugin, which is measured evidence that the device-plugin API and the default kubelet socket path work on these nodes.)

It lists devices; it never opens one — the thing worth being sure about, since opening is what fails today. The v0.37.0 scan path uses only os.ReadDir / os.ReadFile / os.Stat / os.Readlink over /sys/class/drm and /dev/dri; grep -nE "os\.Open|OpenFile|ioctl|syscall|unix\." across its 908-line source returns nothing. So it needs no cgroup grant of its own and no privilege, and /dev/dri + /sys/class/drm are mounted readOnly.

Resource name by derivation, not hope: scan() calls devTree.AddDevice(devProps.driver(), ...), so the suffix is the kernel driver name; these nodes have i915 bound (not xe).

Policy scope read off the policies: all four app-scoped Kyverno policies (restrict-host-namespaces, require-egress-class-label, require-auth-mode-label, require-default-deny-baseline) match on app-namespace: "true". The new namespace is labelled "false", so none apply. No intel* namespace exists.

Offline rendering: kubectl kustomize platform/intel-gpu-plugin renders Namespace/intel-gpu-plugin (app-namespace=false) and DaemonSet/intel-gpu-plugin, image resolved to docker.io/intel/intel-gpu-plugin:0.37.0. clusters/base renders with intel-gpu-plugin → ./platform/intel-gpu-plugin wait=true dependsOn=cilium; both clusters/prod and clusters/dev carry it; no Kustomization anywhere lists intel-gpu-plugin in dependsOn.

Gate — rc 3 (UNDETERMINED). Not green.

Run from the worktree in the pinned toolbox (DEV_KUBECONFIG=/dev/null plus an ENGINE shim bind-mounting the main checkout's gitdir):

stage result
spec-immutability SKIPPED — spec repo not beside a worktree (expected, tolerated)
decisions-log green
lint green
l0 green (21 cases, 0 failures)
render-golden green (all 10 onboarded service goldens reproduce)
l1 green (3 whole-repo checks)
approval-policy green (event='none' locally — N/A off a PR event)
gitea-settings SKIPPED — SETTINGS_API_TOKEN unset
conformance SKIPPED — CLUSTER_AVAILABLE != true, no L2/L3

Exactly the three expected skips, zero failures, zero lint errors. rc 3 is not a pass and is not reported as one. Note also that the Gitea gate job runs ./ci/gate.sh || [ "$?" -eq 3 ] (.gitea/workflows/gate.yml:138), so a green tick on this PR cannot distinguish rc 0 from rc 3.

What is NOT verified

Nothing about this component's runtime behaviour. It has never run. No pod has been scheduled, no image pulled, no scan performed, no admission request evaluated, and gpu.intel.com/i915 has never existed on any node. It cannot run until you merge and Flux reconciles (kubectl apply / flux reconcile are operator actions, CLAUDE.md §5). The open() test has not been re-run — the EPERM is exactly as it was, and Plex is still software-transcoding.

Also unverified: seLinuxOptions: type: container_device_plugin_t is carried from upstream and I did not measure whether SELinux is enabled on these Debian nodes (expected inert; first suspect if the pods fail to start). Renovate has not been run against the pin. The upstream memory limit (115Mi) is taken as-is with no measurement of this plugin's footprint here.

Post-merge verification (for you)

D=./infra/devcontainer/dev.sh

# 1. Flux applied it and (wait:true) the DaemonSet went healthy.
$D run bash -c 'flux get kustomization intel-gpu-plugin'
$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin get ds intel-gpu-plugin'
#    want: DESIRED=3 CURRENT=3 READY=3 AVAILABLE=3

# 2. The resource is advertised on ALL THREE nodes. THIS IS THE GATE ON THE RENDERER PR.
$D run bash -c 'kubectl --context k3s-prod get nodes -o json \
  | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
#    want: three lines, each i915=1   (before this change: ABSENT on all three)

# 3. If a node shows ABSENT, read that node's plugin logs.
$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin logs -l app=intel-gpu-plugin --tail=50 --prefix'

The decisive test comes later and is not step 2. Resource-advertised proves the plugin registered, not that the cgroup allow rule reaches Plex. After the renderer PR switches Plex to request the resource:

$D run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
  s6-setuidgid plex python3 -c "import os; os.close(os.open(\"/dev/dri/renderD128\", os.O_RDWR)); print(\"rc=0 OK\")"'

Passing as uid 0 is weaker evidence and must not be accepted as proof — Plex transcodes as uid 1000, and the two differ in group membership, which is the half the image's cont-init hook supplies.

🤖 Generated with Claude Code

Deploys the **Intel Device Plugin for Kubernetes (GPU)**, standalone DaemonSet, pinned at **v0.37.0**, so the three prod nodes advertise **`gpu.intel.com/i915`**. This PR's only job is to **make the resource exist**. No consumer is switched over here — see *Ordering* below, which is a real constraint on the next PR and not a formality. ## Why — measured, not theoretical The `/dev/dri` hostPath merged on `main` @ `cdf3e14` makes the device node *visible* and **not openable**. Measured in the running Plex pod on `k3s-prod-2`: ``` uid 0, O_RDWR -> EPERM "Operation not permitted" uid 1000, O_RDWR -> EPERM "Operation not permitted" ``` `EPERM` from the **device cgroup**, not `EACCES` from file permissions — proven three ways: root on the **host** opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc `config.json` reads ``` linux.resources.devices: [{'allow': False, 'access': 'rwm'}] linux.devices: None ``` One deny-all rule, no allow entry for major 226, and the runtime was never asked to grant a device. A hostPath is a **filesystem** operation: it makes the device node appear and adds no cgroup allow rule, because Kubernetes cannot infer one from a volume. **A device plugin is the mechanism that closes exactly that gap.** Verified by reading the v0.37.0 source (`cmd/gpu_plugin/gpu_plugin.go`, `createDeviceSpecsFromDrmFiles`): the `Allocate` response carries `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, and it is that `DeviceSpec` which makes kubelet populate `linux.devices` **and** the matching cgroup allow rule. It is the only **non-privileged** way to get there — `privileged: true` is blocked by Kyverno `restrict-host-namespaces` under Enforce in app namespaces, and `plex` is labelled `app-namespace=true` (measured). Correctly so. ## Placement: its own Flux Kustomization, not a line in `platform/` The cheap option was one line in the shared `platform` tier; neither that file nor a new `platform/<name>/` directory is a sensitive path in `approval-policy.json`, so it would have needed no human reviewer *on the policy's terms*. It is still the wrong place, and the repo already argued the case when Cilium was split out: - `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot distinguish "the plugin is wedged" from "longhorn is slow". - The only emergency valve would be `flux suspend kustomization platform` — freezing storage, TLS, DNS, ingress, monitoring, Keycloak and KubeVirt to stop touching a device plugin. - The tier's 10m/5m interval/timeout are called out there as **wrong for a 3-node DaemonSet rollout**. This is a 3-node DaemonSet rollout. The consequence is worse than average here because of what sits downstream: `platform` → `platform-config` → `apps-wave-0` (`wait: true`) → `apps-wave-1`. A plugin that never went healthy inside that tier would stall **every app** on "dependency not ready" — so a component added to fix a *silent no-op* would have acquired the ability to cause a *real outage*. The approval cost of touching `clusters/**` is nil: `required_approvals: 1` with `block_admin_merge_override: true` applies to **every** PR here regardless of path, and authors cannot self-approve. This needed your approval either way. **`dependsOn: [cilium]`** is the minimum *and* the earliest — the plugin never uses the network and never talks to the API server (it talks to the local kubelet over a unix socket), so a working CNI is its only real prerequisite, and `cilium` already depends on `flux-system` (which applies the `cluster-vars` ConfigMap this substitutes from). Depending on `cilium` rather than `platform` also means it reconciles *in parallel with* the platform tier instead of behind all eleven of its components, shrinking the cold-start window in which a node has not yet advertised the resource. **`wait: true` is a health signal, not a gate.** A deliberate departure from the `headlamp`/`crafty` leaf template: those use `wait: false` because a leaf UI can be legitimately not-Ready for a long time during a cold start, making its health useless as a signal. This DaemonSet reads sysfs and opens a socket, so not-Ready genuinely means **broken**. Nothing in the rendered graph `dependsOn` it (verified), so `wait: true` gates nothing, while making `flux get kustomization intel-gpu-plugin` a one-line answer to the question that must be answered before the consumer PR may merge. ## Deliberate non-choices - **Standalone DaemonSet, not the operator.** The operator adds a CRD, a controller-manager Deployment, cert-manager-backed webhooks and a second namespace — three more things that can be unhealthy — to buy declarative management of a DaemonSet already declared in git. - **`--shared-dev-num` left at the upstream default of 1.** A higher count is a lie the scheduler believes: there is no GPU-side partitioning behind it, so N containers get the same `renderD128` and contend for the same two media engines. Raising it converts a loud "Unschedulable" into a quiet "transcodes are mysteriously slow". One-line change the day a second consumer exists, when the right number will be *known*. - **`--health-management` left OFF.** It is the one flag that *would* open a device (it starts a Level-Zero sidecar) and would reintroduce the exact `EPERM` this fixes. - **No NFD overlay** — all three nodes are identical, so a GPU feature-label selector would gate on something true everywhere. - **No `supplementalGroups`, anywhere.** The `pms-docker` cont-init hook already stats each `/dev/dri` device and adds `plex` to its group (measured live: `Groups: 44 100 992 1000`, GID 992 = host `render`). That works identically on a plugin-injected device. ## Pinned image `docker.io/intel/intel-gpu-plugin:0.37.0` — verified present on Docker Hub 2026-10-01, manifest digest `sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1` (single amd64 manifest). v0.37.0 is the latest upstream release (2026-09-16), not a prerelease. Pinned by tag via the `images:` transformer, matching how this repo pins upstream platform components (`kubevirt`, `external-snapshotter`); digest pinning here is reserved for the conformance runner, where it is a security boundary. Renovate tracks it through the inline-marker idiom, because `renovate.json` sets `"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}` — an image in a plain manifest is tracked by nothing. **Caveat, flagged rather than hidden:** upstream *release* tags carry a `v` (`v0.37.0`), *image* tags do not (`0.37.0`). Whether Renovate normalises that for this dep is unverified; if it proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in `renovate.json`. The DaemonSet is vendored as `kubectl kustomize` output of upstream's `gpu_plugin/base` rather than left as a remote base (a remote base is `git fetch`ed on *every* reconcile and those fetches have flaked a whole tier — see `external-snapshotter`). Generated rather than byte-copied because upstream mixes YAML sequence-indentation styles within the one document and `ci/yamllint.yaml` sets `indent-sequences: consistent`: a verbatim copy is a hard lint error. That is measured, not guessed — **the first gate run of this branch returned rc 1 BLOCKED on exactly that**, and regenerating through kustomize is the fix. The transform is proven formatting-only (`yq -P 'sort_keys(..)'` of both files diffs clean). ## Ordering — DO NOT REORDER THE STACK `catalog/render.py`, `catalog/plex.yaml` and `apps/wave-0/plex.yaml` are **untouched**. The renderer change that makes `hardware.gpu: intel-quicksync` emit `gpu.intel.com/i915: 1` instead of the hostPath **must not land until this plugin is merged AND confirmed advertising the resource on all three nodes.** If the renderer requests a resource no node advertises, the Plex pod is **Unschedulable**; `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so that would stall the whole app tier — turning today's silent no-op into a real outage. ## Verification **Before-state, measured on prod (read-only):** three nodes `k3s-prod-{1,2,3}`, each `v1.36.4+k3s1`, Debian 13 trixie, `containerd://2.3.4-k3s1.36`. Every extended resource currently advertised is `devices.kubevirt.io/{kvm,sev,tun,vhost-net}`. Queried explicitly for `gpu.intel.com/*` → **"NONE — no node advertises gpu.intel.com/\*"**. (Those `devices.kubevirt.io/*` resources are themselves advertised by a device plugin, which is measured evidence that the device-plugin API and the default kubelet socket path work on these nodes.) **It lists devices; it never opens one** — the thing worth being sure about, since *opening* is what fails today. The v0.37.0 scan path uses only `os.ReadDir` / `os.ReadFile` / `os.Stat` / `os.Readlink` over `/sys/class/drm` and `/dev/dri`; `grep -nE "os\.Open|OpenFile|ioctl|syscall|unix\."` across its 908-line source returns **nothing**. So it needs no cgroup grant of its own and no privilege, and `/dev/dri` + `/sys/class/drm` are mounted `readOnly`. **Resource name by derivation, not hope:** `scan()` calls `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the kernel driver name; these nodes have `i915` bound (not `xe`). **Policy scope read off the policies:** all four app-scoped Kyverno policies (`restrict-host-namespaces`, `require-egress-class-label`, `require-auth-mode-label`, `require-default-deny-baseline`) match on `app-namespace: "true"`. The new namespace is labelled `"false"`, so none apply. No `intel*` namespace exists. **Offline rendering:** `kubectl kustomize platform/intel-gpu-plugin` renders `Namespace/intel-gpu-plugin` (`app-namespace=false`) and `DaemonSet/intel-gpu-plugin`, image resolved to `docker.io/intel/intel-gpu-plugin:0.37.0`. `clusters/base` renders with `intel-gpu-plugin → ./platform/intel-gpu-plugin wait=true dependsOn=cilium`; both `clusters/prod` and `clusters/dev` carry it; no Kustomization anywhere lists `intel-gpu-plugin` in `dependsOn`. ### Gate — `rc 3` (UNDETERMINED). **Not green.** Run from the worktree in the pinned toolbox (`DEV_KUBECONFIG=/dev/null` plus an `ENGINE` shim bind-mounting the main checkout's gitdir): | stage | result | |---|---| | spec-immutability | **SKIPPED** — spec repo not beside a worktree (expected, tolerated) | | decisions-log | green | | lint | green | | l0 | green (21 cases, 0 failures) | | render-golden | green (all 10 onboarded service goldens reproduce) | | l1 | green (3 whole-repo checks) | | approval-policy | green (`event='none'` locally — N/A off a PR event) | | gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset | | conformance | **SKIPPED** — `CLUSTER_AVAILABLE != true`, no L2/L3 | Exactly the three expected skips, zero failures, zero lint errors. `rc 3` is **not** a pass and is not reported as one. Note also that the Gitea `gate` job runs `./ci/gate.sh || [ "$?" -eq 3 ]` (`.gitea/workflows/gate.yml:138`), so a green tick on this PR cannot distinguish rc 0 from rc 3. ### What is NOT verified **Nothing about this component's runtime behaviour. It has never run.** No pod has been scheduled, no image pulled, no scan performed, no admission request evaluated, and `gpu.intel.com/i915` has never existed on any node. It cannot run until you merge and Flux reconciles (`kubectl apply` / `flux reconcile` are operator actions, CLAUDE.md §5). **The `open()` test has not been re-run** — the `EPERM` is exactly as it was, and Plex is still software-transcoding. Also unverified: `seLinuxOptions: type: container_device_plugin_t` is carried from upstream and I did not measure whether SELinux is enabled on these Debian nodes (expected inert; first suspect if the pods fail to start). Renovate has not been run against the pin. The upstream memory limit (115Mi) is taken as-is with no measurement of this plugin's footprint here. ## Post-merge verification (for you) ```bash D=./infra/devcontainer/dev.sh # 1. Flux applied it and (wait:true) the DaemonSet went healthy. $D run bash -c 'flux get kustomization intel-gpu-plugin' $D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin get ds intel-gpu-plugin' # want: DESIRED=3 CURRENT=3 READY=3 AVAILABLE=3 # 2. The resource is advertised on ALL THREE nodes. THIS IS THE GATE ON THE RENDERER PR. $D run bash -c 'kubectl --context k3s-prod get nodes -o json \ | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""' # want: three lines, each i915=1 (before this change: ABSENT on all three) # 3. If a node shows ABSENT, read that node's plugin logs. $D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin logs -l app=intel-gpu-plugin --tail=50 --prefix' ``` **The decisive test comes later and is not step 2.** Resource-advertised proves the plugin registered, not that the cgroup allow rule reaches Plex. After the renderer PR switches Plex to request the resource: ```bash $D run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \ s6-setuidgid plex python3 -c "import os; os.close(os.open(\"/dev/dri/renderD128\", os.O_RDWR)); print(\"rc=0 OK\")"' ``` **Passing as uid 0 is weaker evidence and must not be accepted as proof** — Plex transcodes as uid 1000, and the two differ in group membership, which is the half the image's cont-init hook supplies. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code)
gitea-admin added 1 commit 2026-10-02 03:29:18 +00:00
feat(gpu): Intel GPU device plugin v0.37.0, in its own Flux Kustomization
gate / gate (pull_request) Successful in 1m41s
Details
gate / conformance (pull_request) Has been skipped
Details
4e1b169256 
Makes the three prod nodes advertise `gpu.intel.com/i915`. This is the measured
fix for the Plex hardware-transcoding blocker, and it only makes the resource
EXIST — no consumer is switched over here.

WHY. The `/dev/dri` hostPath merged on main @ cdf3e14 makes the device node
visible and not openable. Measured in the running pod on k3s-prod-2:

    uid 0,    O_RDWR -> EPERM "Operation not permitted"
    uid 1000, O_RDWR -> EPERM "Operation not permitted"

EPERM from the device cgroup, not EACCES from file permissions — root on the
HOST opens the same device rc=0, the node is unified cgroup2fs, and the
container's runc config.json carries one deny-all rule with `linux.devices:
None`. A hostPath is a filesystem operation: it makes the device node appear
and adds no cgroup allow rule, because Kubernetes cannot infer one from a
volume. A device plugin's Allocate response is what makes kubelet populate
`linux.devices` AND the matching cgroup allow rule — verified in the v0.37.0
source, which emits `DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}`
per DRM node. It is the only non-privileged way to get there; `privileged:
true` is blocked by Kyverno restrict-host-namespaces under Enforce in app
namespaces, and `plex` is labelled app-namespace=true.

PLACEMENT: its own Kustomization (dependsOn cilium, wait:true), not a line in
platform/kustomization.yaml. The shared tier was cheaper and needed no human
reviewer on approval-policy.json's terms, but it is wait:true over eleven-plus
components and the repo already argued this when Cilium was split out: its
Ready condition cannot tell "the plugin is wedged" from "longhorn is slow", the
only emergency valve is freezing the whole tier, and the tier's 10m/5m
interval/timeout are called out there as wrong for a 3-node DaemonSet rollout —
which is exactly what this is. Worse here than average: platform gates
platform-config gates apps-wave-0 (wait:true) gates apps-wave-1, so a plugin
that never went healthy inside it would stall every app — a component added to
fix a silent no-op would have gained the power to cause a real outage.
Touching clusters/** costs nothing extra: required_approvals:1 with
block_admin_merge_override:true applies to every PR here regardless of path.

wait:true is a health SIGNAL, not a gate — nothing in the rendered graph
dependsOn this, so it gates nothing, while making `flux get kustomization
intel-gpu-plugin` answer "did it come up on all three nodes".

NON-CHOICES, each argued in the README: standalone DaemonSet over the operator
(no CRD, controller-manager, webhook or second namespace to keep healthy);
`--shared-dev-num` left at the upstream default of 1 (a higher count is a lie
the scheduler believes — no GPU-side partitioning backs it, so it converts a
loud Unschedulable into a quiet "transcodes are slow"); `--health-management`
left OFF, the one flag that would open a device and reintroduce this EPERM; no
NFD overlay (all three nodes identical); no supplementalGroups (the pms-docker
cont-init hook already adds `plex` to each device's group, measured live as
Groups: 44 100 992 1000).

The DaemonSet is vendored as `kubectl kustomize` output of upstream v0.37.0's
gpu_plugin/base, rather than referenced as a remote kustomize base, for the
reason external-snapshotter records: a remote base is git-fetched on every
reconcile and those fetches have flaked a whole tier. It is generated rather
than byte-copied because upstream mixes YAML sequence-indentation styles within
the one document and ci/yamllint.yaml sets `indent-sequences: consistent`, so a
verbatim copy is a hard lint ERROR — measured, not guessed: the first gate run
of this branch returned rc 1 BLOCKED on exactly that. The transform is proven
formatting-only (`yq -P 'sort_keys(..)'` of both diffs clean). Image pinned to
docker.io/intel/intel-gpu-plugin:0.37.0 (digest verified present:
sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1).

ORDERING — DO NOT REORDER. catalog/render.py, catalog/plex.yaml and
apps/wave-0/plex.yaml are untouched. The renderer change that emits
`gpu.intel.com/i915: 1` instead of the hostPath must NOT land until this plugin
is merged AND confirmed advertising the resource on all three nodes: a request
for a resource no node advertises makes the Plex pod Unschedulable, and
apps-wave-0 is wait:true with apps-wave-1 depending on it.

VERIFIED: manifests render offline (component, clusters/base, both overlays);
no node currently advertises gpu.intel.com/* (queried — "NONE"); the plugin
only lists/stats devices and never opens one (no os.Open/OpenFile/ioctl/syscall
anywhere in its 908-line source); all four app-scoped Kyverno policies are
gated on app-namespace=true and the new namespace is labelled "false".
NOT VERIFIED: anything runtime. The plugin has never executed, the resource has
never been advertised, and the open() test has not been re-run. Gate returned
rc 3 (UNDETERMINED) with exactly the three expected skips — not green.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85 approved these changes 2026-10-02 03:46:50 +00:00
jakman85 merged commit 8214642d87 into main 2026-10-02 03:53:50 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
jakman85
Labels
Clear labels
agent
automated-update
blast-radius
breaking
ci-tooling
dependencies
needs-review
own-image
range-widen
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
gitea-admin jakman85 (Keith Jackson) pm-agent renovate-bot
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#36