feat(gpu): Intel GPU device plugin v0.37.0, in its own Flux Kustomization
Makes the three prod nodes advertise `gpu.intel.com/i915`. This is the measured
fix for the Plex hardware-transcoding blocker, and it only makes the resource
EXIST — no consumer is switched over here.
WHY. The `/dev/dri` hostPath merged on main @ cdf3e14 makes the device node
visible and not openable. Measured in the running pod on k3s-prod-2:
uid 0, O_RDWR -> EPERM "Operation not permitted"
uid 1000, O_RDWR -> EPERM "Operation not permitted"
EPERM from the device cgroup, not EACCES from file permissions — root on the
HOST opens the same device rc=0, the node is unified cgroup2fs, and the
container's runc config.json carries one deny-all rule with `linux.devices:
None`. A hostPath is a filesystem operation: it makes the device node appear
and adds no cgroup allow rule, because Kubernetes cannot infer one from a
volume. A device plugin's Allocate response is what makes kubelet populate
`linux.devices` AND the matching cgroup allow rule — verified in the v0.37.0
source, which emits `DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}`
per DRM node. It is the only non-privileged way to get there; `privileged:
true` is blocked by Kyverno restrict-host-namespaces under Enforce in app
namespaces, and `plex` is labelled app-namespace=true.
PLACEMENT: its own Kustomization (dependsOn cilium, wait:true), not a line in
platform/kustomization.yaml. The shared tier was cheaper and needed no human
reviewer on approval-policy.json's terms, but it is wait:true over eleven-plus
components and the repo already argued this when Cilium was split out: its
Ready condition cannot tell "the plugin is wedged" from "longhorn is slow", the
only emergency valve is freezing the whole tier, and the tier's 10m/5m
interval/timeout are called out there as wrong for a 3-node DaemonSet rollout —
which is exactly what this is. Worse here than average: platform gates
platform-config gates apps-wave-0 (wait:true) gates apps-wave-1, so a plugin
that never went healthy inside it would stall every app — a component added to
fix a silent no-op would have gained the power to cause a real outage.
Touching clusters/** costs nothing extra: required_approvals:1 with
block_admin_merge_override:true applies to every PR here regardless of path.
wait:true is a health SIGNAL, not a gate — nothing in the rendered graph
dependsOn this, so it gates nothing, while making `flux get kustomization
intel-gpu-plugin` answer "did it come up on all three nodes".
NON-CHOICES, each argued in the README: standalone DaemonSet over the operator
(no CRD, controller-manager, webhook or second namespace to keep healthy);
`--shared-dev-num` left at the upstream default of 1 (a higher count is a lie
the scheduler believes — no GPU-side partitioning backs it, so it converts a
loud Unschedulable into a quiet "transcodes are slow"); `--health-management`
left OFF, the one flag that would open a device and reintroduce this EPERM; no
NFD overlay (all three nodes identical); no supplementalGroups (the pms-docker
cont-init hook already adds `plex` to each device's group, measured live as
Groups: 44 100 992 1000).
The DaemonSet is vendored as `kubectl kustomize` output of upstream v0.37.0's
gpu_plugin/base, rather than referenced as a remote kustomize base, for the
reason external-snapshotter records: a remote base is git-fetched on every
reconcile and those fetches have flaked a whole tier. It is generated rather
than byte-copied because upstream mixes YAML sequence-indentation styles within
the one document and ci/yamllint.yaml sets `indent-sequences: consistent`, so a
verbatim copy is a hard lint ERROR — measured, not guessed: the first gate run
of this branch returned rc 1 BLOCKED on exactly that. The transform is proven
formatting-only (`yq -P 'sort_keys(..)'` of both diffs clean). Image pinned to
docker.io/intel/intel-gpu-plugin:0.37.0 (digest verified present:
sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1).
ORDERING — DO NOT REORDER. catalog/render.py, catalog/plex.yaml and
apps/wave-0/plex.yaml are untouched. The renderer change that emits
`gpu.intel.com/i915: 1` instead of the hostPath must NOT land until this plugin
is merged AND confirmed advertising the resource on all three nodes: a request
for a resource no node advertises makes the Plex pod Unschedulable, and
apps-wave-0 is wait:true with apps-wave-1 depending on it.
VERIFIED: manifests render offline (component, clusters/base, both overlays);
no node currently advertises gpu.intel.com/* (queried — "NONE"); the plugin
only lists/stats devices and never opens one (no os.Open/OpenFile/ioctl/syscall
anywhere in its 908-line source); all four app-scoped Kyverno policies are
gated on app-namespace=true and the new namespace is labelled "false".
NOT VERIFIED: anything runtime. The plugin has never executed, the resource has
never been advertised, and the open() test has not been re-run. Gate returned
rc 3 (UNDETERMINED) with exactly the three expected skips — not green.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
cdf3e14e27
commit
4e1b169256
7 files changed
+786
No files matched your search
@@ -0,0 +1,225 @@
|
||||
# platform/intel-gpu-plugin
|
||||
|
||||
**Intel Device Plugin for Kubernetes (GPU)** — the standalone GPU plugin DaemonSet from
|
||||
[`intel/intel-device-plugins-for-kubernetes`](https://github.com/intel/intel-device-plugins-for-kubernetes),
|
||||
pinned at **v0.37.0**. It makes every node advertise **`gpu.intel.com/i915`** so a workload can
|
||||
*request* a GPU instead of hoping a `hostPath` is enough.
|
||||
|
||||
- **Serves:** hardware transcoding for Plex (`hardware.gpu: intel-quicksync`), and any later
|
||||
consumer of an Intel iGPU.
|
||||
- **Tier:** its **own** Flux Kustomization
|
||||
([`clusters/base/intel-gpu-plugin.yaml`](../../clusters/base/intel-gpu-plugin.yaml),
|
||||
`dependsOn: cilium`, `wait: true`) — **not** a line in `platform/kustomization.yaml`. See
|
||||
*Placement* below.
|
||||
- **Namespace:** `intel-gpu-plugin` — a PLATFORM namespace, labelled
|
||||
`homelab.keithjackson.dev/app-namespace: "false"`, so the app-scoped Kyverno policies
|
||||
(default-deny generation, auth-mode, egress-class, host-namespaces) do not apply.
|
||||
- **No RBAC, no ServiceAccount, no CRDs, no webhook, no API-server access.** A device plugin talks
|
||||
to the *local kubelet* over a unix socket.
|
||||
|
||||
## What this component owns
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `namespace.yaml` | `intel-gpu-plugin` ns, explicitly `app-namespace=false` |
|
||||
| `intel-gpu-plugin.yaml` | the DaemonSet, **vendored** (machine-generated) from upstream v0.37.0 |
|
||||
| `kustomization.yaml` | namespace scoping + the pinned image (`images:` transformer) |
|
||||
|
||||
## The problem this fixes
|
||||
|
||||
Plex was given a `/dev/dri` hostPath (merged on `main` @ `cdf3e14`). The device is **visible** in
|
||||
the container and **cannot be opened**. Measured in the running pod on `k3s-prod-2`:
|
||||
|
||||
```
|
||||
uid 0, O_RDWR -> EPERM "Operation not permitted"
|
||||
uid 1000, O_RDWR -> EPERM "Operation not permitted"
|
||||
```
|
||||
|
||||
`EPERM` from the **device cgroup**, not `EACCES` from file permissions — proven three ways:
|
||||
|
||||
1. root on the **host** opens the same device with `rc=0`;
|
||||
2. the node is unified `cgroup2fs`;
|
||||
3. the container's runc `config.json` reads
|
||||
`linux.resources.devices: [{'allow': False, 'access': 'rwm'}]` and `linux.devices: None` —
|
||||
one deny-all rule, no allow entry for major 226, and the runtime was never asked to grant a
|
||||
device.
|
||||
|
||||
A hostPath is a **filesystem** operation. It makes the device *node* appear and adds no cgroup
|
||||
allow rule, because Kubernetes cannot infer one from a volume.
|
||||
|
||||
**A device plugin is the mechanism that closes exactly this gap.** Verified in the v0.37.0 source
|
||||
(`cmd/gpu_plugin/gpu_plugin.go`, `createDeviceSpecsFromDrmFiles`), the `Allocate` response carries
|
||||
per DRM node:
|
||||
|
||||
```go
|
||||
specs = append(specs, pluginapi.DeviceSpec{
|
||||
HostPath: devPath, ContainerPath: devPath, Permissions: "rw",
|
||||
})
|
||||
```
|
||||
|
||||
and it is that `DeviceSpec` which makes kubelet populate `linux.devices` **and** the matching
|
||||
cgroup allow rule. It is the only **non-privileged** way to get there. `privileged: true` would
|
||||
also work and is not available: `platform/kyverno/policies/restrict-host-namespaces.yaml` blocks it
|
||||
under Enforce in app namespaces, and `plex` is labelled `app-namespace=true` (measured
|
||||
2026-10-01). Correctly so.
|
||||
|
||||
## Placement — why its own Kustomization
|
||||
|
||||
The cheap option was one line in the shared `platform` tier. Neither that file nor a new
|
||||
`platform/<name>/` directory is a sensitive path in `infra/gitea/approval-policy.json`, so it would
|
||||
have needed no human reviewer *on the policy's terms*. It is still the wrong place, and the repo
|
||||
already argued the case when Cilium was split out (read the comment in
|
||||
`platform/kustomization.yaml`):
|
||||
|
||||
* `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot distinguish
|
||||
"the GPU plugin is wedged" from "longhorn is slow".
|
||||
* The only emergency valve would be `flux suspend kustomization platform` — freezing storage, TLS,
|
||||
DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device plugin.
|
||||
* The tier's 10m/5m interval/timeout are called out there as **wrong for a 3-node DaemonSet
|
||||
rollout**. This is a 3-node DaemonSet rollout.
|
||||
|
||||
The consequence is worse than average here because of what sits downstream: `platform` gates
|
||||
`platform-config`, which gates `apps-wave-0` (`wait: true`), which gates `apps-wave-1`. A plugin
|
||||
that never went healthy inside that tier would stall **every app** on "dependency not ready" — so a
|
||||
component added to fix a *silent no-op* (software transcoding) would have acquired the ability to
|
||||
cause a *real outage*.
|
||||
|
||||
The usual objection — that `clusters/**` needs a human approval — costs nothing: Gitea enforces
|
||||
`required_approvals: 1` with `block_admin_merge_override: true` on **every** PR here regardless of
|
||||
path, and authors cannot self-approve. The approval was required either way.
|
||||
|
||||
### `wait: true` is a health signal, not a gate
|
||||
|
||||
This departs from the `headlamp`/`crafty` leaf template, deliberately. Those run `wait: false`
|
||||
because a leaf UI can be legitimately not-Ready for a long time during a cold start while it waits
|
||||
on Keycloak/DNS/TLS, making its health useless as a signal. This DaemonSet has no such dependency —
|
||||
it reads sysfs and opens a socket, so on healthy nodes it is Ready in seconds and **not-Ready
|
||||
genuinely means broken**. `wait: true` makes `flux get kustomization intel-gpu-plugin` a one-line
|
||||
answer to "did it come up on all three nodes", which is precisely the question that must be
|
||||
answered before the consumer PR may merge.
|
||||
|
||||
Nothing in the graph `dependsOn` this, so `wait: true` gates nothing. **Think hard before adding
|
||||
`dependsOn: [intel-gpu-plugin]`** anywhere — for the app waves that would hand a device plugin the
|
||||
power to stall every app, which is the thing this placement exists to prevent.
|
||||
|
||||
## Deliberate non-choices
|
||||
|
||||
**Standalone DaemonSet, not the operator.** The operator adds a CRD (`GpuDevicePlugin`), a
|
||||
controller-manager Deployment, cert-manager-backed admission webhooks and a second namespace —
|
||||
three more things that can be unhealthy — to buy declarative management of a DaemonSet that is
|
||||
already declared in git. For three identical nodes and one consumer that is strictly more blast
|
||||
radius for no gain.
|
||||
|
||||
**It lists devices; it never opens them** — worth being certain about, since *opening* is exactly
|
||||
what fails today. Verified in v0.37.0 source: the scan path uses only `os.ReadDir`, `os.ReadFile`,
|
||||
`os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. There is no `os.Open`, no
|
||||
`OpenFile`, no `ioctl` and no raw syscall against a DRM node anywhere in
|
||||
`cmd/gpu_plugin/gpu_plugin.go`. Listing and `stat()` demonstrably work today — that is how the Plex
|
||||
probe found the device before failing to open it — so the plugin needs no cgroup grant of its own
|
||||
and no privilege. Accordingly `/dev/dri` and `/sys/class/drm` are mounted **readOnly**.
|
||||
|
||||
* **`--health-management` stays OFF** (upstream default). It is the one flag that *would* need to
|
||||
open a device: it starts a Level-Zero sidecar (`setupLevelZeroService` returns early unless the
|
||||
flag is set). Enabling it would reintroduce the exact `EPERM` this component exists to fix.
|
||||
* `/dev/dri/by-path` being absent is non-fatal — `newDevicePlugin` warns and sets
|
||||
`bypathFound = false`.
|
||||
|
||||
**`--shared-dev-num` left at the upstream default of 1** (i.e. not set). The flag multiplies each
|
||||
card into N schedulable device IDs, letting N containers share one GPU. Not raised, because:
|
||||
|
||||
* Plex is the only consumer today and wants one device;
|
||||
* the count is a **lie the scheduler believes** — there is no GPU-side partitioning or quota behind
|
||||
it, so N containers get the same `renderD128` and contend for the same two media engines
|
||||
(`vcs0`/`vecs0`). Raising it converts "Unschedulable", which is loud and obvious, into
|
||||
"transcodes are mysteriously slow", which is neither;
|
||||
* it is a one-line, zero-migration change on the day a second consumer exists — and *then* the
|
||||
right number will be known rather than guessed.
|
||||
|
||||
**No NFD overlay.** All three nodes are identical (HP EliteDesk 800 G5 Mini, i7-9700T, UHD Graphics
|
||||
630, PCI `8086:3e98`, `i915` bound), so a nodeSelector on a GPU feature label would gate on
|
||||
something true everywhere. If a GPU-less node ever joins, add upstream's `nfd` overlay *then*
|
||||
rather than relying on the plugin's behaviour on such a node.
|
||||
|
||||
**No `supplementalGroups`, anywhere.** The `pms-docker` image's own cont-init hook stats each
|
||||
`/dev/dri` char device and adds the `plex` user to its group — measured live as
|
||||
`Groups: 44 100 992 1000` on the running Plex process, where GID 992 is the host `render` group.
|
||||
That mechanism is image-side and works identically on a plugin-injected device.
|
||||
|
||||
## Resource name
|
||||
|
||||
`gpu.intel.com/i915`. Not a guess: `scan()` calls `devTree.AddDevice(devProps.driver(), ...)`, so
|
||||
the suffix is the **kernel driver name**. These nodes have `i915` bound (not `xe`), so the resource
|
||||
is `i915`. A future kernel moving CoffeeLake to `xe` would silently rename it to `gpu.intel.com/xe`
|
||||
and a consumer's request would stop matching — worth knowing before that kernel arrives.
|
||||
|
||||
## Versioning
|
||||
|
||||
Pinned to **v0.37.0** via the `images:` transformer in `kustomization.yaml`; the DaemonSet itself is
|
||||
vendored as `kubectl kustomize` output of the upstream base (see its header for the exact re-vendor
|
||||
command and both sha256s). It is *generated* rather than a byte copy because upstream mixes YAML
|
||||
sequence-indentation styles within the one document and `ci/yamllint.yaml` sets
|
||||
`indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate
|
||||
(measured: it did). `kubectl kustomize` normalises the formatting, and the result was proven to be
|
||||
a formatting-only change: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file
|
||||
diff clean. This is also the re-vendor idiom `platform/external-snapshotter` documents.
|
||||
|
||||
Vendored rather than referenced as a remote kustomize base for the same reason as
|
||||
`platform/external-snapshotter`: a remote base is `git fetch`ed on **every** reconcile, those
|
||||
fetches intermittently exceeded kustomize-controller's 27s timeout, and transient GitHub slowness
|
||||
then rendered as "kustomize build failed". A local file makes the build offline and deterministic —
|
||||
which also means the CI gate's `kustomize build` proves the manifests rather than proving GitHub
|
||||
was up.
|
||||
|
||||
Renovate tracks it through the inline-marker idiom (`# renovate: datasource=github-releases
|
||||
depName=...` immediately above the version literal), because `renovate.json` sets
|
||||
`"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}` — an image reference in a
|
||||
plain manifest is tracked by nothing. **Caveat:** upstream *release* tags carry a `v` (`v0.37.0`)
|
||||
while *image* tags do not (`0.37.0`). Whether Renovate normalises that for this dep is unverified;
|
||||
if it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
|
||||
`renovate.json`, not a hand-edit.
|
||||
|
||||
## Ordering — do not reorder the stack
|
||||
|
||||
The renderer change (`catalog/render.py` emitting `gpu.intel.com/i915: 1` for
|
||||
`hardware.gpu: intel-quicksync` instead of the hostPath) **must not land until this plugin is
|
||||
merged AND confirmed advertising the resource on all three nodes.** If the renderer requests a
|
||||
resource no node advertises, the Plex pod is **Unschedulable**; `apps-wave-0` is `wait: true` and
|
||||
`apps-wave-1` `dependsOn` it, so that stalls the whole app tier — turning today's silent no-op into
|
||||
a real outage.
|
||||
|
||||
## Post-merge verification
|
||||
|
||||
Nothing about this component's runtime behaviour can be verified from a PR: it does not exist until
|
||||
the operator merges and Flux reconciles. Afterwards:
|
||||
|
||||
```bash
|
||||
D=./infra/devcontainer/dev.sh
|
||||
|
||||
# 1. Flux applied it, and (wait:true) the DaemonSet went healthy.
|
||||
$D run bash -c 'flux get kustomization intel-gpu-plugin'
|
||||
$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin get ds intel-gpu-plugin'
|
||||
# want: DESIRED=3 CURRENT=3 READY=3 AVAILABLE=3
|
||||
|
||||
# 2. The resource is advertised on ALL THREE nodes. This is the gate on the renderer PR.
|
||||
$D run bash -c 'kubectl --context k3s-prod get nodes -o json \
|
||||
| jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
|
||||
# want: three lines, each i915=1 (before this change: ABSENT on all three)
|
||||
|
||||
# 3. If a node shows ABSENT, read that node's plugin logs.
|
||||
$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin logs -l app=intel-gpu-plugin --tail=50 --prefix'
|
||||
```
|
||||
|
||||
### The decisive test
|
||||
|
||||
Resource-advertised is necessary but **not** sufficient — it proves the plugin registered, not that
|
||||
the cgroup allow rule reaches Plex. The decisive test is re-running the `open()` probe **as uid 1000
|
||||
through `s6-setuidgid plex`** inside the Plex pod, *after* the renderer PR has switched Plex to
|
||||
request the resource:
|
||||
|
||||
```bash
|
||||
$D run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
|
||||
s6-setuidgid plex python3 -c "import os; os.close(os.open(\"/dev/dri/renderD128\", os.O_RDWR)); print(\"rc=0 OK\")"'
|
||||
```
|
||||
|
||||
**Passing as uid 0 is weaker evidence and must not be accepted as proof** — Plex transcodes as uid
|
||||
1000, and the two differ in group membership, which is the half the image's cont-init hook supplies.
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
# VENDORED, MACHINE-GENERATED, from intel/intel-device-plugins-for-kubernetes @ v0.37.0:
|
||||
# deployments/gpu_plugin/base
|
||||
#
|
||||
# RE-VENDOR on a version bump (and bump `newTag` in kustomization.yaml to match):
|
||||
# kubectl kustomize \
|
||||
# "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/base?ref=vX.Y.Z" \
|
||||
# > platform/intel-gpu-plugin/intel-gpu-plugin.yaml
|
||||
# # then re-add this header, and READ THE DIFF — upstream has changed the securityContext
|
||||
# # and the volume set between minor releases before.
|
||||
#
|
||||
# Generated 2026-10-01 at v0.37.0; sha256 of the body below (header excluded):
|
||||
# 80ebd070f983f3116f4004f719d9c872220997ec766b4ab7e7290585ac8ef842
|
||||
# The raw upstream source file reviewed to produce it
|
||||
# (deployments/gpu_plugin/base/intel-gpu-plugin.yaml) was sha256
|
||||
# c8773918c2370e711d4ba84120a9971a290bf75ecf81dc520782bcccd7ef24b6
|
||||
# and the two were proven EQUIVALENT, not merely similar: `yq -P 'sort_keys(..)'` of each diffs
|
||||
# clean, so passing it through kustomize changed formatting only and no content.
|
||||
#
|
||||
# WHY GENERATED RATHER THAN A BYTE COPY of the upstream file: upstream mixes YAML sequence
|
||||
# indentation styles within the one document (`containers:` non-indented, `env:` indented), and
|
||||
# ci/yamllint.yaml sets `indentation: indent-sequences: consistent`, so a verbatim copy is a hard
|
||||
# lint ERROR and blocks the merge gate — measured, this file failed exactly that way first time
|
||||
# round. `kubectl kustomize` normalises the output, which is also precisely the re-vendor idiom
|
||||
# platform/external-snapshotter documents for the same reason.
|
||||
#
|
||||
# WHY VENDORED AT ALL rather than a remote kustomize base (`...gpu_plugin/base?ref=v0.37.0` left in
|
||||
# `resources:`): the same reason platform/external-snapshotter is vendored. A remote base is
|
||||
# `git fetch`ed by Flux's kustomize-controller on EVERY reconcile, those fetches intermittently
|
||||
# exceeded the controller's 27s timeout, and a transient GitHub slowness then rendered as
|
||||
# "kustomize build failed" on the whole tier. A local file makes the build OFFLINE and
|
||||
# deterministic — which also means `kustomize build` in the CI gate proves the real thing instead
|
||||
# of proving GitHub was up. (platform/kubevirt still carries the remote-base pattern and its latent
|
||||
# fragility; not this component's problem.)
|
||||
#
|
||||
# DO NOT HAND-EDIT THE BODY. Every local change belongs in kustomization.yaml as a transformer or
|
||||
# patch, so the next re-vendor stays a mechanical regenerate-and-diff instead of a three-way merge.
|
||||
# The image tag below is re-pinned (and fully qualified) by the `images:` transformer there. The
|
||||
# rationale for each thing deliberately NOT changed — no privileged, no supplementalGroups, no
|
||||
# -shared-dev-num, no NFD overlay, no health-management — is in README.md.
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
labels:
|
||||
app: intel-gpu-plugin
|
||||
name: intel-gpu-plugin
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: intel-gpu-plugin
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: intel-gpu-plugin
|
||||
spec:
|
||||
containers:
|
||||
- env:
|
||||
- name: NODE_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: spec.nodeName
|
||||
- name: HOST_IP
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: status.hostIP
|
||||
image: intel/intel-gpu-plugin:0.37.0
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: intel-gpu-plugin
|
||||
resources:
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 115Mi
|
||||
requests:
|
||||
cpu: 40m
|
||||
memory: 45Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
seLinuxOptions:
|
||||
type: container_device_plugin_t
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
volumeMounts:
|
||||
- mountPath: /dev/dri
|
||||
name: devfs
|
||||
readOnly: true
|
||||
- mountPath: /sys/class/drm
|
||||
name: sysfsdrm
|
||||
readOnly: true
|
||||
- mountPath: /var/lib/kubelet/device-plugins
|
||||
name: kubeletsockets
|
||||
- mountPath: /var/run/cdi
|
||||
name: cdipath
|
||||
nodeSelector:
|
||||
kubernetes.io/arch: amd64
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /dev/dri
|
||||
name: devfs
|
||||
- hostPath:
|
||||
path: /sys/class/drm
|
||||
name: sysfsdrm
|
||||
- hostPath:
|
||||
path: /var/lib/kubelet/device-plugins
|
||||
name: kubeletsockets
|
||||
- hostPath:
|
||||
path: /var/run/cdi
|
||||
type: DirectoryOrCreate
|
||||
name: cdipath
|
||||
updateStrategy:
|
||||
rollingUpdate:
|
||||
maxSurge: 0
|
||||
maxUnavailable: 1
|
||||
type: RollingUpdate
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
# Intel GPU device plugin for Kubernetes — makes the nodes advertise `gpu.intel.com/i915`.
|
||||
#
|
||||
# WHAT THIS FIXES, and why nothing cheaper works. Plex already has a `/dev/dri` hostPath mount
|
||||
# (merged on main @ cdf3e14). The device node is VISIBLE in the container and cannot be OPENED.
|
||||
# Measured in the running pod on k3s-prod-2:
|
||||
# uid 0, O_RDWR -> EPERM "Operation not permitted"
|
||||
# uid 1000, O_RDWR -> EPERM "Operation not permitted"
|
||||
# EPERM from the DEVICE CGROUP, not EACCES from file permissions — proven three ways: root on the
|
||||
# HOST opens the same device rc=0; the node is unified cgroup2fs; and the container's runc
|
||||
# config.json carries one deny-all rule and nothing else:
|
||||
# linux.resources.devices: [{'allow': False, 'access': 'rwm'}]
|
||||
# linux.devices: None
|
||||
# No allow entry for major 226, and the runtime was never asked to grant a device. A hostPath is a
|
||||
# FILESYSTEM operation: it makes the device node appear, and adds no cgroup allow rule, because
|
||||
# Kubernetes cannot infer one from a volume.
|
||||
#
|
||||
# A device plugin is the mechanism that closes exactly that gap. Verified by reading the v0.37.0
|
||||
# source (cmd/gpu_plugin/gpu_plugin.go, createDeviceSpecsFromDrmFiles): the Allocate response
|
||||
# carries `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, and it
|
||||
# is that DeviceSpec which makes kubelet populate `linux.devices` AND the matching cgroup allow
|
||||
# rule. It is the only NON-PRIVILEGED way to get there. `privileged: true` would also work and is
|
||||
# not available: platform/kyverno/policies/restrict-host-namespaces.yaml blocks it under Enforce in
|
||||
# app namespaces, and `plex` is labelled app-namespace=true (measured 2026-10-01). Correctly so.
|
||||
#
|
||||
# STANDALONE DAEMONSET, NOT THE OPERATOR. Upstream ships both. The operator adds a CRD
|
||||
# (GpuDevicePlugin), a controller-manager Deployment, cert-manager-backed admission webhooks and a
|
||||
# second namespace — i.e. three more things that can be unhealthy — to buy declarative management
|
||||
# of a DaemonSet we are declaring in git anyway. For three identical nodes and one consumer that is
|
||||
# strictly more blast radius for no gain. The DaemonSet has no CRDs, no controller, no webhook, and
|
||||
# no API-server access at all: a device plugin talks to the LOCAL kubelet over a unix socket, which
|
||||
# is why there is no ServiceAccount or RBAC here (and therefore no `**/rbac.yaml` sensitive path).
|
||||
#
|
||||
# LISTING, NOT OPENING — the thing worth being sure about, since opening is precisely what fails
|
||||
# today. Verified in v0.37.0 source: the scan path uses only os.ReadDir / os.ReadFile / os.Stat /
|
||||
# os.Readlink over /sys/class/drm and /dev/dri. There is no os.Open, no OpenFile, no ioctl and no
|
||||
# syscall of any kind against a DRM node anywhere in cmd/gpu_plugin/gpu_plugin.go. Listing and
|
||||
# stat() demonstrably work today (that is how the Plex probe found the device before failing to
|
||||
# open it), so the plugin needs no cgroup grant of its own and no privilege.
|
||||
# * `--health-management` is left OFF (upstream default). It is the one flag that WOULD need to
|
||||
# open a device: it starts a Level-Zero sidecar (setupLevelZeroService returns early unless the
|
||||
# flag is set). Turning it on would reintroduce the exact EPERM this component exists to fix.
|
||||
# * /dev/dri and /sys/class/drm are mounted readOnly, consistent with list-and-stat only.
|
||||
# * /dev/dri/by-path absence is non-fatal (newDevicePlugin warns and sets bypathFound=false).
|
||||
#
|
||||
# RESOURCE NAME: `gpu.intel.com/i915`. Not a guess — `scan()` does
|
||||
# `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the KERNEL DRIVER name. These nodes
|
||||
# have i915 bound (not xe) on UHD Graphics 630 / PCI 8086:3e98, so the resource is i915. A future
|
||||
# kernel moving CoffeeLake to xe would silently rename it to `gpu.intel.com/xe` and the consumer's
|
||||
# request would stop matching — worth knowing before that kernel arrives.
|
||||
#
|
||||
# `--shared-dev-num` LEFT AT THE UPSTREAM DEFAULT OF 1 (so: not set). The flag multiplies each
|
||||
# card into N schedulable device IDs (`for i := 0; i < sharedDevNum; i++`), letting N containers
|
||||
# share one GPU. Deliberately not raised:
|
||||
# * Plex is the only consumer and wants one device.
|
||||
# * The count is a LIE the scheduler believes. There is no GPU-side partitioning or quota behind
|
||||
# it — N containers get the same renderD128 and contend for the same two media engines
|
||||
# (vcs0/vecs0). Raising it converts "Unschedulable", which is loud and obvious, into
|
||||
# "transcodes are mysteriously slow", which is neither.
|
||||
# * It is a one-line, zero-migration change on the day a second consumer actually exists, and at
|
||||
# that point the right number will be known rather than guessed.
|
||||
# No NFD overlay either: all three nodes are identical (established cluster fact), so a nodeSelector
|
||||
# on a GPU feature label would gate on something that is true everywhere. If a GPU-less node ever
|
||||
# joins, add upstream's nfd overlay THEN rather than relying on the plugin's behaviour there.
|
||||
#
|
||||
# NO supplementalGroups, anywhere, on purpose. The pms-docker image's own cont-init hook stats each
|
||||
# /dev/dri char device and adds the `plex` user to its group — measured live as `Groups: 44 100 992
|
||||
# 1000` on the running Plex process, where GID 992 is the host `render` group. That mechanism is
|
||||
# image-side and works identically on a plugin-injected device, so adding a supplementalGroup here
|
||||
# or on the consumer would duplicate a thing that already works.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: intel-gpu-plugin
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- intel-gpu-plugin.yaml
|
||||
images:
|
||||
# PINNED, never `latest`. Upstream release v0.37.0 (2026-09-16); verified present on Docker Hub
|
||||
# 2026-10-01, manifest digest sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1
|
||||
# (single amd64 manifest, not an index). Recorded here rather than pinned by digest because this
|
||||
# repo pins upstream platform components by VERSION TAG (platform/kubevirt, external-snapshotter);
|
||||
# digest pinning is reserved for the conformance runner, where it is a security boundary.
|
||||
# `newName` fully qualifies the registry: upstream writes the bare `intel/intel-gpu-plugin`, which
|
||||
# relies on containerd's docker.io default rather than saying so.
|
||||
#
|
||||
# RENOVATE: the inline-marker idiom (customManagers in renovate.json), not the docker datasource —
|
||||
# renovate.json sets `"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}`, so an
|
||||
# image reference in a plain manifest is tracked by NOTHING. The marker must sit on the line
|
||||
# IMMEDIATELY above the version literal for the regex to match, which is why it is down here on
|
||||
# `newTag` and not up in this header.
|
||||
# CAVEAT, stated because it will bite someone: the GitHub release tags carry a `v`
|
||||
# (`v0.37.0`) and the IMAGE tags do not (`0.37.0` — confirmed against the Docker Hub tag list).
|
||||
# Whether Renovate normalises that for this dep is UNVERIFIED here (no Renovate run available).
|
||||
# If it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
|
||||
# renovate.json, not a hand-edit of this line.
|
||||
- name: intel/intel-gpu-plugin
|
||||
newName: docker.io/intel/intel-gpu-plugin
|
||||
# renovate: datasource=github-releases depName=intel/intel-device-plugins-for-kubernetes
|
||||
newTag: "0.37.0"
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
# intel-gpu-plugin — namespace for the Intel GPU device plugin DaemonSet.
|
||||
#
|
||||
# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and that distinction is load-bearing here rather
|
||||
# than cosmetic. Every Kyverno policy that could object to this DaemonSet is scoped by a
|
||||
# `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"` — verified by
|
||||
# reading the policies, not assumed:
|
||||
# * platform/kyverno/policies/restrict-host-namespaces.yaml (hostNetwork/PID/IPC + privileged)
|
||||
# * platform/kyverno/policies/require-egress-class-label.yaml
|
||||
# * platform/kyverno/policies/require-auth-mode-label.yaml
|
||||
# * platform/kyverno/policies/require-default-deny-baseline.yaml (a *generate* rule, so an
|
||||
# app-namespace also gets a default-deny NetworkPolicy minted into it)
|
||||
# Labelling this namespace "false" therefore puts it out of scope for all four. That is the same
|
||||
# mechanism restrict-host-namespaces' own header describes: "platform components that do [need host
|
||||
# access] live in platform namespaces that do NOT carry the app-namespace label".
|
||||
#
|
||||
# Measured on prod 2026-10-01: `kyverno` and `vpn-egress` are the two namespaces that carry the
|
||||
# label explicitly as "false"; the rest of the platform namespaces leave it unset. Both are out of
|
||||
# scope identically — an unset label does not match `"true"`. We set it EXPLICITLY, following
|
||||
# vpn-egress, because "someone chose this" and "nobody thought about it" should not look the same
|
||||
# in git, and because this component's correctness depends on the answer.
|
||||
#
|
||||
# NOTE this plugin does NOT actually need any of the things those policies forbid: it sets no
|
||||
# hostNetwork/hostPID/hostIPC and is not privileged (see intel-gpu-plugin.yaml). The namespace
|
||||
# placement is not a workaround for a policy — there is nothing to work around. It is here because
|
||||
# a node-level platform DaemonSet is not an application, and because the app-namespace fabric
|
||||
# (default-deny NetworkPolicy, egress-class, auth-mode) describes obligations that are meaningless
|
||||
# for a process that talks only to the local kubelet over a unix socket and never uses the network.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: intel-gpu-plugin
|
||||
labels:
|
||||
homelab.keithjackson.dev/app-namespace: "false"
|
||||
Reference in new issue
Block a user