feat(gpu): Intel GPU device plugin v0.37.0, in its own Flux Kustomization
Makes the three prod nodes advertise `gpu.intel.com/i915`. This is the measured
fix for the Plex hardware-transcoding blocker, and it only makes the resource
EXIST — no consumer is switched over here.
WHY. The `/dev/dri` hostPath merged on main @ cdf3e14 makes the device node
visible and not openable. Measured in the running pod on k3s-prod-2:
uid 0, O_RDWR -> EPERM "Operation not permitted"
uid 1000, O_RDWR -> EPERM "Operation not permitted"
EPERM from the device cgroup, not EACCES from file permissions — root on the
HOST opens the same device rc=0, the node is unified cgroup2fs, and the
container's runc config.json carries one deny-all rule with `linux.devices:
None`. A hostPath is a filesystem operation: it makes the device node appear
and adds no cgroup allow rule, because Kubernetes cannot infer one from a
volume. A device plugin's Allocate response is what makes kubelet populate
`linux.devices` AND the matching cgroup allow rule — verified in the v0.37.0
source, which emits `DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}`
per DRM node. It is the only non-privileged way to get there; `privileged:
true` is blocked by Kyverno restrict-host-namespaces under Enforce in app
namespaces, and `plex` is labelled app-namespace=true.
PLACEMENT: its own Kustomization (dependsOn cilium, wait:true), not a line in
platform/kustomization.yaml. The shared tier was cheaper and needed no human
reviewer on approval-policy.json's terms, but it is wait:true over eleven-plus
components and the repo already argued this when Cilium was split out: its
Ready condition cannot tell "the plugin is wedged" from "longhorn is slow", the
only emergency valve is freezing the whole tier, and the tier's 10m/5m
interval/timeout are called out there as wrong for a 3-node DaemonSet rollout —
which is exactly what this is. Worse here than average: platform gates
platform-config gates apps-wave-0 (wait:true) gates apps-wave-1, so a plugin
that never went healthy inside it would stall every app — a component added to
fix a silent no-op would have gained the power to cause a real outage.
Touching clusters/** costs nothing extra: required_approvals:1 with
block_admin_merge_override:true applies to every PR here regardless of path.
wait:true is a health SIGNAL, not a gate — nothing in the rendered graph
dependsOn this, so it gates nothing, while making `flux get kustomization
intel-gpu-plugin` answer "did it come up on all three nodes".
NON-CHOICES, each argued in the README: standalone DaemonSet over the operator
(no CRD, controller-manager, webhook or second namespace to keep healthy);
`--shared-dev-num` left at the upstream default of 1 (a higher count is a lie
the scheduler believes — no GPU-side partitioning backs it, so it converts a
loud Unschedulable into a quiet "transcodes are slow"); `--health-management`
left OFF, the one flag that would open a device and reintroduce this EPERM; no
NFD overlay (all three nodes identical); no supplementalGroups (the pms-docker
cont-init hook already adds `plex` to each device's group, measured live as
Groups: 44 100 992 1000).
The DaemonSet is vendored as `kubectl kustomize` output of upstream v0.37.0's
gpu_plugin/base, rather than referenced as a remote kustomize base, for the
reason external-snapshotter records: a remote base is git-fetched on every
reconcile and those fetches have flaked a whole tier. It is generated rather
than byte-copied because upstream mixes YAML sequence-indentation styles within
the one document and ci/yamllint.yaml sets `indent-sequences: consistent`, so a
verbatim copy is a hard lint ERROR — measured, not guessed: the first gate run
of this branch returned rc 1 BLOCKED on exactly that. The transform is proven
formatting-only (`yq -P 'sort_keys(..)'` of both diffs clean). Image pinned to
docker.io/intel/intel-gpu-plugin:0.37.0 (digest verified present:
sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1).
ORDERING — DO NOT REORDER. catalog/render.py, catalog/plex.yaml and
apps/wave-0/plex.yaml are untouched. The renderer change that emits
`gpu.intel.com/i915: 1` instead of the hostPath must NOT land until this plugin
is merged AND confirmed advertising the resource on all three nodes: a request
for a resource no node advertises makes the Plex pod Unschedulable, and
apps-wave-0 is wait:true with apps-wave-1 depending on it.
VERIFIED: manifests render offline (component, clusters/base, both overlays);
no node currently advertises gpu.intel.com/* (queried — "NONE"); the plugin
only lists/stats devices and never opens one (no os.Open/OpenFile/ioctl/syscall
anywhere in its 908-line source); all four app-scoped Kyverno policies are
gated on app-namespace=true and the new namespace is labelled "false".
NOT VERIFIED: anything runtime. The plugin has never executed, the resource has
never been advertised, and the open() test has not been re-run. Gate returned
rc 3 (UNDETERMINED) with exactly the three expected skips — not green.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
cdf3e14e27
commit
4e1b169256
7 files changed
+786
No files matched your search
@@ -0,0 +1,225 @@
|
||||
# platform/intel-gpu-plugin
|
||||
|
||||
**Intel Device Plugin for Kubernetes (GPU)** — the standalone GPU plugin DaemonSet from
|
||||
[`intel/intel-device-plugins-for-kubernetes`](https://github.com/intel/intel-device-plugins-for-kubernetes),
|
||||
pinned at **v0.37.0**. It makes every node advertise **`gpu.intel.com/i915`** so a workload can
|
||||
*request* a GPU instead of hoping a `hostPath` is enough.
|
||||
|
||||
- **Serves:** hardware transcoding for Plex (`hardware.gpu: intel-quicksync`), and any later
|
||||
consumer of an Intel iGPU.
|
||||
- **Tier:** its **own** Flux Kustomization
|
||||
([`clusters/base/intel-gpu-plugin.yaml`](../../clusters/base/intel-gpu-plugin.yaml),
|
||||
`dependsOn: cilium`, `wait: true`) — **not** a line in `platform/kustomization.yaml`. See
|
||||
*Placement* below.
|
||||
- **Namespace:** `intel-gpu-plugin` — a PLATFORM namespace, labelled
|
||||
`homelab.keithjackson.dev/app-namespace: "false"`, so the app-scoped Kyverno policies
|
||||
(default-deny generation, auth-mode, egress-class, host-namespaces) do not apply.
|
||||
- **No RBAC, no ServiceAccount, no CRDs, no webhook, no API-server access.** A device plugin talks
|
||||
to the *local kubelet* over a unix socket.
|
||||
|
||||
## What this component owns
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `namespace.yaml` | `intel-gpu-plugin` ns, explicitly `app-namespace=false` |
|
||||
| `intel-gpu-plugin.yaml` | the DaemonSet, **vendored** (machine-generated) from upstream v0.37.0 |
|
||||
| `kustomization.yaml` | namespace scoping + the pinned image (`images:` transformer) |
|
||||
|
||||
## The problem this fixes
|
||||
|
||||
Plex was given a `/dev/dri` hostPath (merged on `main` @ `cdf3e14`). The device is **visible** in
|
||||
the container and **cannot be opened**. Measured in the running pod on `k3s-prod-2`:
|
||||
|
||||
```
|
||||
uid 0, O_RDWR -> EPERM "Operation not permitted"
|
||||
uid 1000, O_RDWR -> EPERM "Operation not permitted"
|
||||
```
|
||||
|
||||
`EPERM` from the **device cgroup**, not `EACCES` from file permissions — proven three ways:
|
||||
|
||||
1. root on the **host** opens the same device with `rc=0`;
|
||||
2. the node is unified `cgroup2fs`;
|
||||
3. the container's runc `config.json` reads
|
||||
`linux.resources.devices: [{'allow': False, 'access': 'rwm'}]` and `linux.devices: None` —
|
||||
one deny-all rule, no allow entry for major 226, and the runtime was never asked to grant a
|
||||
device.
|
||||
|
||||
A hostPath is a **filesystem** operation. It makes the device *node* appear and adds no cgroup
|
||||
allow rule, because Kubernetes cannot infer one from a volume.
|
||||
|
||||
**A device plugin is the mechanism that closes exactly this gap.** Verified in the v0.37.0 source
|
||||
(`cmd/gpu_plugin/gpu_plugin.go`, `createDeviceSpecsFromDrmFiles`), the `Allocate` response carries
|
||||
per DRM node:
|
||||
|
||||
```go
|
||||
specs = append(specs, pluginapi.DeviceSpec{
|
||||
HostPath: devPath, ContainerPath: devPath, Permissions: "rw",
|
||||
})
|
||||
```
|
||||
|
||||
and it is that `DeviceSpec` which makes kubelet populate `linux.devices` **and** the matching
|
||||
cgroup allow rule. It is the only **non-privileged** way to get there. `privileged: true` would
|
||||
also work and is not available: `platform/kyverno/policies/restrict-host-namespaces.yaml` blocks it
|
||||
under Enforce in app namespaces, and `plex` is labelled `app-namespace=true` (measured
|
||||
2026-10-01). Correctly so.
|
||||
|
||||
## Placement — why its own Kustomization
|
||||
|
||||
The cheap option was one line in the shared `platform` tier. Neither that file nor a new
|
||||
`platform/<name>/` directory is a sensitive path in `infra/gitea/approval-policy.json`, so it would
|
||||
have needed no human reviewer *on the policy's terms*. It is still the wrong place, and the repo
|
||||
already argued the case when Cilium was split out (read the comment in
|
||||
`platform/kustomization.yaml`):
|
||||
|
||||
* `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot distinguish
|
||||
"the GPU plugin is wedged" from "longhorn is slow".
|
||||
* The only emergency valve would be `flux suspend kustomization platform` — freezing storage, TLS,
|
||||
DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device plugin.
|
||||
* The tier's 10m/5m interval/timeout are called out there as **wrong for a 3-node DaemonSet
|
||||
rollout**. This is a 3-node DaemonSet rollout.
|
||||
|
||||
The consequence is worse than average here because of what sits downstream: `platform` gates
|
||||
`platform-config`, which gates `apps-wave-0` (`wait: true`), which gates `apps-wave-1`. A plugin
|
||||
that never went healthy inside that tier would stall **every app** on "dependency not ready" — so a
|
||||
component added to fix a *silent no-op* (software transcoding) would have acquired the ability to
|
||||
cause a *real outage*.
|
||||
|
||||
The usual objection — that `clusters/**` needs a human approval — costs nothing: Gitea enforces
|
||||
`required_approvals: 1` with `block_admin_merge_override: true` on **every** PR here regardless of
|
||||
path, and authors cannot self-approve. The approval was required either way.
|
||||
|
||||
### `wait: true` is a health signal, not a gate
|
||||
|
||||
This departs from the `headlamp`/`crafty` leaf template, deliberately. Those run `wait: false`
|
||||
because a leaf UI can be legitimately not-Ready for a long time during a cold start while it waits
|
||||
on Keycloak/DNS/TLS, making its health useless as a signal. This DaemonSet has no such dependency —
|
||||
it reads sysfs and opens a socket, so on healthy nodes it is Ready in seconds and **not-Ready
|
||||
genuinely means broken**. `wait: true` makes `flux get kustomization intel-gpu-plugin` a one-line
|
||||
answer to "did it come up on all three nodes", which is precisely the question that must be
|
||||
answered before the consumer PR may merge.
|
||||
|
||||
Nothing in the graph `dependsOn` this, so `wait: true` gates nothing. **Think hard before adding
|
||||
`dependsOn: [intel-gpu-plugin]`** anywhere — for the app waves that would hand a device plugin the
|
||||
power to stall every app, which is the thing this placement exists to prevent.
|
||||
|
||||
## Deliberate non-choices
|
||||
|
||||
**Standalone DaemonSet, not the operator.** The operator adds a CRD (`GpuDevicePlugin`), a
|
||||
controller-manager Deployment, cert-manager-backed admission webhooks and a second namespace —
|
||||
three more things that can be unhealthy — to buy declarative management of a DaemonSet that is
|
||||
already declared in git. For three identical nodes and one consumer that is strictly more blast
|
||||
radius for no gain.
|
||||
|
||||
**It lists devices; it never opens them** — worth being certain about, since *opening* is exactly
|
||||
what fails today. Verified in v0.37.0 source: the scan path uses only `os.ReadDir`, `os.ReadFile`,
|
||||
`os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. There is no `os.Open`, no
|
||||
`OpenFile`, no `ioctl` and no raw syscall against a DRM node anywhere in
|
||||
`cmd/gpu_plugin/gpu_plugin.go`. Listing and `stat()` demonstrably work today — that is how the Plex
|
||||
probe found the device before failing to open it — so the plugin needs no cgroup grant of its own
|
||||
and no privilege. Accordingly `/dev/dri` and `/sys/class/drm` are mounted **readOnly**.
|
||||
|
||||
* **`--health-management` stays OFF** (upstream default). It is the one flag that *would* need to
|
||||
open a device: it starts a Level-Zero sidecar (`setupLevelZeroService` returns early unless the
|
||||
flag is set). Enabling it would reintroduce the exact `EPERM` this component exists to fix.
|
||||
* `/dev/dri/by-path` being absent is non-fatal — `newDevicePlugin` warns and sets
|
||||
`bypathFound = false`.
|
||||
|
||||
**`--shared-dev-num` left at the upstream default of 1** (i.e. not set). The flag multiplies each
|
||||
card into N schedulable device IDs, letting N containers share one GPU. Not raised, because:
|
||||
|
||||
* Plex is the only consumer today and wants one device;
|
||||
* the count is a **lie the scheduler believes** — there is no GPU-side partitioning or quota behind
|
||||
it, so N containers get the same `renderD128` and contend for the same two media engines
|
||||
(`vcs0`/`vecs0`). Raising it converts "Unschedulable", which is loud and obvious, into
|
||||
"transcodes are mysteriously slow", which is neither;
|
||||
* it is a one-line, zero-migration change on the day a second consumer exists — and *then* the
|
||||
right number will be known rather than guessed.
|
||||
|
||||
**No NFD overlay.** All three nodes are identical (HP EliteDesk 800 G5 Mini, i7-9700T, UHD Graphics
|
||||
630, PCI `8086:3e98`, `i915` bound), so a nodeSelector on a GPU feature label would gate on
|
||||
something true everywhere. If a GPU-less node ever joins, add upstream's `nfd` overlay *then*
|
||||
rather than relying on the plugin's behaviour on such a node.
|
||||
|
||||
**No `supplementalGroups`, anywhere.** The `pms-docker` image's own cont-init hook stats each
|
||||
`/dev/dri` char device and adds the `plex` user to its group — measured live as
|
||||
`Groups: 44 100 992 1000` on the running Plex process, where GID 992 is the host `render` group.
|
||||
That mechanism is image-side and works identically on a plugin-injected device.
|
||||
|
||||
## Resource name
|
||||
|
||||
`gpu.intel.com/i915`. Not a guess: `scan()` calls `devTree.AddDevice(devProps.driver(), ...)`, so
|
||||
the suffix is the **kernel driver name**. These nodes have `i915` bound (not `xe`), so the resource
|
||||
is `i915`. A future kernel moving CoffeeLake to `xe` would silently rename it to `gpu.intel.com/xe`
|
||||
and a consumer's request would stop matching — worth knowing before that kernel arrives.
|
||||
|
||||
## Versioning
|
||||
|
||||
Pinned to **v0.37.0** via the `images:` transformer in `kustomization.yaml`; the DaemonSet itself is
|
||||
vendored as `kubectl kustomize` output of the upstream base (see its header for the exact re-vendor
|
||||
command and both sha256s). It is *generated* rather than a byte copy because upstream mixes YAML
|
||||
sequence-indentation styles within the one document and `ci/yamllint.yaml` sets
|
||||
`indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate
|
||||
(measured: it did). `kubectl kustomize` normalises the formatting, and the result was proven to be
|
||||
a formatting-only change: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file
|
||||
diff clean. This is also the re-vendor idiom `platform/external-snapshotter` documents.
|
||||
|
||||
Vendored rather than referenced as a remote kustomize base for the same reason as
|
||||
`platform/external-snapshotter`: a remote base is `git fetch`ed on **every** reconcile, those
|
||||
fetches intermittently exceeded kustomize-controller's 27s timeout, and transient GitHub slowness
|
||||
then rendered as "kustomize build failed". A local file makes the build offline and deterministic —
|
||||
which also means the CI gate's `kustomize build` proves the manifests rather than proving GitHub
|
||||
was up.
|
||||
|
||||
Renovate tracks it through the inline-marker idiom (`# renovate: datasource=github-releases
|
||||
depName=...` immediately above the version literal), because `renovate.json` sets
|
||||
`"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}` — an image reference in a
|
||||
plain manifest is tracked by nothing. **Caveat:** upstream *release* tags carry a `v` (`v0.37.0`)
|
||||
while *image* tags do not (`0.37.0`). Whether Renovate normalises that for this dep is unverified;
|
||||
if it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
|
||||
`renovate.json`, not a hand-edit.
|
||||
|
||||
## Ordering — do not reorder the stack
|
||||
|
||||
The renderer change (`catalog/render.py` emitting `gpu.intel.com/i915: 1` for
|
||||
`hardware.gpu: intel-quicksync` instead of the hostPath) **must not land until this plugin is
|
||||
merged AND confirmed advertising the resource on all three nodes.** If the renderer requests a
|
||||
resource no node advertises, the Plex pod is **Unschedulable**; `apps-wave-0` is `wait: true` and
|
||||
`apps-wave-1` `dependsOn` it, so that stalls the whole app tier — turning today's silent no-op into
|
||||
a real outage.
|
||||
|
||||
## Post-merge verification
|
||||
|
||||
Nothing about this component's runtime behaviour can be verified from a PR: it does not exist until
|
||||
the operator merges and Flux reconciles. Afterwards:
|
||||
|
||||
```bash
|
||||
D=./infra/devcontainer/dev.sh
|
||||
|
||||
# 1. Flux applied it, and (wait:true) the DaemonSet went healthy.
|
||||
$D run bash -c 'flux get kustomization intel-gpu-plugin'
|
||||
$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin get ds intel-gpu-plugin'
|
||||
# want: DESIRED=3 CURRENT=3 READY=3 AVAILABLE=3
|
||||
|
||||
# 2. The resource is advertised on ALL THREE nodes. This is the gate on the renderer PR.
|
||||
$D run bash -c 'kubectl --context k3s-prod get nodes -o json \
|
||||
| jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
|
||||
# want: three lines, each i915=1 (before this change: ABSENT on all three)
|
||||
|
||||
# 3. If a node shows ABSENT, read that node's plugin logs.
|
||||
$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin logs -l app=intel-gpu-plugin --tail=50 --prefix'
|
||||
```
|
||||
|
||||
### The decisive test
|
||||
|
||||
Resource-advertised is necessary but **not** sufficient — it proves the plugin registered, not that
|
||||
the cgroup allow rule reaches Plex. The decisive test is re-running the `open()` probe **as uid 1000
|
||||
through `s6-setuidgid plex`** inside the Plex pod, *after* the renderer PR has switched Plex to
|
||||
request the resource:
|
||||
|
||||
```bash
|
||||
$D run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
|
||||
s6-setuidgid plex python3 -c "import os; os.close(os.open(\"/dev/dri/renderD128\", os.O_RDWR)); print(\"rc=0 OK\")"'
|
||||
```
|
||||
|
||||
**Passing as uid 0 is weaker evidence and must not be accepted as proof** — Plex transcodes as uid
|
||||
1000, and the two differ in group membership, which is the half the image's cont-init hook supplies.
|
||||
Reference in new issue
Block a user