From 4e1b169256b70f1ffbb39f4e252b1b7caccd285f Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Thu, 1 Oct 2026 22:17:59 -0500 Subject: [PATCH] feat(gpu): Intel GPU device plugin v0.37.0, in its own Flux Kustomization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Makes the three prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the Plex hardware-transcoding blocker, and it only makes the resource EXIST — no consumer is switched over here. WHY. The `/dev/dri` hostPath merged on main @ cdf3e14 makes the device node visible and not openable. Measured in the running pod on k3s-prod-2: uid 0, O_RDWR -> EPERM "Operation not permitted" uid 1000, O_RDWR -> EPERM "Operation not permitted" EPERM from the device cgroup, not EACCES from file permissions — root on the HOST opens the same device rc=0, the node is unified cgroup2fs, and the container's runc config.json carries one deny-all rule with `linux.devices: None`. A hostPath is a filesystem operation: it makes the device node appear and adds no cgroup allow rule, because Kubernetes cannot infer one from a volume. A device plugin's Allocate response is what makes kubelet populate `linux.devices` AND the matching cgroup allow rule — verified in the v0.37.0 source, which emits `DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node. It is the only non-privileged way to get there; `privileged: true` is blocked by Kyverno restrict-host-namespaces under Enforce in app namespaces, and `plex` is labelled app-namespace=true. PLACEMENT: its own Kustomization (dependsOn cilium, wait:true), not a line in platform/kustomization.yaml. The shared tier was cheaper and needed no human reviewer on approval-policy.json's terms, but it is wait:true over eleven-plus components and the repo already argued this when Cilium was split out: its Ready condition cannot tell "the plugin is wedged" from "longhorn is slow", the only emergency valve is freezing the whole tier, and the tier's 10m/5m interval/timeout are called out there as wrong for a 3-node DaemonSet rollout — which is exactly what this is. Worse here than average: platform gates platform-config gates apps-wave-0 (wait:true) gates apps-wave-1, so a plugin that never went healthy inside it would stall every app — a component added to fix a silent no-op would have gained the power to cause a real outage. Touching clusters/** costs nothing extra: required_approvals:1 with block_admin_merge_override:true applies to every PR here regardless of path. wait:true is a health SIGNAL, not a gate — nothing in the rendered graph dependsOn this, so it gates nothing, while making `flux get kustomization intel-gpu-plugin` answer "did it come up on all three nodes". NON-CHOICES, each argued in the README: standalone DaemonSet over the operator (no CRD, controller-manager, webhook or second namespace to keep healthy); `--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler believes — no GPU-side partitioning backs it, so it converts a loud Unschedulable into a quiet "transcodes are slow"); `--health-management` left OFF, the one flag that would open a device and reintroduce this EPERM; no NFD overlay (all three nodes identical); no supplementalGroups (the pms-docker cont-init hook already adds `plex` to each device's group, measured live as Groups: 44 100 992 1000). The DaemonSet is vendored as `kubectl kustomize` output of upstream v0.37.0's gpu_plugin/base, rather than referenced as a remote kustomize base, for the reason external-snapshotter records: a remote base is git-fetched on every reconcile and those fetches have flaked a whole tier. It is generated rather than byte-copied because upstream mixes YAML sequence-indentation styles within the one document and ci/yamllint.yaml sets `indent-sequences: consistent`, so a verbatim copy is a hard lint ERROR — measured, not guessed: the first gate run of this branch returned rc 1 BLOCKED on exactly that. The transform is proven formatting-only (`yq -P 'sort_keys(..)'` of both diffs clean). Image pinned to docker.io/intel/intel-gpu-plugin:0.37.0 (digest verified present: sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1). ORDERING — DO NOT REORDER. catalog/render.py, catalog/plex.yaml and apps/wave-0/plex.yaml are untouched. The renderer change that emits `gpu.intel.com/i915: 1` instead of the hostPath must NOT land until this plugin is merged AND confirmed advertising the resource on all three nodes: a request for a resource no node advertises makes the Plex pod Unschedulable, and apps-wave-0 is wait:true with apps-wave-1 depending on it. VERIFIED: manifests render offline (component, clusters/base, both overlays); no node currently advertises gpu.intel.com/* (queried — "NONE"); the plugin only lists/stats devices and never opens one (no os.Open/OpenFile/ioctl/syscall anywhere in its 908-line source); all four app-scoped Kyverno policies are gated on app-namespace=true and the new namespace is labelled "false". NOT VERIFIED: anything runtime. The plugin has never executed, the resource has never been advertised, and the open() test has not been re-run. Gate returned rc 3 (UNDETERMINED) with exactly the three expected skips — not green. Co-Authored-By: Claude Opus 5 (1M context) --- clusters/base/intel-gpu-plugin.yaml | 104 ++++++++ clusters/base/kustomization.yaml | 9 + .../2026-10-01-intel-gpu-device-plugin.md | 198 +++++++++++++++ platform/intel-gpu-plugin/README.md | 225 ++++++++++++++++++ .../intel-gpu-plugin/intel-gpu-plugin.yaml | 117 +++++++++ platform/intel-gpu-plugin/kustomization.yaml | 99 ++++++++ platform/intel-gpu-plugin/namespace.yaml | 34 +++ 7 files changed, 786 insertions(+) create mode 100644 clusters/base/intel-gpu-plugin.yaml create mode 100644 decisions/2026-10-01-intel-gpu-device-plugin.md create mode 100644 platform/intel-gpu-plugin/README.md create mode 100644 platform/intel-gpu-plugin/intel-gpu-plugin.yaml create mode 100644 platform/intel-gpu-plugin/kustomization.yaml create mode 100644 platform/intel-gpu-plugin/namespace.yaml diff --git a/clusters/base/intel-gpu-plugin.yaml b/clusters/base/intel-gpu-plugin.yaml new file mode 100644 index 0000000..64b5bc6 --- /dev/null +++ b/clusters/base/intel-gpu-plugin.yaml @@ -0,0 +1,104 @@ +--- +# Flux Kustomization: Intel GPU device plugin — reconciles ./platform/intel-gpu-plugin. +# Makes the three prod nodes advertise `gpu.intel.com/i915`, which is the resource a later PR will +# have `hardware.gpu: intel-quicksync` request in place of today's non-working /dev/dri hostPath. +# +# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml +# +# The cheap option was a line in the shared `platform` tier plus a new platform/ directory, and +# neither path is sensitive in infra/gitea/approval-policy.json, so it would have needed no human +# reviewer on the policy's terms. It is still the wrong place, for the reason the repo already +# wrote down in platform/kustomization.yaml when Cilium was split out of that tier: +# +# * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot tell +# you "the GPU plugin is wedged" from "longhorn is slow". +# * The only emergency valve would be `flux suspend kustomization platform` — freezing storage, +# TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device +# plugin. +# * The tier's 10m interval / 5m timeout are explicitly called out there as WRONG for a 3-node +# DaemonSet rollout. This is a 3-node DaemonSet rollout. +# +# And the consequence is worse here than for an average component, because of what sits downstream: +# `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates apps-wave-1. +# A plugin DaemonSet that never went healthy inside that tier would stall every app on "dependency +# not ready" — i.e. a component added to fix a SILENT NO-OP (software transcoding) would have +# acquired the ability to cause a REAL OUTAGE. That trade is the whole argument. +# +# The usual objection to this placement is the approval cost, and it does not apply: Gitea enforces +# `required_approvals: 1` with `block_admin_merge_override: true` on EVERY PR here regardless of +# path, and authors cannot self-approve. So this change needed the operator's approval either way; +# touching clusters/** costs nothing extra and buys a blast radius of exactly one component. +# +# dependsOn: [cilium], WHICH IS THE MINIMUM AND ALSO THE EARLIEST +# +# Not platform-config (what headlamp/crafty/homepage/oauth2-proxy use) — those are leaf UIs that +# genuinely need Keycloak + wildcard TLS + split-horizon DNS + Gateway. This plugin needs none of +# it. It never uses the network and never talks to the API server; it talks to the local kubelet +# over a unix socket. Its only real prerequisite is a working CNI, so its pod can get a sandbox at +# all, and `cilium` already depends on `flux-system` (the root Kustomization that applies the +# cluster-vars ConfigMap this one substitutes from), so that edge is transitively satisfied too. +# +# Depending on `cilium` rather than `platform` also gets the resource advertised EARLIEST — this +# reconciles in parallel with the `platform` tier instead of behind all eleven of its components. +# That matters on a cold start once the consumer PR lands: a Plex pod requesting a resource no node +# advertises is Unschedulable, and apps-wave-0 is wait:true. Reconciling early shrinks that window +# instead of widening it. It is not a GUARANTEE — see the ordering note below, which is a real +# constraint on the next PR and not a formality. +# +# Kyverno (in `platform`) is deliberately NOT a dependency: every policy that could match this +# workload is scoped to namespaces labelled app-namespace=true, and this namespace is labelled +# "false" (platform/intel-gpu-plugin/namespace.yaml). So there is no admission race to lose. +# +# wait: true — A HEALTH SIGNAL, NOT A GATE. This is the one place this component departs from the +# headlamp/crafty leaf template, and the departure is deliberate. Those run wait:false because a +# leaf UI can be legitimately not-Ready for a long time during a cold start while it waits on +# Keycloak/DNS/TLS, so its health is not a useful signal. This DaemonSet has no such dependency: it +# reads sysfs and opens a socket, so on healthy nodes it should be Ready in seconds, and +# not-Ready genuinely means BROKEN. wait:true turns `flux get kustomization intel-gpu-plugin` into +# a one-line answer to "did the plugin come up on all three nodes", which is exactly the question +# the operator must answer before the consumer PR may merge. +# +# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy +# plugin is this Kustomization's own Ready condition and nothing else. Anyone later adding +# `dependsOn: [intel-gpu-plugin]` is choosing to make plugin health gate their thing; for the app +# waves that would hand a device plugin the power to stall every app, which is the thing this file +# exists to prevent. Think before adding that edge. +# +# ORDERING — DO NOT REORDER THE STACK. The renderer change (catalog/render.py emitting +# `gpu.intel.com/i915: 1` for `hardware.gpu: intel-quicksync` instead of the hostPath) must NOT +# land until this plugin is merged AND confirmed advertising the resource on all three nodes. If +# the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is +# wait:true and apps-wave-1 dependsOn it, so that stalls the whole app tier — turning today's silent +# no-op into a real outage. This PR's only job is to make the resource exist. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: intel-gpu-plugin + namespace: flux-system +spec: + # Static vendored manifests with nothing to re-render; kustomize-controller is source-triggered, + # so this is only the drift-correction floor. + interval: 10m + retryInterval: 2m + # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can + # exceed 5m on a cold start (image pull on three nodes), and the Cilium split-out comment calls + # out that exact mismatch as one of its reasons for leaving the shared tier. + timeout: 10m + prune: true + # See the header: a health SIGNAL for a component nothing depends on, not a gate. + wait: true + dependsOn: + - name: cilium # the CNI — the only real prerequisite; transitively gives us flux-system too + sourceRef: + kind: GitRepository + name: flux-system + path: ./platform/intel-gpu-plugin + # Retained for parity with every other tier / future overlay knobs. This component references no + # ${VARS} today, and its manifests contain no `$` literals for envsubst to eat (unlike + # clusters/base/runbooks.yaml, which omits substitution for exactly that reason). + postBuild: + substituteFrom: + - kind: ConfigMap + name: cluster-vars + # No SOPS decryption patch: this component carries no Secret, so clusters/dev adds nothing for it + # (same as homepage/runbooks, unlike headlamp/crafty/oauth2-proxy/backup). diff --git a/clusters/base/kustomization.yaml b/clusters/base/kustomization.yaml index 66c42a1..025ddac 100644 --- a/clusters/base/kustomization.yaml +++ b/clusters/base/kustomization.yaml @@ -12,6 +12,15 @@ resources: # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not # freeze the other eleven platform components. See clusters/base/cilium.yaml. - cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true) + # Intel GPU device plugin — its OWN Kustomization (dependsOn cilium, wait:true), NOT a line in + # platform/kustomization.yaml. Same reasoning as the Cilium split-out, applied to a leaf: inside + # that wait:true tier a wedged 3-node DaemonSet would be indistinguishable from eleven unrelated + # components AND would stall platform-config + every apps wave on "dependency not ready" — so a + # component added to fix a silent no-op (Plex software transcoding) could cause a real outage. + # wait:true here is a HEALTH SIGNAL, not a gate: nothing in the graph dependsOn it. + # It advertises `gpu.intel.com/i915`; the renderer change that REQUESTS that resource must land + # strictly later. See clusters/base/intel-gpu-plugin.yaml. + - intel-gpu-plugin.yaml # Flux Kustomization -> ./platform/intel-gpu-plugin (dependsOn cilium) - platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs) - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform) # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any diff --git a/decisions/2026-10-01-intel-gpu-device-plugin.md b/decisions/2026-10-01-intel-gpu-device-plugin.md new file mode 100644 index 0000000..44dd3d5 --- /dev/null +++ b/decisions/2026-10-01-intel-gpu-device-plugin.md @@ -0,0 +1,198 @@ +--- +date: 2026-10-01 +branch: feat/intel-gpu-device-plugin +title: Intel GPU device plugin, in its own Kustomization — the resource exists; nothing has run yet +--- + +## What changed + +Deployed the **Intel Device Plugin for Kubernetes (GPU)**, standalone DaemonSet, pinned at +**v0.37.0**, so the prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the +Plex hardware-transcoding blocker: the `/dev/dri` hostPath merged on `main` @ `cdf3e14` makes the +device node *visible* and not *openable* (`EPERM` from the device cgroup at both uid 0 and uid +1000; the container's runc `config.json` carries one deny-all rule and `linux.devices: None`). A +hostPath is a filesystem operation and adds no cgroup allow rule. A device plugin's `Allocate` +response is what makes kubelet populate `linux.devices` *and* the matching cgroup allow rule, and +it is the only non-privileged way to get there — `privileged: true` is blocked by Kyverno +`restrict-host-namespaces` under Enforce in app namespaces, and `plex` is labelled +`app-namespace=true`. + +New files: + +- `platform/intel-gpu-plugin/intel-gpu-plugin.yaml` — the DaemonSet, **vendored** as + `kubectl kustomize` output of upstream `deployments/gpu_plugin/base` at `v0.37.0`, under a + provenance header carrying the exact re-vendor command and both sha256s (generated body + `80ebd070…`, raw upstream source `c8773918…`). Generated rather than byte-copied because + upstream mixes YAML sequence-indentation styles within the one document and `ci/yamllint.yaml` + sets `indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate. + That is not a guess: the first gate run of this branch returned **rc 1 BLOCKED** on exactly that + error, and regenerating through kustomize is the fix (and is also the re-vendor idiom + `platform/external-snapshotter` documents). Vendored at all, rather than referenced as a remote + kustomize base, for the other reason `external-snapshotter` records: a remote base is + `git fetch`ed on every reconcile and those fetches have flaked a whole tier. +- `platform/intel-gpu-plugin/namespace.yaml` — a PLATFORM namespace, explicitly + `homelab.keithjackson.dev/app-namespace: "false"` (following `vpn-egress`/`kyverno`). +- `platform/intel-gpu-plugin/kustomization.yaml` — namespace scoping plus the image pin. +- `platform/intel-gpu-plugin/README.md` — the measurement, the placement argument, the + non-choices, and the post-merge verification sequence. +- `clusters/base/intel-gpu-plugin.yaml` — **its own Flux Kustomization**: `dependsOn: [cilium]`, + `wait: true`, `interval: 10m`, `timeout: 10m`, `prune: true`. +- One wired line (plus its rationale comment) in `clusters/base/kustomization.yaml`. + +**Placement: its own Kustomization, not a line in the shared `platform` tier.** The shared tier was +cheaper and needed no human reviewer on `approval-policy.json`'s terms, but `platform` is +`wait: true` over eleven-plus components and the repo already argued this case when Cilium was +split out: its Ready condition cannot distinguish "the plugin is wedged" from "longhorn is slow", +the only emergency valve is freezing the entire tier, and the tier's 10m/5m interval/timeout are +called out there as *wrong for a 3-node DaemonSet rollout* — which is exactly what this is. The +consequence is worse than average here, because `platform` gates `platform-config` gates +`apps-wave-0` (`wait: true`) gates `apps-wave-1`: a plugin that never went healthy inside that tier +would stall every app, so a component added to fix a **silent no-op** would have gained the ability +to cause a **real outage**. The approval cost of touching `clusters/**` is nil, because +`required_approvals: 1` with `block_admin_merge_override: true` applies to every PR here regardless +of path and authors cannot self-approve. + +`dependsOn: [cilium]` is the minimum *and* the earliest: the plugin never uses the network and +never talks to the API server (it talks to the local kubelet over a unix socket), so a working CNI +is its only real prerequisite, and `cilium` already depends on `flux-system`, which applies the +`cluster-vars` ConfigMap this Kustomization substitutes from. Depending on `cilium` rather than +`platform` also means it reconciles *in parallel with* the platform tier rather than behind all +eleven of its components, which shrinks the cold-start window in which a node has not yet +advertised the resource. + +`wait: true` is a deliberate departure from the `headlamp`/`crafty` leaf template. Those use +`wait: false` because a leaf UI can be legitimately not-Ready for a long time during a cold start, +making its health a useless signal. This DaemonSet reads sysfs and opens a socket, so not-Ready +genuinely means broken — and since **nothing in the graph `dependsOn` it** (verified against the +rendered graph), `wait: true` gates nothing while turning +`flux get kustomization intel-gpu-plugin` into a one-line answer to the question that must be +answered before the consumer PR may merge. + +Deliberate non-choices, each argued in the README: standalone DaemonSet rather than the operator +(no CRD, no controller-manager, no cert-manager webhook, no second namespace to keep healthy); +`--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler +believes — there is no GPU-side partitioning behind it, so it converts a loud "Unschedulable" into +a quiet "transcodes are slow"); `--health-management` left OFF, because it is the one flag that +*would* open a device and would reintroduce the exact `EPERM` this fixes; no NFD overlay (all three +nodes are identical); no `supplementalGroups` anywhere, because the `pms-docker` cont-init hook +already adds the `plex` user to each device's group and that works identically on a +plugin-injected device. + +**Scope held deliberately:** `catalog/render.py`, `catalog/plex.yaml` and `apps/wave-0/plex.yaml` +are untouched. Switching Plex from the hostPath to a `gpu.intel.com/i915: 1` request is a strictly +later PR, and **it must not land until this plugin is merged AND confirmed advertising the resource +on all three nodes** — a request for a resource no node advertises makes the Plex pod +Unschedulable, and `apps-wave-0` is `wait: true` with `apps-wave-1` depending on it, so that would +stall the whole app tier and turn today's silent no-op into a real outage. + +## What I verified + +Mechanism, read from the v0.37.0 source rather than assumed: + +- **The plugin only lists devices; it never opens one.** The scan path uses `os.ReadDir`, + `os.ReadFile`, `os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. `grep` for + `os.Open|OpenFile|ioctl|syscall|unix\.` across `cmd/gpu_plugin/gpu_plugin.go` (908 lines) returns + **nothing**. Listing and `stat()` are what demonstrably work today, so the plugin needs no cgroup + grant and no extra privilege — it does not need more than the upstream manifest gives it. +- **The fix mechanism is real:** `createDeviceSpecsFromDrmFiles` emits + `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, which is the + thing kubelet turns into `linux.devices` plus a cgroup allow rule. +- **The resource name is `gpu.intel.com/i915` by derivation, not by hope:** `scan()` calls + `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the kernel driver name, and these + nodes have `i915` bound. +- `--shared-dev-num` default is `1` (`flag.IntVar(..., 1, ...)`); `--health-management` default is + `false` and `setupLevelZeroService` returns early unless it is set. +- Missing `/dev/dri/by-path` is non-fatal (`newDevicePlugin` warns, sets `bypathFound = false`). + +Image pin, verified against the registry: + +- `docker.io/intel/intel-gpu-plugin:0.37.0` exists; manifest digest + `sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1` (single amd64 + manifest, not an index), fetched from `registry-1.docker.io` on 2026-10-01. `v0.37.0` is the + latest upstream release (2026-09-16) and is not a prerelease. +- The Docker Hub tag list carries `0.35.0 / 0.36.0 / 0.37.0` — **image tags have no `v` prefix** + while GitHub release tags do. + +Policy scope, read from the policies rather than assumed: + +- All four app-scoped Kyverno policies (`restrict-host-namespaces`, `require-egress-class-label`, + `require-auth-mode-label`, `require-default-deny-baseline`) are gated on + `namespaceSelector`/`selector` `matchLabels: homelab.keithjackson.dev/app-namespace: "true"`. + The new namespace is labelled `"false"`, so none of them match it. +- Measured on prod: `plex` **is** `app-namespace=true`, confirming `privileged: true` is genuinely + unavailable there; `kyverno` and `vpn-egress` are the two namespaces carrying the label as + `"false"`; the other platform namespaces leave it unset. No `intel*` namespace exists, so no + collision. + +Cluster before-state (read-only, `kubectl` in the pinned toolbox with the prod kubeconfig +bind-mounted read-only — never copied into the worktree): + +- Three nodes `k3s-prod-{1,2,3}`, each `v1.36.4+k3s1`, Debian 13 trixie, `containerd://2.3.4-k3s1.36`. +- Every extended resource currently advertised is `devices.kubevirt.io/{kvm,sev,tun,vhost-net}` on + all three nodes. **No node advertises `gpu.intel.com/*`** — queried explicitly, returned + "NONE". That is the before-state this change is meant to move. +- Those `devices.kubevirt.io/*` resources are themselves advertised by a device plugin, which is + measured evidence that the device-plugin API and the default kubelet socket path + (`/var/lib/kubelet/device-plugins`) work on these nodes. + +Offline rendering (the main proof available to a PR): + +- `kubectl kustomize platform/intel-gpu-plugin` renders two objects: `Namespace/intel-gpu-plugin` + carrying `app-namespace=false`, and `DaemonSet/intel-gpu-plugin` in namespace + `intel-gpu-plugin`, with the image resolved to `docker.io/intel/intel-gpu-plugin:0.37.0`. +- `kubectl kustomize clusters/base` renders, and the new Flux Kustomization appears as + `intel-gpu-plugin ./platform/intel-gpu-plugin wait=true dependsOn=cilium`. Both overlays + (`clusters/prod`, `clusters/dev`) also render and carry it. +- Across the whole rendered graph, **no Kustomization lists `intel-gpu-plugin` in `dependsOn`** — + so `wait: true` gates nothing. +- The vendored DaemonSet is a **formatting-only** transform of upstream, proven rather than + asserted: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file `diff` + clean, so passing it through kustomize changed no content. The committed body's sha256 matches + the one recorded in its own provenance header (`80ebd070…`), so the file has not drifted from + what was generated. + +Gate (pinned toolbox, from this worktree, with the `ENGINE` shim bind-mounting the main checkout's +gitdir and `DEV_KUBECONFIG=/dev/null`): see the PR description for the per-stage table. **`rc 3` +(UNDETERMINED), with exactly the three expected skips** — `spec-immutability` (sibling spec repo is +not beside a worktree), `gitea-settings` (no CI token), `conformance` (no cluster in CI). Every +other stage passed. **rc 3 is not green** and is not reported as such. + +## What I did NOT verify + +**Nothing about this component's runtime behaviour. It has never run.** + +- **The plugin has never executed.** No pod has been scheduled, no image has been pulled, no scan + has happened. It cannot run until the operator merges this and Flux reconciles — `kubectl apply` + and `flux reconcile` are operator actions (CLAUDE.md §5), so this is expected, not an oversight. +- **The resource has never been advertised.** `gpu.intel.com/i915` does not exist on any node and + I have not seen it exist. The claim that it *will* be `i915` rather than `xe` is derived from + source plus the established fact that `i915` is bound — it is a derivation, not an observation. +- **The DaemonSet has never been admitted.** That the Kyverno policies do not match this namespace + is read off the policy manifests; no admission request has actually been evaluated. Nor has the + `kubectl kustomize` output ever been server-side dry-run against the API — rendering is not + admission. +- **The `open()` test has NOT been re-run.** The `EPERM` is exactly as it was. Nothing in this PR + touches Plex, so Plex is still software-transcoding and will continue to until the later renderer + PR lands. The decisive evidence is an `open()` as **uid 1000 through `s6-setuidgid plex`**; + passing as uid 0 is weaker and must not be accepted as proof. +- **A green gate proves the manifests render, nothing more.** And this gate was not green: it was + `rc 3`. Note also that the Gitea `gate` job runs `./ci/gate.sh || [ "$?" -eq 3 ]` + (`.gitea/workflows/gate.yml:138`), so a green tick on this PR cannot distinguish rc 0 from rc 3, + and `vars.CLUSTER_AVAILABLE` is unset so no L2/L3 conformance runs in CI. +- **`seLinuxOptions: type: container_device_plugin_t`** is carried verbatim from upstream. These + nodes are Debian 13 and I did not measure whether SELinux is enabled or whether that type exists + there. Expected to be inert (runc skips labelling when SELinux is disabled), but unverified — if + the pods fail to start with an SELinux error, that line is the first suspect. +- **Renovate has not been run** against this pin. The inline-marker idiom is copied from + `platform/kubevirt` / `platform/external-snapshotter`, but whether Renovate normalises the + `v`-prefix mismatch (release `v0.37.0` vs image `0.37.0`) for this dep is unknown. If it proposes + `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in `renovate.json`. +- **Resource limits unexamined under load.** The upstream 115Mi/100m limits are taken as-is; I have + no measurement of this plugin's footprint on these nodes. Worth a glance given + `keycloak-operator` is OOMing at 91% of its limit elsewhere in this cluster. +- **`clusters/dev` renders it, and there is no dev cluster** (the operator confirmed the + `k3s-dev-*` nodes are gone). That is inert rather than wrong — the unified graph (R-ENV-02) has + both overlays render the same base — but it is not a thing I could test. +- **Branch protection / required-checks state not re-measured** in this session. CLAUDE.md §2b + says re-measure before relying on it; I did not, so treat the approval/gating claims in §2 as + unconfirmed-by-me. diff --git a/platform/intel-gpu-plugin/README.md b/platform/intel-gpu-plugin/README.md new file mode 100644 index 0000000..7f1e7ff --- /dev/null +++ b/platform/intel-gpu-plugin/README.md @@ -0,0 +1,225 @@ +# platform/intel-gpu-plugin + +**Intel Device Plugin for Kubernetes (GPU)** — the standalone GPU plugin DaemonSet from +[`intel/intel-device-plugins-for-kubernetes`](https://github.com/intel/intel-device-plugins-for-kubernetes), +pinned at **v0.37.0**. It makes every node advertise **`gpu.intel.com/i915`** so a workload can +*request* a GPU instead of hoping a `hostPath` is enough. + +- **Serves:** hardware transcoding for Plex (`hardware.gpu: intel-quicksync`), and any later + consumer of an Intel iGPU. +- **Tier:** its **own** Flux Kustomization + ([`clusters/base/intel-gpu-plugin.yaml`](../../clusters/base/intel-gpu-plugin.yaml), + `dependsOn: cilium`, `wait: true`) — **not** a line in `platform/kustomization.yaml`. See + *Placement* below. +- **Namespace:** `intel-gpu-plugin` — a PLATFORM namespace, labelled + `homelab.keithjackson.dev/app-namespace: "false"`, so the app-scoped Kyverno policies + (default-deny generation, auth-mode, egress-class, host-namespaces) do not apply. +- **No RBAC, no ServiceAccount, no CRDs, no webhook, no API-server access.** A device plugin talks + to the *local kubelet* over a unix socket. + +## What this component owns + +| File | Purpose | +|---|---| +| `namespace.yaml` | `intel-gpu-plugin` ns, explicitly `app-namespace=false` | +| `intel-gpu-plugin.yaml` | the DaemonSet, **vendored** (machine-generated) from upstream v0.37.0 | +| `kustomization.yaml` | namespace scoping + the pinned image (`images:` transformer) | + +## The problem this fixes + +Plex was given a `/dev/dri` hostPath (merged on `main` @ `cdf3e14`). The device is **visible** in +the container and **cannot be opened**. Measured in the running pod on `k3s-prod-2`: + +``` +uid 0, O_RDWR -> EPERM "Operation not permitted" +uid 1000, O_RDWR -> EPERM "Operation not permitted" +``` + +`EPERM` from the **device cgroup**, not `EACCES` from file permissions — proven three ways: + +1. root on the **host** opens the same device with `rc=0`; +2. the node is unified `cgroup2fs`; +3. the container's runc `config.json` reads + `linux.resources.devices: [{'allow': False, 'access': 'rwm'}]` and `linux.devices: None` — + one deny-all rule, no allow entry for major 226, and the runtime was never asked to grant a + device. + +A hostPath is a **filesystem** operation. It makes the device *node* appear and adds no cgroup +allow rule, because Kubernetes cannot infer one from a volume. + +**A device plugin is the mechanism that closes exactly this gap.** Verified in the v0.37.0 source +(`cmd/gpu_plugin/gpu_plugin.go`, `createDeviceSpecsFromDrmFiles`), the `Allocate` response carries +per DRM node: + +```go +specs = append(specs, pluginapi.DeviceSpec{ + HostPath: devPath, ContainerPath: devPath, Permissions: "rw", +}) +``` + +and it is that `DeviceSpec` which makes kubelet populate `linux.devices` **and** the matching +cgroup allow rule. It is the only **non-privileged** way to get there. `privileged: true` would +also work and is not available: `platform/kyverno/policies/restrict-host-namespaces.yaml` blocks it +under Enforce in app namespaces, and `plex` is labelled `app-namespace=true` (measured +2026-10-01). Correctly so. + +## Placement — why its own Kustomization + +The cheap option was one line in the shared `platform` tier. Neither that file nor a new +`platform//` directory is a sensitive path in `infra/gitea/approval-policy.json`, so it would +have needed no human reviewer *on the policy's terms*. It is still the wrong place, and the repo +already argued the case when Cilium was split out (read the comment in +`platform/kustomization.yaml`): + +* `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot distinguish + "the GPU plugin is wedged" from "longhorn is slow". +* The only emergency valve would be `flux suspend kustomization platform` — freezing storage, TLS, + DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device plugin. +* The tier's 10m/5m interval/timeout are called out there as **wrong for a 3-node DaemonSet + rollout**. This is a 3-node DaemonSet rollout. + +The consequence is worse than average here because of what sits downstream: `platform` gates +`platform-config`, which gates `apps-wave-0` (`wait: true`), which gates `apps-wave-1`. A plugin +that never went healthy inside that tier would stall **every app** on "dependency not ready" — so a +component added to fix a *silent no-op* (software transcoding) would have acquired the ability to +cause a *real outage*. + +The usual objection — that `clusters/**` needs a human approval — costs nothing: Gitea enforces +`required_approvals: 1` with `block_admin_merge_override: true` on **every** PR here regardless of +path, and authors cannot self-approve. The approval was required either way. + +### `wait: true` is a health signal, not a gate + +This departs from the `headlamp`/`crafty` leaf template, deliberately. Those run `wait: false` +because a leaf UI can be legitimately not-Ready for a long time during a cold start while it waits +on Keycloak/DNS/TLS, making its health useless as a signal. This DaemonSet has no such dependency — +it reads sysfs and opens a socket, so on healthy nodes it is Ready in seconds and **not-Ready +genuinely means broken**. `wait: true` makes `flux get kustomization intel-gpu-plugin` a one-line +answer to "did it come up on all three nodes", which is precisely the question that must be +answered before the consumer PR may merge. + +Nothing in the graph `dependsOn` this, so `wait: true` gates nothing. **Think hard before adding +`dependsOn: [intel-gpu-plugin]`** anywhere — for the app waves that would hand a device plugin the +power to stall every app, which is the thing this placement exists to prevent. + +## Deliberate non-choices + +**Standalone DaemonSet, not the operator.** The operator adds a CRD (`GpuDevicePlugin`), a +controller-manager Deployment, cert-manager-backed admission webhooks and a second namespace — +three more things that can be unhealthy — to buy declarative management of a DaemonSet that is +already declared in git. For three identical nodes and one consumer that is strictly more blast +radius for no gain. + +**It lists devices; it never opens them** — worth being certain about, since *opening* is exactly +what fails today. Verified in v0.37.0 source: the scan path uses only `os.ReadDir`, `os.ReadFile`, +`os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. There is no `os.Open`, no +`OpenFile`, no `ioctl` and no raw syscall against a DRM node anywhere in +`cmd/gpu_plugin/gpu_plugin.go`. Listing and `stat()` demonstrably work today — that is how the Plex +probe found the device before failing to open it — so the plugin needs no cgroup grant of its own +and no privilege. Accordingly `/dev/dri` and `/sys/class/drm` are mounted **readOnly**. + +* **`--health-management` stays OFF** (upstream default). It is the one flag that *would* need to + open a device: it starts a Level-Zero sidecar (`setupLevelZeroService` returns early unless the + flag is set). Enabling it would reintroduce the exact `EPERM` this component exists to fix. +* `/dev/dri/by-path` being absent is non-fatal — `newDevicePlugin` warns and sets + `bypathFound = false`. + +**`--shared-dev-num` left at the upstream default of 1** (i.e. not set). The flag multiplies each +card into N schedulable device IDs, letting N containers share one GPU. Not raised, because: + +* Plex is the only consumer today and wants one device; +* the count is a **lie the scheduler believes** — there is no GPU-side partitioning or quota behind + it, so N containers get the same `renderD128` and contend for the same two media engines + (`vcs0`/`vecs0`). Raising it converts "Unschedulable", which is loud and obvious, into + "transcodes are mysteriously slow", which is neither; +* it is a one-line, zero-migration change on the day a second consumer exists — and *then* the + right number will be known rather than guessed. + +**No NFD overlay.** All three nodes are identical (HP EliteDesk 800 G5 Mini, i7-9700T, UHD Graphics +630, PCI `8086:3e98`, `i915` bound), so a nodeSelector on a GPU feature label would gate on +something true everywhere. If a GPU-less node ever joins, add upstream's `nfd` overlay *then* +rather than relying on the plugin's behaviour on such a node. + +**No `supplementalGroups`, anywhere.** The `pms-docker` image's own cont-init hook stats each +`/dev/dri` char device and adds the `plex` user to its group — measured live as +`Groups: 44 100 992 1000` on the running Plex process, where GID 992 is the host `render` group. +That mechanism is image-side and works identically on a plugin-injected device. + +## Resource name + +`gpu.intel.com/i915`. Not a guess: `scan()` calls `devTree.AddDevice(devProps.driver(), ...)`, so +the suffix is the **kernel driver name**. These nodes have `i915` bound (not `xe`), so the resource +is `i915`. A future kernel moving CoffeeLake to `xe` would silently rename it to `gpu.intel.com/xe` +and a consumer's request would stop matching — worth knowing before that kernel arrives. + +## Versioning + +Pinned to **v0.37.0** via the `images:` transformer in `kustomization.yaml`; the DaemonSet itself is +vendored as `kubectl kustomize` output of the upstream base (see its header for the exact re-vendor +command and both sha256s). It is *generated* rather than a byte copy because upstream mixes YAML +sequence-indentation styles within the one document and `ci/yamllint.yaml` sets +`indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate +(measured: it did). `kubectl kustomize` normalises the formatting, and the result was proven to be +a formatting-only change: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file +diff clean. This is also the re-vendor idiom `platform/external-snapshotter` documents. + +Vendored rather than referenced as a remote kustomize base for the same reason as +`platform/external-snapshotter`: a remote base is `git fetch`ed on **every** reconcile, those +fetches intermittently exceeded kustomize-controller's 27s timeout, and transient GitHub slowness +then rendered as "kustomize build failed". A local file makes the build offline and deterministic — +which also means the CI gate's `kustomize build` proves the manifests rather than proving GitHub +was up. + +Renovate tracks it through the inline-marker idiom (`# renovate: datasource=github-releases +depName=...` immediately above the version literal), because `renovate.json` sets +`"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}` — an image reference in a +plain manifest is tracked by nothing. **Caveat:** upstream *release* tags carry a `v` (`v0.37.0`) +while *image* tags do not (`0.37.0`). Whether Renovate normalises that for this dep is unverified; +if it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in +`renovate.json`, not a hand-edit. + +## Ordering — do not reorder the stack + +The renderer change (`catalog/render.py` emitting `gpu.intel.com/i915: 1` for +`hardware.gpu: intel-quicksync` instead of the hostPath) **must not land until this plugin is +merged AND confirmed advertising the resource on all three nodes.** If the renderer requests a +resource no node advertises, the Plex pod is **Unschedulable**; `apps-wave-0` is `wait: true` and +`apps-wave-1` `dependsOn` it, so that stalls the whole app tier — turning today's silent no-op into +a real outage. + +## Post-merge verification + +Nothing about this component's runtime behaviour can be verified from a PR: it does not exist until +the operator merges and Flux reconciles. Afterwards: + +```bash +D=./infra/devcontainer/dev.sh + +# 1. Flux applied it, and (wait:true) the DaemonSet went healthy. +$D run bash -c 'flux get kustomization intel-gpu-plugin' +$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin get ds intel-gpu-plugin' +# want: DESIRED=3 CURRENT=3 READY=3 AVAILABLE=3 + +# 2. The resource is advertised on ALL THREE nodes. This is the gate on the renderer PR. +$D run bash -c 'kubectl --context k3s-prod get nodes -o json \ + | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""' +# want: three lines, each i915=1 (before this change: ABSENT on all three) + +# 3. If a node shows ABSENT, read that node's plugin logs. +$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin logs -l app=intel-gpu-plugin --tail=50 --prefix' +``` + +### The decisive test + +Resource-advertised is necessary but **not** sufficient — it proves the plugin registered, not that +the cgroup allow rule reaches Plex. The decisive test is re-running the `open()` probe **as uid 1000 +through `s6-setuidgid plex`** inside the Plex pod, *after* the renderer PR has switched Plex to +request the resource: + +```bash +$D run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \ + s6-setuidgid plex python3 -c "import os; os.close(os.open(\"/dev/dri/renderD128\", os.O_RDWR)); print(\"rc=0 OK\")"' +``` + +**Passing as uid 0 is weaker evidence and must not be accepted as proof** — Plex transcodes as uid +1000, and the two differ in group membership, which is the half the image's cont-init hook supplies. diff --git a/platform/intel-gpu-plugin/intel-gpu-plugin.yaml b/platform/intel-gpu-plugin/intel-gpu-plugin.yaml new file mode 100644 index 0000000..ca0738b --- /dev/null +++ b/platform/intel-gpu-plugin/intel-gpu-plugin.yaml @@ -0,0 +1,117 @@ +--- +# VENDORED, MACHINE-GENERATED, from intel/intel-device-plugins-for-kubernetes @ v0.37.0: +# deployments/gpu_plugin/base +# +# RE-VENDOR on a version bump (and bump `newTag` in kustomization.yaml to match): +# kubectl kustomize \ +# "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/base?ref=vX.Y.Z" \ +# > platform/intel-gpu-plugin/intel-gpu-plugin.yaml +# # then re-add this header, and READ THE DIFF — upstream has changed the securityContext +# # and the volume set between minor releases before. +# +# Generated 2026-10-01 at v0.37.0; sha256 of the body below (header excluded): +# 80ebd070f983f3116f4004f719d9c872220997ec766b4ab7e7290585ac8ef842 +# The raw upstream source file reviewed to produce it +# (deployments/gpu_plugin/base/intel-gpu-plugin.yaml) was sha256 +# c8773918c2370e711d4ba84120a9971a290bf75ecf81dc520782bcccd7ef24b6 +# and the two were proven EQUIVALENT, not merely similar: `yq -P 'sort_keys(..)'` of each diffs +# clean, so passing it through kustomize changed formatting only and no content. +# +# WHY GENERATED RATHER THAN A BYTE COPY of the upstream file: upstream mixes YAML sequence +# indentation styles within the one document (`containers:` non-indented, `env:` indented), and +# ci/yamllint.yaml sets `indentation: indent-sequences: consistent`, so a verbatim copy is a hard +# lint ERROR and blocks the merge gate — measured, this file failed exactly that way first time +# round. `kubectl kustomize` normalises the output, which is also precisely the re-vendor idiom +# platform/external-snapshotter documents for the same reason. +# +# WHY VENDORED AT ALL rather than a remote kustomize base (`...gpu_plugin/base?ref=v0.37.0` left in +# `resources:`): the same reason platform/external-snapshotter is vendored. A remote base is +# `git fetch`ed by Flux's kustomize-controller on EVERY reconcile, those fetches intermittently +# exceeded the controller's 27s timeout, and a transient GitHub slowness then rendered as +# "kustomize build failed" on the whole tier. A local file makes the build OFFLINE and +# deterministic — which also means `kustomize build` in the CI gate proves the real thing instead +# of proving GitHub was up. (platform/kubevirt still carries the remote-base pattern and its latent +# fragility; not this component's problem.) +# +# DO NOT HAND-EDIT THE BODY. Every local change belongs in kustomization.yaml as a transformer or +# patch, so the next re-vendor stays a mechanical regenerate-and-diff instead of a three-way merge. +# The image tag below is re-pinned (and fully qualified) by the `images:` transformer there. The +# rationale for each thing deliberately NOT changed — no privileged, no supplementalGroups, no +# -shared-dev-num, no NFD overlay, no health-management — is in README.md. +apiVersion: apps/v1 +kind: DaemonSet +metadata: + labels: + app: intel-gpu-plugin + name: intel-gpu-plugin +spec: + selector: + matchLabels: + app: intel-gpu-plugin + template: + metadata: + labels: + app: intel-gpu-plugin + spec: + containers: + - env: + - name: NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: HOST_IP + valueFrom: + fieldRef: + fieldPath: status.hostIP + image: intel/intel-gpu-plugin:0.37.0 + imagePullPolicy: IfNotPresent + name: intel-gpu-plugin + resources: + limits: + cpu: 100m + memory: 115Mi + requests: + cpu: 40m + memory: 45Mi + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + seLinuxOptions: + type: container_device_plugin_t + seccompProfile: + type: RuntimeDefault + volumeMounts: + - mountPath: /dev/dri + name: devfs + readOnly: true + - mountPath: /sys/class/drm + name: sysfsdrm + readOnly: true + - mountPath: /var/lib/kubelet/device-plugins + name: kubeletsockets + - mountPath: /var/run/cdi + name: cdipath + nodeSelector: + kubernetes.io/arch: amd64 + volumes: + - hostPath: + path: /dev/dri + name: devfs + - hostPath: + path: /sys/class/drm + name: sysfsdrm + - hostPath: + path: /var/lib/kubelet/device-plugins + name: kubeletsockets + - hostPath: + path: /var/run/cdi + type: DirectoryOrCreate + name: cdipath + updateStrategy: + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 + type: RollingUpdate diff --git a/platform/intel-gpu-plugin/kustomization.yaml b/platform/intel-gpu-plugin/kustomization.yaml new file mode 100644 index 0000000..1ca08fa --- /dev/null +++ b/platform/intel-gpu-plugin/kustomization.yaml @@ -0,0 +1,99 @@ +--- +# Intel GPU device plugin for Kubernetes — makes the nodes advertise `gpu.intel.com/i915`. +# +# WHAT THIS FIXES, and why nothing cheaper works. Plex already has a `/dev/dri` hostPath mount +# (merged on main @ cdf3e14). The device node is VISIBLE in the container and cannot be OPENED. +# Measured in the running pod on k3s-prod-2: +# uid 0, O_RDWR -> EPERM "Operation not permitted" +# uid 1000, O_RDWR -> EPERM "Operation not permitted" +# EPERM from the DEVICE CGROUP, not EACCES from file permissions — proven three ways: root on the +# HOST opens the same device rc=0; the node is unified cgroup2fs; and the container's runc +# config.json carries one deny-all rule and nothing else: +# linux.resources.devices: [{'allow': False, 'access': 'rwm'}] +# linux.devices: None +# No allow entry for major 226, and the runtime was never asked to grant a device. A hostPath is a +# FILESYSTEM operation: it makes the device node appear, and adds no cgroup allow rule, because +# Kubernetes cannot infer one from a volume. +# +# A device plugin is the mechanism that closes exactly that gap. Verified by reading the v0.37.0 +# source (cmd/gpu_plugin/gpu_plugin.go, createDeviceSpecsFromDrmFiles): the Allocate response +# carries `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, and it +# is that DeviceSpec which makes kubelet populate `linux.devices` AND the matching cgroup allow +# rule. It is the only NON-PRIVILEGED way to get there. `privileged: true` would also work and is +# not available: platform/kyverno/policies/restrict-host-namespaces.yaml blocks it under Enforce in +# app namespaces, and `plex` is labelled app-namespace=true (measured 2026-10-01). Correctly so. +# +# STANDALONE DAEMONSET, NOT THE OPERATOR. Upstream ships both. The operator adds a CRD +# (GpuDevicePlugin), a controller-manager Deployment, cert-manager-backed admission webhooks and a +# second namespace — i.e. three more things that can be unhealthy — to buy declarative management +# of a DaemonSet we are declaring in git anyway. For three identical nodes and one consumer that is +# strictly more blast radius for no gain. The DaemonSet has no CRDs, no controller, no webhook, and +# no API-server access at all: a device plugin talks to the LOCAL kubelet over a unix socket, which +# is why there is no ServiceAccount or RBAC here (and therefore no `**/rbac.yaml` sensitive path). +# +# LISTING, NOT OPENING — the thing worth being sure about, since opening is precisely what fails +# today. Verified in v0.37.0 source: the scan path uses only os.ReadDir / os.ReadFile / os.Stat / +# os.Readlink over /sys/class/drm and /dev/dri. There is no os.Open, no OpenFile, no ioctl and no +# syscall of any kind against a DRM node anywhere in cmd/gpu_plugin/gpu_plugin.go. Listing and +# stat() demonstrably work today (that is how the Plex probe found the device before failing to +# open it), so the plugin needs no cgroup grant of its own and no privilege. +# * `--health-management` is left OFF (upstream default). It is the one flag that WOULD need to +# open a device: it starts a Level-Zero sidecar (setupLevelZeroService returns early unless the +# flag is set). Turning it on would reintroduce the exact EPERM this component exists to fix. +# * /dev/dri and /sys/class/drm are mounted readOnly, consistent with list-and-stat only. +# * /dev/dri/by-path absence is non-fatal (newDevicePlugin warns and sets bypathFound=false). +# +# RESOURCE NAME: `gpu.intel.com/i915`. Not a guess — `scan()` does +# `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the KERNEL DRIVER name. These nodes +# have i915 bound (not xe) on UHD Graphics 630 / PCI 8086:3e98, so the resource is i915. A future +# kernel moving CoffeeLake to xe would silently rename it to `gpu.intel.com/xe` and the consumer's +# request would stop matching — worth knowing before that kernel arrives. +# +# `--shared-dev-num` LEFT AT THE UPSTREAM DEFAULT OF 1 (so: not set). The flag multiplies each +# card into N schedulable device IDs (`for i := 0; i < sharedDevNum; i++`), letting N containers +# share one GPU. Deliberately not raised: +# * Plex is the only consumer and wants one device. +# * The count is a LIE the scheduler believes. There is no GPU-side partitioning or quota behind +# it — N containers get the same renderD128 and contend for the same two media engines +# (vcs0/vecs0). Raising it converts "Unschedulable", which is loud and obvious, into +# "transcodes are mysteriously slow", which is neither. +# * It is a one-line, zero-migration change on the day a second consumer actually exists, and at +# that point the right number will be known rather than guessed. +# No NFD overlay either: all three nodes are identical (established cluster fact), so a nodeSelector +# on a GPU feature label would gate on something that is true everywhere. If a GPU-less node ever +# joins, add upstream's nfd overlay THEN rather than relying on the plugin's behaviour there. +# +# NO supplementalGroups, anywhere, on purpose. The pms-docker image's own cont-init hook stats each +# /dev/dri char device and adds the `plex` user to its group — measured live as `Groups: 44 100 992 +# 1000` on the running Plex process, where GID 992 is the host `render` group. That mechanism is +# image-side and works identically on a plugin-injected device, so adding a supplementalGroup here +# or on the consumer would duplicate a thing that already works. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: intel-gpu-plugin +resources: + - namespace.yaml + - intel-gpu-plugin.yaml +images: + # PINNED, never `latest`. Upstream release v0.37.0 (2026-09-16); verified present on Docker Hub + # 2026-10-01, manifest digest sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1 + # (single amd64 manifest, not an index). Recorded here rather than pinned by digest because this + # repo pins upstream platform components by VERSION TAG (platform/kubevirt, external-snapshotter); + # digest pinning is reserved for the conformance runner, where it is a security boundary. + # `newName` fully qualifies the registry: upstream writes the bare `intel/intel-gpu-plugin`, which + # relies on containerd's docker.io default rather than saying so. + # + # RENOVATE: the inline-marker idiom (customManagers in renovate.json), not the docker datasource — + # renovate.json sets `"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}`, so an + # image reference in a plain manifest is tracked by NOTHING. The marker must sit on the line + # IMMEDIATELY above the version literal for the regex to match, which is why it is down here on + # `newTag` and not up in this header. + # CAVEAT, stated because it will bite someone: the GitHub release tags carry a `v` + # (`v0.37.0`) and the IMAGE tags do not (`0.37.0` — confirmed against the Docker Hub tag list). + # Whether Renovate normalises that for this dep is UNVERIFIED here (no Renovate run available). + # If it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in + # renovate.json, not a hand-edit of this line. + - name: intel/intel-gpu-plugin + newName: docker.io/intel/intel-gpu-plugin + # renovate: datasource=github-releases depName=intel/intel-device-plugins-for-kubernetes + newTag: "0.37.0" diff --git a/platform/intel-gpu-plugin/namespace.yaml b/platform/intel-gpu-plugin/namespace.yaml new file mode 100644 index 0000000..1cc1e16 --- /dev/null +++ b/platform/intel-gpu-plugin/namespace.yaml @@ -0,0 +1,34 @@ +--- +# intel-gpu-plugin — namespace for the Intel GPU device plugin DaemonSet. +# +# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and that distinction is load-bearing here rather +# than cosmetic. Every Kyverno policy that could object to this DaemonSet is scoped by a +# `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"` — verified by +# reading the policies, not assumed: +# * platform/kyverno/policies/restrict-host-namespaces.yaml (hostNetwork/PID/IPC + privileged) +# * platform/kyverno/policies/require-egress-class-label.yaml +# * platform/kyverno/policies/require-auth-mode-label.yaml +# * platform/kyverno/policies/require-default-deny-baseline.yaml (a *generate* rule, so an +# app-namespace also gets a default-deny NetworkPolicy minted into it) +# Labelling this namespace "false" therefore puts it out of scope for all four. That is the same +# mechanism restrict-host-namespaces' own header describes: "platform components that do [need host +# access] live in platform namespaces that do NOT carry the app-namespace label". +# +# Measured on prod 2026-10-01: `kyverno` and `vpn-egress` are the two namespaces that carry the +# label explicitly as "false"; the rest of the platform namespaces leave it unset. Both are out of +# scope identically — an unset label does not match `"true"`. We set it EXPLICITLY, following +# vpn-egress, because "someone chose this" and "nobody thought about it" should not look the same +# in git, and because this component's correctness depends on the answer. +# +# NOTE this plugin does NOT actually need any of the things those policies forbid: it sets no +# hostNetwork/hostPID/hostIPC and is not privileged (see intel-gpu-plugin.yaml). The namespace +# placement is not a workaround for a policy — there is nothing to work around. It is here because +# a node-level platform DaemonSet is not an application, and because the app-namespace fabric +# (default-deny NetworkPolicy, egress-class, auth-mode) describes obligations that are meaningless +# for a process that talks only to the local kubelet over a unix socket and never uses the network. +apiVersion: v1 +kind: Namespace +metadata: + name: intel-gpu-plugin + labels: + homelab.keithjackson.dev/app-namespace: "false"