Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4e1b169256 | ||
|
|
cdf3e14e27 | ||
|
|
41a20dfd20 |
No files matched your search
@@ -0,0 +1,104 @@
|
|||||||
|
---
|
||||||
|
# Flux Kustomization: Intel GPU device plugin — reconciles ./platform/intel-gpu-plugin.
|
||||||
|
# Makes the three prod nodes advertise `gpu.intel.com/i915`, which is the resource a later PR will
|
||||||
|
# have `hardware.gpu: intel-quicksync` request in place of today's non-working /dev/dri hostPath.
|
||||||
|
#
|
||||||
|
# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
|
||||||
|
#
|
||||||
|
# The cheap option was a line in the shared `platform` tier plus a new platform/ directory, and
|
||||||
|
# neither path is sensitive in infra/gitea/approval-policy.json, so it would have needed no human
|
||||||
|
# reviewer on the policy's terms. It is still the wrong place, for the reason the repo already
|
||||||
|
# wrote down in platform/kustomization.yaml when Cilium was split out of that tier:
|
||||||
|
#
|
||||||
|
# * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot tell
|
||||||
|
# you "the GPU plugin is wedged" from "longhorn is slow".
|
||||||
|
# * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
|
||||||
|
# TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device
|
||||||
|
# plugin.
|
||||||
|
# * The tier's 10m interval / 5m timeout are explicitly called out there as WRONG for a 3-node
|
||||||
|
# DaemonSet rollout. This is a 3-node DaemonSet rollout.
|
||||||
|
#
|
||||||
|
# And the consequence is worse here than for an average component, because of what sits downstream:
|
||||||
|
# `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates apps-wave-1.
|
||||||
|
# A plugin DaemonSet that never went healthy inside that tier would stall every app on "dependency
|
||||||
|
# not ready" — i.e. a component added to fix a SILENT NO-OP (software transcoding) would have
|
||||||
|
# acquired the ability to cause a REAL OUTAGE. That trade is the whole argument.
|
||||||
|
#
|
||||||
|
# The usual objection to this placement is the approval cost, and it does not apply: Gitea enforces
|
||||||
|
# `required_approvals: 1` with `block_admin_merge_override: true` on EVERY PR here regardless of
|
||||||
|
# path, and authors cannot self-approve. So this change needed the operator's approval either way;
|
||||||
|
# touching clusters/** costs nothing extra and buys a blast radius of exactly one component.
|
||||||
|
#
|
||||||
|
# dependsOn: [cilium], WHICH IS THE MINIMUM AND ALSO THE EARLIEST
|
||||||
|
#
|
||||||
|
# Not platform-config (what headlamp/crafty/homepage/oauth2-proxy use) — those are leaf UIs that
|
||||||
|
# genuinely need Keycloak + wildcard TLS + split-horizon DNS + Gateway. This plugin needs none of
|
||||||
|
# it. It never uses the network and never talks to the API server; it talks to the local kubelet
|
||||||
|
# over a unix socket. Its only real prerequisite is a working CNI, so its pod can get a sandbox at
|
||||||
|
# all, and `cilium` already depends on `flux-system` (the root Kustomization that applies the
|
||||||
|
# cluster-vars ConfigMap this one substitutes from), so that edge is transitively satisfied too.
|
||||||
|
#
|
||||||
|
# Depending on `cilium` rather than `platform` also gets the resource advertised EARLIEST — this
|
||||||
|
# reconciles in parallel with the `platform` tier instead of behind all eleven of its components.
|
||||||
|
# That matters on a cold start once the consumer PR lands: a Plex pod requesting a resource no node
|
||||||
|
# advertises is Unschedulable, and apps-wave-0 is wait:true. Reconciling early shrinks that window
|
||||||
|
# instead of widening it. It is not a GUARANTEE — see the ordering note below, which is a real
|
||||||
|
# constraint on the next PR and not a formality.
|
||||||
|
#
|
||||||
|
# Kyverno (in `platform`) is deliberately NOT a dependency: every policy that could match this
|
||||||
|
# workload is scoped to namespaces labelled app-namespace=true, and this namespace is labelled
|
||||||
|
# "false" (platform/intel-gpu-plugin/namespace.yaml). So there is no admission race to lose.
|
||||||
|
#
|
||||||
|
# wait: true — A HEALTH SIGNAL, NOT A GATE. This is the one place this component departs from the
|
||||||
|
# headlamp/crafty leaf template, and the departure is deliberate. Those run wait:false because a
|
||||||
|
# leaf UI can be legitimately not-Ready for a long time during a cold start while it waits on
|
||||||
|
# Keycloak/DNS/TLS, so its health is not a useful signal. This DaemonSet has no such dependency: it
|
||||||
|
# reads sysfs and opens a socket, so on healthy nodes it should be Ready in seconds, and
|
||||||
|
# not-Ready genuinely means BROKEN. wait:true turns `flux get kustomization intel-gpu-plugin` into
|
||||||
|
# a one-line answer to "did the plugin come up on all three nodes", which is exactly the question
|
||||||
|
# the operator must answer before the consumer PR may merge.
|
||||||
|
#
|
||||||
|
# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
|
||||||
|
# plugin is this Kustomization's own Ready condition and nothing else. Anyone later adding
|
||||||
|
# `dependsOn: [intel-gpu-plugin]` is choosing to make plugin health gate their thing; for the app
|
||||||
|
# waves that would hand a device plugin the power to stall every app, which is the thing this file
|
||||||
|
# exists to prevent. Think before adding that edge.
|
||||||
|
#
|
||||||
|
# ORDERING — DO NOT REORDER THE STACK. The renderer change (catalog/render.py emitting
|
||||||
|
# `gpu.intel.com/i915: 1` for `hardware.gpu: intel-quicksync` instead of the hostPath) must NOT
|
||||||
|
# land until this plugin is merged AND confirmed advertising the resource on all three nodes. If
|
||||||
|
# the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is
|
||||||
|
# wait:true and apps-wave-1 dependsOn it, so that stalls the whole app tier — turning today's silent
|
||||||
|
# no-op into a real outage. This PR's only job is to make the resource exist.
|
||||||
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||||
|
kind: Kustomization
|
||||||
|
metadata:
|
||||||
|
name: intel-gpu-plugin
|
||||||
|
namespace: flux-system
|
||||||
|
spec:
|
||||||
|
# Static vendored manifests with nothing to re-render; kustomize-controller is source-triggered,
|
||||||
|
# so this is only the drift-correction floor.
|
||||||
|
interval: 10m
|
||||||
|
retryInterval: 2m
|
||||||
|
# 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
|
||||||
|
# exceed 5m on a cold start (image pull on three nodes), and the Cilium split-out comment calls
|
||||||
|
# out that exact mismatch as one of its reasons for leaving the shared tier.
|
||||||
|
timeout: 10m
|
||||||
|
prune: true
|
||||||
|
# See the header: a health SIGNAL for a component nothing depends on, not a gate.
|
||||||
|
wait: true
|
||||||
|
dependsOn:
|
||||||
|
- name: cilium # the CNI — the only real prerequisite; transitively gives us flux-system too
|
||||||
|
sourceRef:
|
||||||
|
kind: GitRepository
|
||||||
|
name: flux-system
|
||||||
|
path: ./platform/intel-gpu-plugin
|
||||||
|
# Retained for parity with every other tier / future overlay knobs. This component references no
|
||||||
|
# ${VARS} today, and its manifests contain no `$` literals for envsubst to eat (unlike
|
||||||
|
# clusters/base/runbooks.yaml, which omits substitution for exactly that reason).
|
||||||
|
postBuild:
|
||||||
|
substituteFrom:
|
||||||
|
- kind: ConfigMap
|
||||||
|
name: cluster-vars
|
||||||
|
# No SOPS decryption patch: this component carries no Secret, so clusters/dev adds nothing for it
|
||||||
|
# (same as homepage/runbooks, unlike headlamp/crafty/oauth2-proxy/backup).
|
||||||
@@ -12,6 +12,15 @@ resources:
|
|||||||
# interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
|
# interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
|
||||||
# freeze the other eleven platform components. See clusters/base/cilium.yaml.
|
# freeze the other eleven platform components. See clusters/base/cilium.yaml.
|
||||||
- cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
|
- cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
|
||||||
|
# Intel GPU device plugin — its OWN Kustomization (dependsOn cilium, wait:true), NOT a line in
|
||||||
|
# platform/kustomization.yaml. Same reasoning as the Cilium split-out, applied to a leaf: inside
|
||||||
|
# that wait:true tier a wedged 3-node DaemonSet would be indistinguishable from eleven unrelated
|
||||||
|
# components AND would stall platform-config + every apps wave on "dependency not ready" — so a
|
||||||
|
# component added to fix a silent no-op (Plex software transcoding) could cause a real outage.
|
||||||
|
# wait:true here is a HEALTH SIGNAL, not a gate: nothing in the graph dependsOn it.
|
||||||
|
# It advertises `gpu.intel.com/i915`; the renderer change that REQUESTS that resource must land
|
||||||
|
# strictly later. See clusters/base/intel-gpu-plugin.yaml.
|
||||||
|
- intel-gpu-plugin.yaml # Flux Kustomization -> ./platform/intel-gpu-plugin (dependsOn cilium)
|
||||||
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
||||||
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
||||||
# SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
|
# SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
|
||||||
|
|||||||
@@ -0,0 +1,198 @@
|
|||||||
|
---
|
||||||
|
date: 2026-10-01
|
||||||
|
branch: feat/intel-gpu-device-plugin
|
||||||
|
title: Intel GPU device plugin, in its own Kustomization — the resource exists; nothing has run yet
|
||||||
|
---
|
||||||
|
|
||||||
|
## What changed
|
||||||
|
|
||||||
|
Deployed the **Intel Device Plugin for Kubernetes (GPU)**, standalone DaemonSet, pinned at
|
||||||
|
**v0.37.0**, so the prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the
|
||||||
|
Plex hardware-transcoding blocker: the `/dev/dri` hostPath merged on `main` @ `cdf3e14` makes the
|
||||||
|
device node *visible* and not *openable* (`EPERM` from the device cgroup at both uid 0 and uid
|
||||||
|
1000; the container's runc `config.json` carries one deny-all rule and `linux.devices: None`). A
|
||||||
|
hostPath is a filesystem operation and adds no cgroup allow rule. A device plugin's `Allocate`
|
||||||
|
response is what makes kubelet populate `linux.devices` *and* the matching cgroup allow rule, and
|
||||||
|
it is the only non-privileged way to get there — `privileged: true` is blocked by Kyverno
|
||||||
|
`restrict-host-namespaces` under Enforce in app namespaces, and `plex` is labelled
|
||||||
|
`app-namespace=true`.
|
||||||
|
|
||||||
|
New files:
|
||||||
|
|
||||||
|
- `platform/intel-gpu-plugin/intel-gpu-plugin.yaml` — the DaemonSet, **vendored** as
|
||||||
|
`kubectl kustomize` output of upstream `deployments/gpu_plugin/base` at `v0.37.0`, under a
|
||||||
|
provenance header carrying the exact re-vendor command and both sha256s (generated body
|
||||||
|
`80ebd070…`, raw upstream source `c8773918…`). Generated rather than byte-copied because
|
||||||
|
upstream mixes YAML sequence-indentation styles within the one document and `ci/yamllint.yaml`
|
||||||
|
sets `indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate.
|
||||||
|
That is not a guess: the first gate run of this branch returned **rc 1 BLOCKED** on exactly that
|
||||||
|
error, and regenerating through kustomize is the fix (and is also the re-vendor idiom
|
||||||
|
`platform/external-snapshotter` documents). Vendored at all, rather than referenced as a remote
|
||||||
|
kustomize base, for the other reason `external-snapshotter` records: a remote base is
|
||||||
|
`git fetch`ed on every reconcile and those fetches have flaked a whole tier.
|
||||||
|
- `platform/intel-gpu-plugin/namespace.yaml` — a PLATFORM namespace, explicitly
|
||||||
|
`homelab.keithjackson.dev/app-namespace: "false"` (following `vpn-egress`/`kyverno`).
|
||||||
|
- `platform/intel-gpu-plugin/kustomization.yaml` — namespace scoping plus the image pin.
|
||||||
|
- `platform/intel-gpu-plugin/README.md` — the measurement, the placement argument, the
|
||||||
|
non-choices, and the post-merge verification sequence.
|
||||||
|
- `clusters/base/intel-gpu-plugin.yaml` — **its own Flux Kustomization**: `dependsOn: [cilium]`,
|
||||||
|
`wait: true`, `interval: 10m`, `timeout: 10m`, `prune: true`.
|
||||||
|
- One wired line (plus its rationale comment) in `clusters/base/kustomization.yaml`.
|
||||||
|
|
||||||
|
**Placement: its own Kustomization, not a line in the shared `platform` tier.** The shared tier was
|
||||||
|
cheaper and needed no human reviewer on `approval-policy.json`'s terms, but `platform` is
|
||||||
|
`wait: true` over eleven-plus components and the repo already argued this case when Cilium was
|
||||||
|
split out: its Ready condition cannot distinguish "the plugin is wedged" from "longhorn is slow",
|
||||||
|
the only emergency valve is freezing the entire tier, and the tier's 10m/5m interval/timeout are
|
||||||
|
called out there as *wrong for a 3-node DaemonSet rollout* — which is exactly what this is. The
|
||||||
|
consequence is worse than average here, because `platform` gates `platform-config` gates
|
||||||
|
`apps-wave-0` (`wait: true`) gates `apps-wave-1`: a plugin that never went healthy inside that tier
|
||||||
|
would stall every app, so a component added to fix a **silent no-op** would have gained the ability
|
||||||
|
to cause a **real outage**. The approval cost of touching `clusters/**` is nil, because
|
||||||
|
`required_approvals: 1` with `block_admin_merge_override: true` applies to every PR here regardless
|
||||||
|
of path and authors cannot self-approve.
|
||||||
|
|
||||||
|
`dependsOn: [cilium]` is the minimum *and* the earliest: the plugin never uses the network and
|
||||||
|
never talks to the API server (it talks to the local kubelet over a unix socket), so a working CNI
|
||||||
|
is its only real prerequisite, and `cilium` already depends on `flux-system`, which applies the
|
||||||
|
`cluster-vars` ConfigMap this Kustomization substitutes from. Depending on `cilium` rather than
|
||||||
|
`platform` also means it reconciles *in parallel with* the platform tier rather than behind all
|
||||||
|
eleven of its components, which shrinks the cold-start window in which a node has not yet
|
||||||
|
advertised the resource.
|
||||||
|
|
||||||
|
`wait: true` is a deliberate departure from the `headlamp`/`crafty` leaf template. Those use
|
||||||
|
`wait: false` because a leaf UI can be legitimately not-Ready for a long time during a cold start,
|
||||||
|
making its health a useless signal. This DaemonSet reads sysfs and opens a socket, so not-Ready
|
||||||
|
genuinely means broken — and since **nothing in the graph `dependsOn` it** (verified against the
|
||||||
|
rendered graph), `wait: true` gates nothing while turning
|
||||||
|
`flux get kustomization intel-gpu-plugin` into a one-line answer to the question that must be
|
||||||
|
answered before the consumer PR may merge.
|
||||||
|
|
||||||
|
Deliberate non-choices, each argued in the README: standalone DaemonSet rather than the operator
|
||||||
|
(no CRD, no controller-manager, no cert-manager webhook, no second namespace to keep healthy);
|
||||||
|
`--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler
|
||||||
|
believes — there is no GPU-side partitioning behind it, so it converts a loud "Unschedulable" into
|
||||||
|
a quiet "transcodes are slow"); `--health-management` left OFF, because it is the one flag that
|
||||||
|
*would* open a device and would reintroduce the exact `EPERM` this fixes; no NFD overlay (all three
|
||||||
|
nodes are identical); no `supplementalGroups` anywhere, because the `pms-docker` cont-init hook
|
||||||
|
already adds the `plex` user to each device's group and that works identically on a
|
||||||
|
plugin-injected device.
|
||||||
|
|
||||||
|
**Scope held deliberately:** `catalog/render.py`, `catalog/plex.yaml` and `apps/wave-0/plex.yaml`
|
||||||
|
are untouched. Switching Plex from the hostPath to a `gpu.intel.com/i915: 1` request is a strictly
|
||||||
|
later PR, and **it must not land until this plugin is merged AND confirmed advertising the resource
|
||||||
|
on all three nodes** — a request for a resource no node advertises makes the Plex pod
|
||||||
|
Unschedulable, and `apps-wave-0` is `wait: true` with `apps-wave-1` depending on it, so that would
|
||||||
|
stall the whole app tier and turn today's silent no-op into a real outage.
|
||||||
|
|
||||||
|
## What I verified
|
||||||
|
|
||||||
|
Mechanism, read from the v0.37.0 source rather than assumed:
|
||||||
|
|
||||||
|
- **The plugin only lists devices; it never opens one.** The scan path uses `os.ReadDir`,
|
||||||
|
`os.ReadFile`, `os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. `grep` for
|
||||||
|
`os.Open|OpenFile|ioctl|syscall|unix\.` across `cmd/gpu_plugin/gpu_plugin.go` (908 lines) returns
|
||||||
|
**nothing**. Listing and `stat()` are what demonstrably work today, so the plugin needs no cgroup
|
||||||
|
grant and no extra privilege — it does not need more than the upstream manifest gives it.
|
||||||
|
- **The fix mechanism is real:** `createDeviceSpecsFromDrmFiles` emits
|
||||||
|
`pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, which is the
|
||||||
|
thing kubelet turns into `linux.devices` plus a cgroup allow rule.
|
||||||
|
- **The resource name is `gpu.intel.com/i915` by derivation, not by hope:** `scan()` calls
|
||||||
|
`devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the kernel driver name, and these
|
||||||
|
nodes have `i915` bound.
|
||||||
|
- `--shared-dev-num` default is `1` (`flag.IntVar(..., 1, ...)`); `--health-management` default is
|
||||||
|
`false` and `setupLevelZeroService` returns early unless it is set.
|
||||||
|
- Missing `/dev/dri/by-path` is non-fatal (`newDevicePlugin` warns, sets `bypathFound = false`).
|
||||||
|
|
||||||
|
Image pin, verified against the registry:
|
||||||
|
|
||||||
|
- `docker.io/intel/intel-gpu-plugin:0.37.0` exists; manifest digest
|
||||||
|
`sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1` (single amd64
|
||||||
|
manifest, not an index), fetched from `registry-1.docker.io` on 2026-10-01. `v0.37.0` is the
|
||||||
|
latest upstream release (2026-09-16) and is not a prerelease.
|
||||||
|
- The Docker Hub tag list carries `0.35.0 / 0.36.0 / 0.37.0` — **image tags have no `v` prefix**
|
||||||
|
while GitHub release tags do.
|
||||||
|
|
||||||
|
Policy scope, read from the policies rather than assumed:
|
||||||
|
|
||||||
|
- All four app-scoped Kyverno policies (`restrict-host-namespaces`, `require-egress-class-label`,
|
||||||
|
`require-auth-mode-label`, `require-default-deny-baseline`) are gated on
|
||||||
|
`namespaceSelector`/`selector` `matchLabels: homelab.keithjackson.dev/app-namespace: "true"`.
|
||||||
|
The new namespace is labelled `"false"`, so none of them match it.
|
||||||
|
- Measured on prod: `plex` **is** `app-namespace=true`, confirming `privileged: true` is genuinely
|
||||||
|
unavailable there; `kyverno` and `vpn-egress` are the two namespaces carrying the label as
|
||||||
|
`"false"`; the other platform namespaces leave it unset. No `intel*` namespace exists, so no
|
||||||
|
collision.
|
||||||
|
|
||||||
|
Cluster before-state (read-only, `kubectl` in the pinned toolbox with the prod kubeconfig
|
||||||
|
bind-mounted read-only — never copied into the worktree):
|
||||||
|
|
||||||
|
- Three nodes `k3s-prod-{1,2,3}`, each `v1.36.4+k3s1`, Debian 13 trixie, `containerd://2.3.4-k3s1.36`.
|
||||||
|
- Every extended resource currently advertised is `devices.kubevirt.io/{kvm,sev,tun,vhost-net}` on
|
||||||
|
all three nodes. **No node advertises `gpu.intel.com/*`** — queried explicitly, returned
|
||||||
|
"NONE". That is the before-state this change is meant to move.
|
||||||
|
- Those `devices.kubevirt.io/*` resources are themselves advertised by a device plugin, which is
|
||||||
|
measured evidence that the device-plugin API and the default kubelet socket path
|
||||||
|
(`/var/lib/kubelet/device-plugins`) work on these nodes.
|
||||||
|
|
||||||
|
Offline rendering (the main proof available to a PR):
|
||||||
|
|
||||||
|
- `kubectl kustomize platform/intel-gpu-plugin` renders two objects: `Namespace/intel-gpu-plugin`
|
||||||
|
carrying `app-namespace=false`, and `DaemonSet/intel-gpu-plugin` in namespace
|
||||||
|
`intel-gpu-plugin`, with the image resolved to `docker.io/intel/intel-gpu-plugin:0.37.0`.
|
||||||
|
- `kubectl kustomize clusters/base` renders, and the new Flux Kustomization appears as
|
||||||
|
`intel-gpu-plugin ./platform/intel-gpu-plugin wait=true dependsOn=cilium`. Both overlays
|
||||||
|
(`clusters/prod`, `clusters/dev`) also render and carry it.
|
||||||
|
- Across the whole rendered graph, **no Kustomization lists `intel-gpu-plugin` in `dependsOn`** —
|
||||||
|
so `wait: true` gates nothing.
|
||||||
|
- The vendored DaemonSet is a **formatting-only** transform of upstream, proven rather than
|
||||||
|
asserted: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file `diff`
|
||||||
|
clean, so passing it through kustomize changed no content. The committed body's sha256 matches
|
||||||
|
the one recorded in its own provenance header (`80ebd070…`), so the file has not drifted from
|
||||||
|
what was generated.
|
||||||
|
|
||||||
|
Gate (pinned toolbox, from this worktree, with the `ENGINE` shim bind-mounting the main checkout's
|
||||||
|
gitdir and `DEV_KUBECONFIG=/dev/null`): see the PR description for the per-stage table. **`rc 3`
|
||||||
|
(UNDETERMINED), with exactly the three expected skips** — `spec-immutability` (sibling spec repo is
|
||||||
|
not beside a worktree), `gitea-settings` (no CI token), `conformance` (no cluster in CI). Every
|
||||||
|
other stage passed. **rc 3 is not green** and is not reported as such.
|
||||||
|
|
||||||
|
## What I did NOT verify
|
||||||
|
|
||||||
|
**Nothing about this component's runtime behaviour. It has never run.**
|
||||||
|
|
||||||
|
- **The plugin has never executed.** No pod has been scheduled, no image has been pulled, no scan
|
||||||
|
has happened. It cannot run until the operator merges this and Flux reconciles — `kubectl apply`
|
||||||
|
and `flux reconcile` are operator actions (CLAUDE.md §5), so this is expected, not an oversight.
|
||||||
|
- **The resource has never been advertised.** `gpu.intel.com/i915` does not exist on any node and
|
||||||
|
I have not seen it exist. The claim that it *will* be `i915` rather than `xe` is derived from
|
||||||
|
source plus the established fact that `i915` is bound — it is a derivation, not an observation.
|
||||||
|
- **The DaemonSet has never been admitted.** That the Kyverno policies do not match this namespace
|
||||||
|
is read off the policy manifests; no admission request has actually been evaluated. Nor has the
|
||||||
|
`kubectl kustomize` output ever been server-side dry-run against the API — rendering is not
|
||||||
|
admission.
|
||||||
|
- **The `open()` test has NOT been re-run.** The `EPERM` is exactly as it was. Nothing in this PR
|
||||||
|
touches Plex, so Plex is still software-transcoding and will continue to until the later renderer
|
||||||
|
PR lands. The decisive evidence is an `open()` as **uid 1000 through `s6-setuidgid plex`**;
|
||||||
|
passing as uid 0 is weaker and must not be accepted as proof.
|
||||||
|
- **A green gate proves the manifests render, nothing more.** And this gate was not green: it was
|
||||||
|
`rc 3`. Note also that the Gitea `gate` job runs `./ci/gate.sh || [ "$?" -eq 3 ]`
|
||||||
|
(`.gitea/workflows/gate.yml:138`), so a green tick on this PR cannot distinguish rc 0 from rc 3,
|
||||||
|
and `vars.CLUSTER_AVAILABLE` is unset so no L2/L3 conformance runs in CI.
|
||||||
|
- **`seLinuxOptions: type: container_device_plugin_t`** is carried verbatim from upstream. These
|
||||||
|
nodes are Debian 13 and I did not measure whether SELinux is enabled or whether that type exists
|
||||||
|
there. Expected to be inert (runc skips labelling when SELinux is disabled), but unverified — if
|
||||||
|
the pods fail to start with an SELinux error, that line is the first suspect.
|
||||||
|
- **Renovate has not been run** against this pin. The inline-marker idiom is copied from
|
||||||
|
`platform/kubevirt` / `platform/external-snapshotter`, but whether Renovate normalises the
|
||||||
|
`v`-prefix mismatch (release `v0.37.0` vs image `0.37.0`) for this dep is unknown. If it proposes
|
||||||
|
`newTag: v0.38.0`, the fix is an `extractVersion` packageRule in `renovate.json`.
|
||||||
|
- **Resource limits unexamined under load.** The upstream 115Mi/100m limits are taken as-is; I have
|
||||||
|
no measurement of this plugin's footprint on these nodes. Worth a glance given
|
||||||
|
`keycloak-operator` is OOMing at 91% of its limit elsewhere in this cluster.
|
||||||
|
- **`clusters/dev` renders it, and there is no dev cluster** (the operator confirmed the
|
||||||
|
`k3s-dev-*` nodes are gone). That is inert rather than wrong — the unified graph (R-ENV-02) has
|
||||||
|
both overlays render the same base — but it is not a thing I could test.
|
||||||
|
- **Branch protection / required-checks state not re-measured** in this session. CLAUDE.md §2b
|
||||||
|
says re-measure before relying on it; I did not, so treat the approval/gating claims in §2 as
|
||||||
|
unconfirmed-by-me.
|
||||||
@@ -0,0 +1,225 @@
|
|||||||
|
# platform/intel-gpu-plugin
|
||||||
|
|
||||||
|
**Intel Device Plugin for Kubernetes (GPU)** — the standalone GPU plugin DaemonSet from
|
||||||
|
[`intel/intel-device-plugins-for-kubernetes`](https://github.com/intel/intel-device-plugins-for-kubernetes),
|
||||||
|
pinned at **v0.37.0**. It makes every node advertise **`gpu.intel.com/i915`** so a workload can
|
||||||
|
*request* a GPU instead of hoping a `hostPath` is enough.
|
||||||
|
|
||||||
|
- **Serves:** hardware transcoding for Plex (`hardware.gpu: intel-quicksync`), and any later
|
||||||
|
consumer of an Intel iGPU.
|
||||||
|
- **Tier:** its **own** Flux Kustomization
|
||||||
|
([`clusters/base/intel-gpu-plugin.yaml`](../../clusters/base/intel-gpu-plugin.yaml),
|
||||||
|
`dependsOn: cilium`, `wait: true`) — **not** a line in `platform/kustomization.yaml`. See
|
||||||
|
*Placement* below.
|
||||||
|
- **Namespace:** `intel-gpu-plugin` — a PLATFORM namespace, labelled
|
||||||
|
`homelab.keithjackson.dev/app-namespace: "false"`, so the app-scoped Kyverno policies
|
||||||
|
(default-deny generation, auth-mode, egress-class, host-namespaces) do not apply.
|
||||||
|
- **No RBAC, no ServiceAccount, no CRDs, no webhook, no API-server access.** A device plugin talks
|
||||||
|
to the *local kubelet* over a unix socket.
|
||||||
|
|
||||||
|
## What this component owns
|
||||||
|
|
||||||
|
| File | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `namespace.yaml` | `intel-gpu-plugin` ns, explicitly `app-namespace=false` |
|
||||||
|
| `intel-gpu-plugin.yaml` | the DaemonSet, **vendored** (machine-generated) from upstream v0.37.0 |
|
||||||
|
| `kustomization.yaml` | namespace scoping + the pinned image (`images:` transformer) |
|
||||||
|
|
||||||
|
## The problem this fixes
|
||||||
|
|
||||||
|
Plex was given a `/dev/dri` hostPath (merged on `main` @ `cdf3e14`). The device is **visible** in
|
||||||
|
the container and **cannot be opened**. Measured in the running pod on `k3s-prod-2`:
|
||||||
|
|
||||||
|
```
|
||||||
|
uid 0, O_RDWR -> EPERM "Operation not permitted"
|
||||||
|
uid 1000, O_RDWR -> EPERM "Operation not permitted"
|
||||||
|
```
|
||||||
|
|
||||||
|
`EPERM` from the **device cgroup**, not `EACCES` from file permissions — proven three ways:
|
||||||
|
|
||||||
|
1. root on the **host** opens the same device with `rc=0`;
|
||||||
|
2. the node is unified `cgroup2fs`;
|
||||||
|
3. the container's runc `config.json` reads
|
||||||
|
`linux.resources.devices: [{'allow': False, 'access': 'rwm'}]` and `linux.devices: None` —
|
||||||
|
one deny-all rule, no allow entry for major 226, and the runtime was never asked to grant a
|
||||||
|
device.
|
||||||
|
|
||||||
|
A hostPath is a **filesystem** operation. It makes the device *node* appear and adds no cgroup
|
||||||
|
allow rule, because Kubernetes cannot infer one from a volume.
|
||||||
|
|
||||||
|
**A device plugin is the mechanism that closes exactly this gap.** Verified in the v0.37.0 source
|
||||||
|
(`cmd/gpu_plugin/gpu_plugin.go`, `createDeviceSpecsFromDrmFiles`), the `Allocate` response carries
|
||||||
|
per DRM node:
|
||||||
|
|
||||||
|
```go
|
||||||
|
specs = append(specs, pluginapi.DeviceSpec{
|
||||||
|
HostPath: devPath, ContainerPath: devPath, Permissions: "rw",
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
and it is that `DeviceSpec` which makes kubelet populate `linux.devices` **and** the matching
|
||||||
|
cgroup allow rule. It is the only **non-privileged** way to get there. `privileged: true` would
|
||||||
|
also work and is not available: `platform/kyverno/policies/restrict-host-namespaces.yaml` blocks it
|
||||||
|
under Enforce in app namespaces, and `plex` is labelled `app-namespace=true` (measured
|
||||||
|
2026-10-01). Correctly so.
|
||||||
|
|
||||||
|
## Placement — why its own Kustomization
|
||||||
|
|
||||||
|
The cheap option was one line in the shared `platform` tier. Neither that file nor a new
|
||||||
|
`platform/<name>/` directory is a sensitive path in `infra/gitea/approval-policy.json`, so it would
|
||||||
|
have needed no human reviewer *on the policy's terms*. It is still the wrong place, and the repo
|
||||||
|
already argued the case when Cilium was split out (read the comment in
|
||||||
|
`platform/kustomization.yaml`):
|
||||||
|
|
||||||
|
* `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot distinguish
|
||||||
|
"the GPU plugin is wedged" from "longhorn is slow".
|
||||||
|
* The only emergency valve would be `flux suspend kustomization platform` — freezing storage, TLS,
|
||||||
|
DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device plugin.
|
||||||
|
* The tier's 10m/5m interval/timeout are called out there as **wrong for a 3-node DaemonSet
|
||||||
|
rollout**. This is a 3-node DaemonSet rollout.
|
||||||
|
|
||||||
|
The consequence is worse than average here because of what sits downstream: `platform` gates
|
||||||
|
`platform-config`, which gates `apps-wave-0` (`wait: true`), which gates `apps-wave-1`. A plugin
|
||||||
|
that never went healthy inside that tier would stall **every app** on "dependency not ready" — so a
|
||||||
|
component added to fix a *silent no-op* (software transcoding) would have acquired the ability to
|
||||||
|
cause a *real outage*.
|
||||||
|
|
||||||
|
The usual objection — that `clusters/**` needs a human approval — costs nothing: Gitea enforces
|
||||||
|
`required_approvals: 1` with `block_admin_merge_override: true` on **every** PR here regardless of
|
||||||
|
path, and authors cannot self-approve. The approval was required either way.
|
||||||
|
|
||||||
|
### `wait: true` is a health signal, not a gate
|
||||||
|
|
||||||
|
This departs from the `headlamp`/`crafty` leaf template, deliberately. Those run `wait: false`
|
||||||
|
because a leaf UI can be legitimately not-Ready for a long time during a cold start while it waits
|
||||||
|
on Keycloak/DNS/TLS, making its health useless as a signal. This DaemonSet has no such dependency —
|
||||||
|
it reads sysfs and opens a socket, so on healthy nodes it is Ready in seconds and **not-Ready
|
||||||
|
genuinely means broken**. `wait: true` makes `flux get kustomization intel-gpu-plugin` a one-line
|
||||||
|
answer to "did it come up on all three nodes", which is precisely the question that must be
|
||||||
|
answered before the consumer PR may merge.
|
||||||
|
|
||||||
|
Nothing in the graph `dependsOn` this, so `wait: true` gates nothing. **Think hard before adding
|
||||||
|
`dependsOn: [intel-gpu-plugin]`** anywhere — for the app waves that would hand a device plugin the
|
||||||
|
power to stall every app, which is the thing this placement exists to prevent.
|
||||||
|
|
||||||
|
## Deliberate non-choices
|
||||||
|
|
||||||
|
**Standalone DaemonSet, not the operator.** The operator adds a CRD (`GpuDevicePlugin`), a
|
||||||
|
controller-manager Deployment, cert-manager-backed admission webhooks and a second namespace —
|
||||||
|
three more things that can be unhealthy — to buy declarative management of a DaemonSet that is
|
||||||
|
already declared in git. For three identical nodes and one consumer that is strictly more blast
|
||||||
|
radius for no gain.
|
||||||
|
|
||||||
|
**It lists devices; it never opens them** — worth being certain about, since *opening* is exactly
|
||||||
|
what fails today. Verified in v0.37.0 source: the scan path uses only `os.ReadDir`, `os.ReadFile`,
|
||||||
|
`os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. There is no `os.Open`, no
|
||||||
|
`OpenFile`, no `ioctl` and no raw syscall against a DRM node anywhere in
|
||||||
|
`cmd/gpu_plugin/gpu_plugin.go`. Listing and `stat()` demonstrably work today — that is how the Plex
|
||||||
|
probe found the device before failing to open it — so the plugin needs no cgroup grant of its own
|
||||||
|
and no privilege. Accordingly `/dev/dri` and `/sys/class/drm` are mounted **readOnly**.
|
||||||
|
|
||||||
|
* **`--health-management` stays OFF** (upstream default). It is the one flag that *would* need to
|
||||||
|
open a device: it starts a Level-Zero sidecar (`setupLevelZeroService` returns early unless the
|
||||||
|
flag is set). Enabling it would reintroduce the exact `EPERM` this component exists to fix.
|
||||||
|
* `/dev/dri/by-path` being absent is non-fatal — `newDevicePlugin` warns and sets
|
||||||
|
`bypathFound = false`.
|
||||||
|
|
||||||
|
**`--shared-dev-num` left at the upstream default of 1** (i.e. not set). The flag multiplies each
|
||||||
|
card into N schedulable device IDs, letting N containers share one GPU. Not raised, because:
|
||||||
|
|
||||||
|
* Plex is the only consumer today and wants one device;
|
||||||
|
* the count is a **lie the scheduler believes** — there is no GPU-side partitioning or quota behind
|
||||||
|
it, so N containers get the same `renderD128` and contend for the same two media engines
|
||||||
|
(`vcs0`/`vecs0`). Raising it converts "Unschedulable", which is loud and obvious, into
|
||||||
|
"transcodes are mysteriously slow", which is neither;
|
||||||
|
* it is a one-line, zero-migration change on the day a second consumer exists — and *then* the
|
||||||
|
right number will be known rather than guessed.
|
||||||
|
|
||||||
|
**No NFD overlay.** All three nodes are identical (HP EliteDesk 800 G5 Mini, i7-9700T, UHD Graphics
|
||||||
|
630, PCI `8086:3e98`, `i915` bound), so a nodeSelector on a GPU feature label would gate on
|
||||||
|
something true everywhere. If a GPU-less node ever joins, add upstream's `nfd` overlay *then*
|
||||||
|
rather than relying on the plugin's behaviour on such a node.
|
||||||
|
|
||||||
|
**No `supplementalGroups`, anywhere.** The `pms-docker` image's own cont-init hook stats each
|
||||||
|
`/dev/dri` char device and adds the `plex` user to its group — measured live as
|
||||||
|
`Groups: 44 100 992 1000` on the running Plex process, where GID 992 is the host `render` group.
|
||||||
|
That mechanism is image-side and works identically on a plugin-injected device.
|
||||||
|
|
||||||
|
## Resource name
|
||||||
|
|
||||||
|
`gpu.intel.com/i915`. Not a guess: `scan()` calls `devTree.AddDevice(devProps.driver(), ...)`, so
|
||||||
|
the suffix is the **kernel driver name**. These nodes have `i915` bound (not `xe`), so the resource
|
||||||
|
is `i915`. A future kernel moving CoffeeLake to `xe` would silently rename it to `gpu.intel.com/xe`
|
||||||
|
and a consumer's request would stop matching — worth knowing before that kernel arrives.
|
||||||
|
|
||||||
|
## Versioning
|
||||||
|
|
||||||
|
Pinned to **v0.37.0** via the `images:` transformer in `kustomization.yaml`; the DaemonSet itself is
|
||||||
|
vendored as `kubectl kustomize` output of the upstream base (see its header for the exact re-vendor
|
||||||
|
command and both sha256s). It is *generated* rather than a byte copy because upstream mixes YAML
|
||||||
|
sequence-indentation styles within the one document and `ci/yamllint.yaml` sets
|
||||||
|
`indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate
|
||||||
|
(measured: it did). `kubectl kustomize` normalises the formatting, and the result was proven to be
|
||||||
|
a formatting-only change: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file
|
||||||
|
diff clean. This is also the re-vendor idiom `platform/external-snapshotter` documents.
|
||||||
|
|
||||||
|
Vendored rather than referenced as a remote kustomize base for the same reason as
|
||||||
|
`platform/external-snapshotter`: a remote base is `git fetch`ed on **every** reconcile, those
|
||||||
|
fetches intermittently exceeded kustomize-controller's 27s timeout, and transient GitHub slowness
|
||||||
|
then rendered as "kustomize build failed". A local file makes the build offline and deterministic —
|
||||||
|
which also means the CI gate's `kustomize build` proves the manifests rather than proving GitHub
|
||||||
|
was up.
|
||||||
|
|
||||||
|
Renovate tracks it through the inline-marker idiom (`# renovate: datasource=github-releases
|
||||||
|
depName=...` immediately above the version literal), because `renovate.json` sets
|
||||||
|
`"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}` — an image reference in a
|
||||||
|
plain manifest is tracked by nothing. **Caveat:** upstream *release* tags carry a `v` (`v0.37.0`)
|
||||||
|
while *image* tags do not (`0.37.0`). Whether Renovate normalises that for this dep is unverified;
|
||||||
|
if it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
|
||||||
|
`renovate.json`, not a hand-edit.
|
||||||
|
|
||||||
|
## Ordering — do not reorder the stack
|
||||||
|
|
||||||
|
The renderer change (`catalog/render.py` emitting `gpu.intel.com/i915: 1` for
|
||||||
|
`hardware.gpu: intel-quicksync` instead of the hostPath) **must not land until this plugin is
|
||||||
|
merged AND confirmed advertising the resource on all three nodes.** If the renderer requests a
|
||||||
|
resource no node advertises, the Plex pod is **Unschedulable**; `apps-wave-0` is `wait: true` and
|
||||||
|
`apps-wave-1` `dependsOn` it, so that stalls the whole app tier — turning today's silent no-op into
|
||||||
|
a real outage.
|
||||||
|
|
||||||
|
## Post-merge verification
|
||||||
|
|
||||||
|
Nothing about this component's runtime behaviour can be verified from a PR: it does not exist until
|
||||||
|
the operator merges and Flux reconciles. Afterwards:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
D=./infra/devcontainer/dev.sh
|
||||||
|
|
||||||
|
# 1. Flux applied it, and (wait:true) the DaemonSet went healthy.
|
||||||
|
$D run bash -c 'flux get kustomization intel-gpu-plugin'
|
||||||
|
$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin get ds intel-gpu-plugin'
|
||||||
|
# want: DESIRED=3 CURRENT=3 READY=3 AVAILABLE=3
|
||||||
|
|
||||||
|
# 2. The resource is advertised on ALL THREE nodes. This is the gate on the renderer PR.
|
||||||
|
$D run bash -c 'kubectl --context k3s-prod get nodes -o json \
|
||||||
|
| jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
|
||||||
|
# want: three lines, each i915=1 (before this change: ABSENT on all three)
|
||||||
|
|
||||||
|
# 3. If a node shows ABSENT, read that node's plugin logs.
|
||||||
|
$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin logs -l app=intel-gpu-plugin --tail=50 --prefix'
|
||||||
|
```
|
||||||
|
|
||||||
|
### The decisive test
|
||||||
|
|
||||||
|
Resource-advertised is necessary but **not** sufficient — it proves the plugin registered, not that
|
||||||
|
the cgroup allow rule reaches Plex. The decisive test is re-running the `open()` probe **as uid 1000
|
||||||
|
through `s6-setuidgid plex`** inside the Plex pod, *after* the renderer PR has switched Plex to
|
||||||
|
request the resource:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
$D run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
|
||||||
|
s6-setuidgid plex python3 -c "import os; os.close(os.open(\"/dev/dri/renderD128\", os.O_RDWR)); print(\"rc=0 OK\")"'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Passing as uid 0 is weaker evidence and must not be accepted as proof** — Plex transcodes as uid
|
||||||
|
1000, and the two differ in group membership, which is the half the image's cont-init hook supplies.
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
---
|
||||||
|
# VENDORED, MACHINE-GENERATED, from intel/intel-device-plugins-for-kubernetes @ v0.37.0:
|
||||||
|
# deployments/gpu_plugin/base
|
||||||
|
#
|
||||||
|
# RE-VENDOR on a version bump (and bump `newTag` in kustomization.yaml to match):
|
||||||
|
# kubectl kustomize \
|
||||||
|
# "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/base?ref=vX.Y.Z" \
|
||||||
|
# > platform/intel-gpu-plugin/intel-gpu-plugin.yaml
|
||||||
|
# # then re-add this header, and READ THE DIFF — upstream has changed the securityContext
|
||||||
|
# # and the volume set between minor releases before.
|
||||||
|
#
|
||||||
|
# Generated 2026-10-01 at v0.37.0; sha256 of the body below (header excluded):
|
||||||
|
# 80ebd070f983f3116f4004f719d9c872220997ec766b4ab7e7290585ac8ef842
|
||||||
|
# The raw upstream source file reviewed to produce it
|
||||||
|
# (deployments/gpu_plugin/base/intel-gpu-plugin.yaml) was sha256
|
||||||
|
# c8773918c2370e711d4ba84120a9971a290bf75ecf81dc520782bcccd7ef24b6
|
||||||
|
# and the two were proven EQUIVALENT, not merely similar: `yq -P 'sort_keys(..)'` of each diffs
|
||||||
|
# clean, so passing it through kustomize changed formatting only and no content.
|
||||||
|
#
|
||||||
|
# WHY GENERATED RATHER THAN A BYTE COPY of the upstream file: upstream mixes YAML sequence
|
||||||
|
# indentation styles within the one document (`containers:` non-indented, `env:` indented), and
|
||||||
|
# ci/yamllint.yaml sets `indentation: indent-sequences: consistent`, so a verbatim copy is a hard
|
||||||
|
# lint ERROR and blocks the merge gate — measured, this file failed exactly that way first time
|
||||||
|
# round. `kubectl kustomize` normalises the output, which is also precisely the re-vendor idiom
|
||||||
|
# platform/external-snapshotter documents for the same reason.
|
||||||
|
#
|
||||||
|
# WHY VENDORED AT ALL rather than a remote kustomize base (`...gpu_plugin/base?ref=v0.37.0` left in
|
||||||
|
# `resources:`): the same reason platform/external-snapshotter is vendored. A remote base is
|
||||||
|
# `git fetch`ed by Flux's kustomize-controller on EVERY reconcile, those fetches intermittently
|
||||||
|
# exceeded the controller's 27s timeout, and a transient GitHub slowness then rendered as
|
||||||
|
# "kustomize build failed" on the whole tier. A local file makes the build OFFLINE and
|
||||||
|
# deterministic — which also means `kustomize build` in the CI gate proves the real thing instead
|
||||||
|
# of proving GitHub was up. (platform/kubevirt still carries the remote-base pattern and its latent
|
||||||
|
# fragility; not this component's problem.)
|
||||||
|
#
|
||||||
|
# DO NOT HAND-EDIT THE BODY. Every local change belongs in kustomization.yaml as a transformer or
|
||||||
|
# patch, so the next re-vendor stays a mechanical regenerate-and-diff instead of a three-way merge.
|
||||||
|
# The image tag below is re-pinned (and fully qualified) by the `images:` transformer there. The
|
||||||
|
# rationale for each thing deliberately NOT changed — no privileged, no supplementalGroups, no
|
||||||
|
# -shared-dev-num, no NFD overlay, no health-management — is in README.md.
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: DaemonSet
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: intel-gpu-plugin
|
||||||
|
name: intel-gpu-plugin
|
||||||
|
spec:
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: intel-gpu-plugin
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: intel-gpu-plugin
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- env:
|
||||||
|
- name: NODE_NAME
|
||||||
|
valueFrom:
|
||||||
|
fieldRef:
|
||||||
|
fieldPath: spec.nodeName
|
||||||
|
- name: HOST_IP
|
||||||
|
valueFrom:
|
||||||
|
fieldRef:
|
||||||
|
fieldPath: status.hostIP
|
||||||
|
image: intel/intel-gpu-plugin:0.37.0
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
|
name: intel-gpu-plugin
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 115Mi
|
||||||
|
requests:
|
||||||
|
cpu: 40m
|
||||||
|
memory: 45Mi
|
||||||
|
securityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
readOnlyRootFilesystem: true
|
||||||
|
seLinuxOptions:
|
||||||
|
type: container_device_plugin_t
|
||||||
|
seccompProfile:
|
||||||
|
type: RuntimeDefault
|
||||||
|
volumeMounts:
|
||||||
|
- mountPath: /dev/dri
|
||||||
|
name: devfs
|
||||||
|
readOnly: true
|
||||||
|
- mountPath: /sys/class/drm
|
||||||
|
name: sysfsdrm
|
||||||
|
readOnly: true
|
||||||
|
- mountPath: /var/lib/kubelet/device-plugins
|
||||||
|
name: kubeletsockets
|
||||||
|
- mountPath: /var/run/cdi
|
||||||
|
name: cdipath
|
||||||
|
nodeSelector:
|
||||||
|
kubernetes.io/arch: amd64
|
||||||
|
volumes:
|
||||||
|
- hostPath:
|
||||||
|
path: /dev/dri
|
||||||
|
name: devfs
|
||||||
|
- hostPath:
|
||||||
|
path: /sys/class/drm
|
||||||
|
name: sysfsdrm
|
||||||
|
- hostPath:
|
||||||
|
path: /var/lib/kubelet/device-plugins
|
||||||
|
name: kubeletsockets
|
||||||
|
- hostPath:
|
||||||
|
path: /var/run/cdi
|
||||||
|
type: DirectoryOrCreate
|
||||||
|
name: cdipath
|
||||||
|
updateStrategy:
|
||||||
|
rollingUpdate:
|
||||||
|
maxSurge: 0
|
||||||
|
maxUnavailable: 1
|
||||||
|
type: RollingUpdate
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
---
|
||||||
|
# Intel GPU device plugin for Kubernetes — makes the nodes advertise `gpu.intel.com/i915`.
|
||||||
|
#
|
||||||
|
# WHAT THIS FIXES, and why nothing cheaper works. Plex already has a `/dev/dri` hostPath mount
|
||||||
|
# (merged on main @ cdf3e14). The device node is VISIBLE in the container and cannot be OPENED.
|
||||||
|
# Measured in the running pod on k3s-prod-2:
|
||||||
|
# uid 0, O_RDWR -> EPERM "Operation not permitted"
|
||||||
|
# uid 1000, O_RDWR -> EPERM "Operation not permitted"
|
||||||
|
# EPERM from the DEVICE CGROUP, not EACCES from file permissions — proven three ways: root on the
|
||||||
|
# HOST opens the same device rc=0; the node is unified cgroup2fs; and the container's runc
|
||||||
|
# config.json carries one deny-all rule and nothing else:
|
||||||
|
# linux.resources.devices: [{'allow': False, 'access': 'rwm'}]
|
||||||
|
# linux.devices: None
|
||||||
|
# No allow entry for major 226, and the runtime was never asked to grant a device. A hostPath is a
|
||||||
|
# FILESYSTEM operation: it makes the device node appear, and adds no cgroup allow rule, because
|
||||||
|
# Kubernetes cannot infer one from a volume.
|
||||||
|
#
|
||||||
|
# A device plugin is the mechanism that closes exactly that gap. Verified by reading the v0.37.0
|
||||||
|
# source (cmd/gpu_plugin/gpu_plugin.go, createDeviceSpecsFromDrmFiles): the Allocate response
|
||||||
|
# carries `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, and it
|
||||||
|
# is that DeviceSpec which makes kubelet populate `linux.devices` AND the matching cgroup allow
|
||||||
|
# rule. It is the only NON-PRIVILEGED way to get there. `privileged: true` would also work and is
|
||||||
|
# not available: platform/kyverno/policies/restrict-host-namespaces.yaml blocks it under Enforce in
|
||||||
|
# app namespaces, and `plex` is labelled app-namespace=true (measured 2026-10-01). Correctly so.
|
||||||
|
#
|
||||||
|
# STANDALONE DAEMONSET, NOT THE OPERATOR. Upstream ships both. The operator adds a CRD
|
||||||
|
# (GpuDevicePlugin), a controller-manager Deployment, cert-manager-backed admission webhooks and a
|
||||||
|
# second namespace — i.e. three more things that can be unhealthy — to buy declarative management
|
||||||
|
# of a DaemonSet we are declaring in git anyway. For three identical nodes and one consumer that is
|
||||||
|
# strictly more blast radius for no gain. The DaemonSet has no CRDs, no controller, no webhook, and
|
||||||
|
# no API-server access at all: a device plugin talks to the LOCAL kubelet over a unix socket, which
|
||||||
|
# is why there is no ServiceAccount or RBAC here (and therefore no `**/rbac.yaml` sensitive path).
|
||||||
|
#
|
||||||
|
# LISTING, NOT OPENING — the thing worth being sure about, since opening is precisely what fails
|
||||||
|
# today. Verified in v0.37.0 source: the scan path uses only os.ReadDir / os.ReadFile / os.Stat /
|
||||||
|
# os.Readlink over /sys/class/drm and /dev/dri. There is no os.Open, no OpenFile, no ioctl and no
|
||||||
|
# syscall of any kind against a DRM node anywhere in cmd/gpu_plugin/gpu_plugin.go. Listing and
|
||||||
|
# stat() demonstrably work today (that is how the Plex probe found the device before failing to
|
||||||
|
# open it), so the plugin needs no cgroup grant of its own and no privilege.
|
||||||
|
# * `--health-management` is left OFF (upstream default). It is the one flag that WOULD need to
|
||||||
|
# open a device: it starts a Level-Zero sidecar (setupLevelZeroService returns early unless the
|
||||||
|
# flag is set). Turning it on would reintroduce the exact EPERM this component exists to fix.
|
||||||
|
# * /dev/dri and /sys/class/drm are mounted readOnly, consistent with list-and-stat only.
|
||||||
|
# * /dev/dri/by-path absence is non-fatal (newDevicePlugin warns and sets bypathFound=false).
|
||||||
|
#
|
||||||
|
# RESOURCE NAME: `gpu.intel.com/i915`. Not a guess — `scan()` does
|
||||||
|
# `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the KERNEL DRIVER name. These nodes
|
||||||
|
# have i915 bound (not xe) on UHD Graphics 630 / PCI 8086:3e98, so the resource is i915. A future
|
||||||
|
# kernel moving CoffeeLake to xe would silently rename it to `gpu.intel.com/xe` and the consumer's
|
||||||
|
# request would stop matching — worth knowing before that kernel arrives.
|
||||||
|
#
|
||||||
|
# `--shared-dev-num` LEFT AT THE UPSTREAM DEFAULT OF 1 (so: not set). The flag multiplies each
|
||||||
|
# card into N schedulable device IDs (`for i := 0; i < sharedDevNum; i++`), letting N containers
|
||||||
|
# share one GPU. Deliberately not raised:
|
||||||
|
# * Plex is the only consumer and wants one device.
|
||||||
|
# * The count is a LIE the scheduler believes. There is no GPU-side partitioning or quota behind
|
||||||
|
# it — N containers get the same renderD128 and contend for the same two media engines
|
||||||
|
# (vcs0/vecs0). Raising it converts "Unschedulable", which is loud and obvious, into
|
||||||
|
# "transcodes are mysteriously slow", which is neither.
|
||||||
|
# * It is a one-line, zero-migration change on the day a second consumer actually exists, and at
|
||||||
|
# that point the right number will be known rather than guessed.
|
||||||
|
# No NFD overlay either: all three nodes are identical (established cluster fact), so a nodeSelector
|
||||||
|
# on a GPU feature label would gate on something that is true everywhere. If a GPU-less node ever
|
||||||
|
# joins, add upstream's nfd overlay THEN rather than relying on the plugin's behaviour there.
|
||||||
|
#
|
||||||
|
# NO supplementalGroups, anywhere, on purpose. The pms-docker image's own cont-init hook stats each
|
||||||
|
# /dev/dri char device and adds the `plex` user to its group — measured live as `Groups: 44 100 992
|
||||||
|
# 1000` on the running Plex process, where GID 992 is the host `render` group. That mechanism is
|
||||||
|
# image-side and works identically on a plugin-injected device, so adding a supplementalGroup here
|
||||||
|
# or on the consumer would duplicate a thing that already works.
|
||||||
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||||
|
kind: Kustomization
|
||||||
|
namespace: intel-gpu-plugin
|
||||||
|
resources:
|
||||||
|
- namespace.yaml
|
||||||
|
- intel-gpu-plugin.yaml
|
||||||
|
images:
|
||||||
|
# PINNED, never `latest`. Upstream release v0.37.0 (2026-09-16); verified present on Docker Hub
|
||||||
|
# 2026-10-01, manifest digest sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1
|
||||||
|
# (single amd64 manifest, not an index). Recorded here rather than pinned by digest because this
|
||||||
|
# repo pins upstream platform components by VERSION TAG (platform/kubevirt, external-snapshotter);
|
||||||
|
# digest pinning is reserved for the conformance runner, where it is a security boundary.
|
||||||
|
# `newName` fully qualifies the registry: upstream writes the bare `intel/intel-gpu-plugin`, which
|
||||||
|
# relies on containerd's docker.io default rather than saying so.
|
||||||
|
#
|
||||||
|
# RENOVATE: the inline-marker idiom (customManagers in renovate.json), not the docker datasource —
|
||||||
|
# renovate.json sets `"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}`, so an
|
||||||
|
# image reference in a plain manifest is tracked by NOTHING. The marker must sit on the line
|
||||||
|
# IMMEDIATELY above the version literal for the regex to match, which is why it is down here on
|
||||||
|
# `newTag` and not up in this header.
|
||||||
|
# CAVEAT, stated because it will bite someone: the GitHub release tags carry a `v`
|
||||||
|
# (`v0.37.0`) and the IMAGE tags do not (`0.37.0` — confirmed against the Docker Hub tag list).
|
||||||
|
# Whether Renovate normalises that for this dep is UNVERIFIED here (no Renovate run available).
|
||||||
|
# If it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
|
||||||
|
# renovate.json, not a hand-edit of this line.
|
||||||
|
- name: intel/intel-gpu-plugin
|
||||||
|
newName: docker.io/intel/intel-gpu-plugin
|
||||||
|
# renovate: datasource=github-releases depName=intel/intel-device-plugins-for-kubernetes
|
||||||
|
newTag: "0.37.0"
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
---
|
||||||
|
# intel-gpu-plugin — namespace for the Intel GPU device plugin DaemonSet.
|
||||||
|
#
|
||||||
|
# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and that distinction is load-bearing here rather
|
||||||
|
# than cosmetic. Every Kyverno policy that could object to this DaemonSet is scoped by a
|
||||||
|
# `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"` — verified by
|
||||||
|
# reading the policies, not assumed:
|
||||||
|
# * platform/kyverno/policies/restrict-host-namespaces.yaml (hostNetwork/PID/IPC + privileged)
|
||||||
|
# * platform/kyverno/policies/require-egress-class-label.yaml
|
||||||
|
# * platform/kyverno/policies/require-auth-mode-label.yaml
|
||||||
|
# * platform/kyverno/policies/require-default-deny-baseline.yaml (a *generate* rule, so an
|
||||||
|
# app-namespace also gets a default-deny NetworkPolicy minted into it)
|
||||||
|
# Labelling this namespace "false" therefore puts it out of scope for all four. That is the same
|
||||||
|
# mechanism restrict-host-namespaces' own header describes: "platform components that do [need host
|
||||||
|
# access] live in platform namespaces that do NOT carry the app-namespace label".
|
||||||
|
#
|
||||||
|
# Measured on prod 2026-10-01: `kyverno` and `vpn-egress` are the two namespaces that carry the
|
||||||
|
# label explicitly as "false"; the rest of the platform namespaces leave it unset. Both are out of
|
||||||
|
# scope identically — an unset label does not match `"true"`. We set it EXPLICITLY, following
|
||||||
|
# vpn-egress, because "someone chose this" and "nobody thought about it" should not look the same
|
||||||
|
# in git, and because this component's correctness depends on the answer.
|
||||||
|
#
|
||||||
|
# NOTE this plugin does NOT actually need any of the things those policies forbid: it sets no
|
||||||
|
# hostNetwork/hostPID/hostIPC and is not privileged (see intel-gpu-plugin.yaml). The namespace
|
||||||
|
# placement is not a workaround for a policy — there is nothing to work around. It is here because
|
||||||
|
# a node-level platform DaemonSet is not an application, and because the app-namespace fabric
|
||||||
|
# (default-deny NetworkPolicy, egress-class, auth-mode) describes obligations that are meaningless
|
||||||
|
# for a process that talks only to the local kubelet over a unix socket and never uses the network.
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Namespace
|
||||||
|
metadata:
|
||||||
|
name: intel-gpu-plugin
|
||||||
|
labels:
|
||||||
|
homelab.keithjackson.dev/app-namespace: "false"
|
||||||
Reference in new issue
Block a user