jakman85andClaude Opus 5
4e1b169256
Makes the three prod nodes advertise `gpu.intel.com/i915`. This is the measured
fix for the Plex hardware-transcoding blocker, and it only makes the resource
EXIST — no consumer is switched over here.
WHY. The `/dev/dri` hostPath merged on main @ cdf3e14 makes the device node
visible and not openable. Measured in the running pod on k3s-prod-2:
uid 0, O_RDWR -> EPERM "Operation not permitted"
uid 1000, O_RDWR -> EPERM "Operation not permitted"
EPERM from the device cgroup, not EACCES from file permissions — root on the
HOST opens the same device rc=0, the node is unified cgroup2fs, and the
container's runc config.json carries one deny-all rule with `linux.devices:
None`. A hostPath is a filesystem operation: it makes the device node appear
and adds no cgroup allow rule, because Kubernetes cannot infer one from a
volume. A device plugin's Allocate response is what makes kubelet populate
`linux.devices` AND the matching cgroup allow rule — verified in the v0.37.0
source, which emits `DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}`
per DRM node. It is the only non-privileged way to get there; `privileged:
true` is blocked by Kyverno restrict-host-namespaces under Enforce in app
namespaces, and `plex` is labelled app-namespace=true.
PLACEMENT: its own Kustomization (dependsOn cilium, wait:true), not a line in
platform/kustomization.yaml. The shared tier was cheaper and needed no human
reviewer on approval-policy.json's terms, but it is wait:true over eleven-plus
components and the repo already argued this when Cilium was split out: its
Ready condition cannot tell "the plugin is wedged" from "longhorn is slow", the
only emergency valve is freezing the whole tier, and the tier's 10m/5m
interval/timeout are called out there as wrong for a 3-node DaemonSet rollout —
which is exactly what this is. Worse here than average: platform gates
platform-config gates apps-wave-0 (wait:true) gates apps-wave-1, so a plugin
that never went healthy inside it would stall every app — a component added to
fix a silent no-op would have gained the power to cause a real outage.
Touching clusters/** costs nothing extra: required_approvals:1 with
block_admin_merge_override:true applies to every PR here regardless of path.
wait:true is a health SIGNAL, not a gate — nothing in the rendered graph
dependsOn this, so it gates nothing, while making `flux get kustomization
intel-gpu-plugin` answer "did it come up on all three nodes".
NON-CHOICES, each argued in the README: standalone DaemonSet over the operator
(no CRD, controller-manager, webhook or second namespace to keep healthy);
`--shared-dev-num` left at the upstream default of 1 (a higher count is a lie
the scheduler believes — no GPU-side partitioning backs it, so it converts a
loud Unschedulable into a quiet "transcodes are slow"); `--health-management`
left OFF, the one flag that would open a device and reintroduce this EPERM; no
NFD overlay (all three nodes identical); no supplementalGroups (the pms-docker
cont-init hook already adds `plex` to each device's group, measured live as
Groups: 44 100 992 1000).
The DaemonSet is vendored as `kubectl kustomize` output of upstream v0.37.0's
gpu_plugin/base, rather than referenced as a remote kustomize base, for the
reason external-snapshotter records: a remote base is git-fetched on every
reconcile and those fetches have flaked a whole tier. It is generated rather
than byte-copied because upstream mixes YAML sequence-indentation styles within
the one document and ci/yamllint.yaml sets `indent-sequences: consistent`, so a
verbatim copy is a hard lint ERROR — measured, not guessed: the first gate run
of this branch returned rc 1 BLOCKED on exactly that. The transform is proven
formatting-only (`yq -P 'sort_keys(..)'` of both diffs clean). Image pinned to
docker.io/intel/intel-gpu-plugin:0.37.0 (digest verified present:
sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1).
ORDERING — DO NOT REORDER. catalog/render.py, catalog/plex.yaml and
apps/wave-0/plex.yaml are untouched. The renderer change that emits
`gpu.intel.com/i915: 1` instead of the hostPath must NOT land until this plugin
is merged AND confirmed advertising the resource on all three nodes: a request
for a resource no node advertises makes the Plex pod Unschedulable, and
apps-wave-0 is wait:true with apps-wave-1 depending on it.
VERIFIED: manifests render offline (component, clusters/base, both overlays);
no node currently advertises gpu.intel.com/* (queried — "NONE"); the plugin
only lists/stats devices and never opens one (no os.Open/OpenFile/ioctl/syscall
anywhere in its 908-line source); all four app-scoped Kyverno policies are
gated on app-namespace=true and the new namespace is labelled "false".
NOT VERIFIED: anything runtime. The plugin has never executed, the resource has
never been advertised, and the open() test has not been re-run. Gate returned
rc 3 (UNDETERMINED) with exactly the three expected skips — not green.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
35 lines
2.2 KiB
YAML
35 lines
2.2 KiB
YAML
---
|
|
# intel-gpu-plugin — namespace for the Intel GPU device plugin DaemonSet.
|
|
#
|
|
# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and that distinction is load-bearing here rather
|
|
# than cosmetic. Every Kyverno policy that could object to this DaemonSet is scoped by a
|
|
# `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"` — verified by
|
|
# reading the policies, not assumed:
|
|
# * platform/kyverno/policies/restrict-host-namespaces.yaml (hostNetwork/PID/IPC + privileged)
|
|
# * platform/kyverno/policies/require-egress-class-label.yaml
|
|
# * platform/kyverno/policies/require-auth-mode-label.yaml
|
|
# * platform/kyverno/policies/require-default-deny-baseline.yaml (a *generate* rule, so an
|
|
# app-namespace also gets a default-deny NetworkPolicy minted into it)
|
|
# Labelling this namespace "false" therefore puts it out of scope for all four. That is the same
|
|
# mechanism restrict-host-namespaces' own header describes: "platform components that do [need host
|
|
# access] live in platform namespaces that do NOT carry the app-namespace label".
|
|
#
|
|
# Measured on prod 2026-10-01: `kyverno` and `vpn-egress` are the two namespaces that carry the
|
|
# label explicitly as "false"; the rest of the platform namespaces leave it unset. Both are out of
|
|
# scope identically — an unset label does not match `"true"`. We set it EXPLICITLY, following
|
|
# vpn-egress, because "someone chose this" and "nobody thought about it" should not look the same
|
|
# in git, and because this component's correctness depends on the answer.
|
|
#
|
|
# NOTE this plugin does NOT actually need any of the things those policies forbid: it sets no
|
|
# hostNetwork/hostPID/hostIPC and is not privileged (see intel-gpu-plugin.yaml). The namespace
|
|
# placement is not a workaround for a policy — there is nothing to work around. It is here because
|
|
# a node-level platform DaemonSet is not an application, and because the app-namespace fabric
|
|
# (default-deny NetworkPolicy, egress-class, auth-mode) describes obligations that are meaningless
|
|
# for a process that talks only to the local kubelet over a unix socket and never uses the network.
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: intel-gpu-plugin
|
|
labels:
|
|
homelab.keithjackson.dev/app-namespace: "false"
|