jakman85andClaude Opus 5
4e1b169256
Makes the three prod nodes advertise `gpu.intel.com/i915`. This is the measured
fix for the Plex hardware-transcoding blocker, and it only makes the resource
EXIST — no consumer is switched over here.
WHY. The `/dev/dri` hostPath merged on main @ cdf3e14 makes the device node
visible and not openable. Measured in the running pod on k3s-prod-2:
uid 0, O_RDWR -> EPERM "Operation not permitted"
uid 1000, O_RDWR -> EPERM "Operation not permitted"
EPERM from the device cgroup, not EACCES from file permissions — root on the
HOST opens the same device rc=0, the node is unified cgroup2fs, and the
container's runc config.json carries one deny-all rule with `linux.devices:
None`. A hostPath is a filesystem operation: it makes the device node appear
and adds no cgroup allow rule, because Kubernetes cannot infer one from a
volume. A device plugin's Allocate response is what makes kubelet populate
`linux.devices` AND the matching cgroup allow rule — verified in the v0.37.0
source, which emits `DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}`
per DRM node. It is the only non-privileged way to get there; `privileged:
true` is blocked by Kyverno restrict-host-namespaces under Enforce in app
namespaces, and `plex` is labelled app-namespace=true.
PLACEMENT: its own Kustomization (dependsOn cilium, wait:true), not a line in
platform/kustomization.yaml. The shared tier was cheaper and needed no human
reviewer on approval-policy.json's terms, but it is wait:true over eleven-plus
components and the repo already argued this when Cilium was split out: its
Ready condition cannot tell "the plugin is wedged" from "longhorn is slow", the
only emergency valve is freezing the whole tier, and the tier's 10m/5m
interval/timeout are called out there as wrong for a 3-node DaemonSet rollout —
which is exactly what this is. Worse here than average: platform gates
platform-config gates apps-wave-0 (wait:true) gates apps-wave-1, so a plugin
that never went healthy inside it would stall every app — a component added to
fix a silent no-op would have gained the power to cause a real outage.
Touching clusters/** costs nothing extra: required_approvals:1 with
block_admin_merge_override:true applies to every PR here regardless of path.
wait:true is a health SIGNAL, not a gate — nothing in the rendered graph
dependsOn this, so it gates nothing, while making `flux get kustomization
intel-gpu-plugin` answer "did it come up on all three nodes".
NON-CHOICES, each argued in the README: standalone DaemonSet over the operator
(no CRD, controller-manager, webhook or second namespace to keep healthy);
`--shared-dev-num` left at the upstream default of 1 (a higher count is a lie
the scheduler believes — no GPU-side partitioning backs it, so it converts a
loud Unschedulable into a quiet "transcodes are slow"); `--health-management`
left OFF, the one flag that would open a device and reintroduce this EPERM; no
NFD overlay (all three nodes identical); no supplementalGroups (the pms-docker
cont-init hook already adds `plex` to each device's group, measured live as
Groups: 44 100 992 1000).
The DaemonSet is vendored as `kubectl kustomize` output of upstream v0.37.0's
gpu_plugin/base, rather than referenced as a remote kustomize base, for the
reason external-snapshotter records: a remote base is git-fetched on every
reconcile and those fetches have flaked a whole tier. It is generated rather
than byte-copied because upstream mixes YAML sequence-indentation styles within
the one document and ci/yamllint.yaml sets `indent-sequences: consistent`, so a
verbatim copy is a hard lint ERROR — measured, not guessed: the first gate run
of this branch returned rc 1 BLOCKED on exactly that. The transform is proven
formatting-only (`yq -P 'sort_keys(..)'` of both diffs clean). Image pinned to
docker.io/intel/intel-gpu-plugin:0.37.0 (digest verified present:
sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1).
ORDERING — DO NOT REORDER. catalog/render.py, catalog/plex.yaml and
apps/wave-0/plex.yaml are untouched. The renderer change that emits
`gpu.intel.com/i915: 1` instead of the hostPath must NOT land until this plugin
is merged AND confirmed advertising the resource on all three nodes: a request
for a resource no node advertises makes the Plex pod Unschedulable, and
apps-wave-0 is wait:true with apps-wave-1 depending on it.
VERIFIED: manifests render offline (component, clusters/base, both overlays);
no node currently advertises gpu.intel.com/* (queried — "NONE"); the plugin
only lists/stats devices and never opens one (no os.Open/OpenFile/ioctl/syscall
anywhere in its 908-line source); all four app-scoped Kyverno policies are
gated on app-namespace=true and the new namespace is labelled "false".
NOT VERIFIED: anything runtime. The plugin has never executed, the resource has
never been advertised, and the open() test has not been re-run. Gate returned
rc 3 (UNDETERMINED) with exactly the three expected skips — not green.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
platform/
Cluster-wide platform services, Flux-managed (R-PLAT-01, R-GITOPS-01/02/03). Reconciled
by the platform Flux Kustomization defined in clusters/base/platform.yaml,
before apps/.
Governing spec: docs/05-gitops-and-delivery.md,
docs/01-architecture-overview.md.
Phase 0 scaffolding only. Every component here is a structurally-correct stub — a Flux HelmRelease and/or Kustomization with a values placeholder, a README citing the governing ADR/spec, and TODOs for the phase that fills it in. No cluster exists; nothing reconciles yet. Do not add working config, secrets, or provisioning under Phase 0 (spec
AGENTS.md§1/§5/§6,roadmap/phases.md).
Reconciliation flow (how a change lands)
git (source of truth)
│ operator PR → CI (lint · schema · Kyverno L0 · Chainsaw) → merge to tracked branch
▼
Flux GitRepository (clusters/base/flux-system.yaml)
▼
Flux Kustomization "platform" → ./platform (this dir) ── substituteFrom cluster-vars ──┐
│ prune=true (drift correction, R-GITOPS-03) · wait=true │ R-ENV-02
▼ │
Flux Kustomization "apps" → ./apps (dependsOn platform) ─ substituteFrom cluster-vars ──┘
Git is authoritative; the cluster conforms to git and drift is auto-corrected (R-GITOPS-01/03).
Manual kubectl apply of platform state is prohibited (spec AGENTS.md §5).
The dev ↔ prod overlay model (R-ENV-02)
Platform manifests are authored once. Every value that legitimately differs between environments
is a ${TOKEN} resolved from the cluster-vars ConfigMap via Flux postBuild.substituteFrom. The
clusters/dev and clusters/prod overlays differ only by the values they patch into
cluster-vars (issuer, replica counts, storage sizes, retention). See
clusters/README.md. Base manifests are never forked per environment.
Components (one directory each)
| Dir | Component | Governing ADR / spec | Serves |
|---|---|---|---|
cilium/ |
CNI + kube-proxy replacement + Gateway API + Egress Gateway + Hubble | ADR-0002, docs/03 | R-NET-* |
longhorn/ |
Replicated block storage; 3-replica default StorageClass + bulk tier | ADR-0003, docs/04 | R-STATE-* |
cert-manager/ |
TLS via Let's Encrypt DNS-01 (Cloudflare); staging + prod ClusterIssuers | ADR-0006, docs/08 | R-TLS-* |
externaldns/ |
Automatic DNS records for public services (Cloudflare) | docs/09 | R-DNS-* |
gateway/ |
Gateway API GatewayClass + shared HTTPS Gateway + wildcard cert | ADR-0002/ADR-0006, docs/09 | R-DNS-01, R-TLS-01 |
monitoring/ |
kube-prometheus-stack + Loki + Grafana Alloy + Alertmanager | docs/10 | R-OBS-* |
velero/ |
Off-cluster backup/DR of volumes + resource state | ADR-0003, docs/04 | R-STATE-03 |
kubevirt/ |
KubeVirt + CDI (IaaS: application VMs on the same K8s surface) | ADR-0005, docs/06 | R-PLAT-03 |
image-automation/ |
Flux image-update-automation (hands-off image tag advance) | ADR-0009, docs/11 | R-UPD-02 |
keycloak/ |
Platform IdP (OIDC + SAML), config-as-code, PostgreSQL on Longhorn | ADR-0014, docs/15 | R-IDP-* |
oauth2-proxy/ |
Forward-auth at the Gateway for non-OIDC apps → Keycloak | ADR-0014, docs/15 | R-IDP-* |
kyverno/ |
L0 policy-as-code gate — OWNED BY THE POLICY ENGINEER (WS-TEST). | ADR-0011, docs/14 | R-TEST-* |
⚠️ INTEGRATION TODO(PM): wire kyverno into the platform kustomization
platform/kyverno/ is authored by a different agent (the policy engineer, WS-TEST). This
workstream did not create any files under platform/kyverno/ and must not.
platform/kustomization.yaml intentionally omits - kyverno. Once the
policy engineer's files land, the PM must add - kyverno to that list (recommended first, so
policies are present before other components reconcile). The line is present as a commented TODO in
the kustomization.
Secrets
Every secret referenced below (Cloudflare API token, Velero backup-target creds, Keycloak admin/DB, oauth2-proxy cookie/client secrets) is referenced by name only and will be committed SOPS-encrypted (WS-SEC, ADR-0007). No plaintext secret appears anywhere in this tree.
Cross-workstream contracts surfaced by these stubs
- cilium (WS-NET ↔ WS-INFRA): Cilium's initial CNI install is done by Ansible at cluster bootstrap (docs/02, Phase 1). The HelmRelease here manages Cilium config thereafter; its Helm values MUST match the Ansible bootstrap values (kubeProxyReplacement, k8sServiceHost/Port, API server host) or reconciliation will fight the bootstrap. Contract owner: WS-NET + WS-INFRA.
- gateway + cert-manager + externaldns (WS-NET + WS-SEC): the shared Gateway listener, the
wildcard
*.home.keithjackson.devcert, and the ExternalDNS source all reference each other; they form the "Issuer + Gateway TLS" interface inroadmap/workstreams.md. - image-automation (WS-UPDATE): this dir holds only the cluster-wide ImageUpdateAutomation +
git write-back. Per-service
ImageRepository/ImagePolicyare rendered from each ServiceDefinition'supdateblock (WS-GITOPS/WS-UPDATE), living withapps/. - keycloak/oauth2-proxy (WS-SEC): the SSO/auth contract (
authblock → Keycloak client / oauth2-proxy protection) is owned by WS-SEC; these are placeholders pending that contract.