fix(ci): catalog/render.py's 106 selftest assertions never ran in the gate

`catalog/render.py --selftest` executes 106 assertions on the renderer's output contract. Nothing in ci/ or .gitea/ had ever invoked it — grepped and confirmed — so all 106 were dev-time only, and the merge gate could not tell a sound renderer from a gutted one. Recorded as the top follow-up of decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md. `ci/render-golden.sh` now runs it and exits 1 when it fails. One file, six lines of invocation; the rest is the rationale comment. WHY HERE. The lead was auditability-proofs-producible/check.sh:291, which drives conformance/lib/{record_run,status}.py --selftest "because nothing else drives them". That is the wrong hook: the check serves R-SEC-03 and its subject is the evidence chain — can a forged or stale run become an audit proof? The renderer is not in that chain, and hanging it there would make the check's name, its intent annotation and its requirement mapping untrue. This stage's subject already IS catalog/render.py, so the renderer's own contract test belongs in it. The two assertions are independent, and that is the point: render-golden catches golden-vs-renderer drift but is satisfied by ANY renderer whose output was re-rendered — including one that just lost a contract invariant and had all ten goldens regenerated in the same commit. That is precisely the hole the follow-up named. The selftest catches it; render-golden cannot. It runs before the empty-catalog exit 3 because the selftest renders synthetic ServiceDefinitions in memory and is catalog-independent — an empty catalog must not be able to skip it. Failure is a hard rc 1, never a skip: the renderer is source in this repo, so once python3 and render.py are both present (asserted above it) there is no legitimate environmental cause for "could not run". PROVEN TO FAIL, not merely to pass. Against mutated COPIES of the renderer via the stage's existing RENDER_PY override, so catalog/render.py — another branch's file this session — was never touched: * deleting the --cookie-name emission (reverting PR #28's fleet-wide cookie fix) -> stage rc 1; * OAUTH2_COOKIE_REFRESH 2m30s -> 10m -> stage rc 1 with the named assertion "parses to 600.0s, which is not inside (0, 300s)" — one of the two flags the cookie entry recorded as UNGATED, and one render-golden's diff would miss. AND PROVEN TO PROPAGATE, in the pinned toolbox: selftest rc 1 -> stage render-golden FAIL -> `MERGE GATE: BLOCKED`, gate.sh rc 1. Worth provoking: this repo flattens exit codes, and .gitea/workflows/gate.yml:138 runs `./ci/gate.sh || [ "$?" -eq 3 ]`, so a failure degraded to a skip would be swallowed by CI. render.py itself is unchanged. No conformance/ path is touched, no gate stage is added, nothing is deployed by this. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-01 21:21:49 -05:00
20 changed files with 261 additions and 1727 deletions
@@ -143,8 +143,6 @@ spec:
        - name: media
          mountPath: /media
          readOnly: true
        - name: dri
          mountPath: /dev/dri
      volumes:
      - name: config
        persistentVolumeClaim:
@@ -152,10 +150,6 @@ spec:
      - name: media
        persistentVolumeClaim:
          claimName: plex-media
      - name: dri
        hostPath:
          path: /dev/dri
          type: DirectoryOrCreate
 ---
 # Service
 apiVersion: v1
@@ -80,58 +80,6 @@ env:
  # remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
  # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
  # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
 # HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
 # INTENT ONLY. This says "this workload needs an Intel Quick Sync render device"; it does NOT name a
 # device path. The renderer currently satisfies it with exactly two things — a `/dev/dri` hostPath
 # volume (`type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container —
 # and nothing else: no nodeSelector, no privileged, no resources (schemas/renderer-contract.md
 # `### hardware`). If that mechanism is later replaced by the Intel GPU device plugin
 # (`gpu.intel.com/i915`), it is a `catalog/render.py` change and THIS BLOCK DOES NOT MOVE. Do not
 # "simplify" it into a raw `/dev/...` path; that is the whole point of the enum.
 #
 # THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
 # anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
 # GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
 # `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Plex happens to run on k3s-prod-2 today, but
 # that is scheduler history rather than a requirement — this field deliberately renders no
 # nodeSelector, because placement is a separate concern with its own justification.
 #
 # NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
 # The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
 # the host `render` GID in as a supplemental group. It cannot work here. MEASURED IN THE RUNNING
 # CONTAINER, 2026-09-30: the image's own
 # `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device under
 # `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
 # `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
 # GID renumbering self-healing (no GID is pinned anywhere in this repo). The service then execs via
 # `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from `/etc/group` rather
 # than appending to it: `Plex Media Server` runs with `Groups: 100 1000`, and root's gid `0` — which
 # PID 1 (`s6-svscan`) does hold — is GONE. Anything Kubernetes supplies therefore reaches s6-svscan
 # and is discarded before Plex ever starts. Mounting the device is sufficient; the group is the
 # image's job and it does it correctly. `render.py --selftest` asserts the ABSENCE of a pod
 # securityContext for this field precisely so nobody adds an inert one back later.
 #
 # `DirectoryOrCreate` IS THE SAFE FAILURE MODE, not laziness. A node that comes back without the
 # device (rebuild, BIOS reset, driver regression, hardware swap) gets an empty directory and Plex
 # degrades to SOFTWARE transcoding. `Directory` would fail the mount instead — and because
 # `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that would stall the WHOLE app
 # tier for a feature that is not load-bearing. The price, stated plainly: a missing device is
 # SILENT. This SD renders no `metricsEndpoint`, so there is no ServiceMonitor, no transcode-mode
 # metric, and no alert that could ever fire. The only proof is an operator reading the Plex
 # dashboard.
 #
 # THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Mounting the device does
 # not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
 # is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
 # set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
 # `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin`) found ZERO `PLEX_PREFERENCE*` support,
 # and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
 # `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
 # unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
 # `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took and the
 # device-cgroup failure mode to expect first.
 hardware:
  gpu: intel-quicksync
 exposure: public              # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
@@ -163,15 +163,6 @@ OAUTH2_COOKIE_REFRESH = "2m30s"
 # are added. Do not read this flag as "metrics are working".
 OAUTH2_PROXY_METRICS_PORT = 44180
 # --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
 # The ServiceDefinition declares the CAPABILITY (schemas/servicedefinition.schema.json
 # `hardware.gpu`); everything below is the MECHANISM that currently satisfies it and is
 # deliberately confined to this file. Swapping to the Intel GPU device plugin
 # (`gpu.intel.com/i915` resource requests instead of a hostPath) is a change here and
 # nowhere else -- no catalog entry moves.
 DRI_VOLUME_NAME = "dri"
 DRI_DEVICE_DIR = "/dev/dri"
 # =============================================================================
 # Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping.
@@ -522,46 +513,6 @@ def r_deployment(s: dict) -> dict:
                "persistentVolumeClaim": {"claimName": f"{s['name']}-{vol['name']}"},
            }
        )
    # hardware capability (schema-optional): `hardware.gpu: intel-quicksync` means this
    # workload needs the host iGPU's render node for hardware video transcode. The SD says
    # only that; the wiring below is the platform's current mechanism for satisfying it.
    #
    # `type: DirectoryOrCreate` is DELIBERATE and load-bearing -- do NOT "tighten" it to
    # `Directory`. A node that comes back without the render device (rebuild, BIOS change,
    # driver regression, hardware swap) must degrade the workload to SOFTWARE transcoding,
    # not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` dependsOn it, so a
    # failed hostPath mount stalls the entire app tier for the sake of a feature that is not
    # load-bearing. `DirectoryOrCreate` turns that failure mode into "transcodes in
    # software" instead of "the app tier does not reconcile". /dev is tmpfs, so the empty
    # directory the kubelet creates on such a node leaves nothing durable behind.
    #
    # The mount is deliberately NOT readOnly: VA-API opens the render node read-write.
    #
    # `podSecurityContext.supplementalGroups` is DELIBERATELY NOT SET here, and adding it
    # would be a regression, not a hardening. MEASURED in the running container on prod-2,
    # 2026-09-30: the image's own cont-init hook
    # `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` `find`s every char device
    # under /dev/dri, `stat -c '%g'` each one's GID, `groupadd`s a matching group INSIDE the
    # container when none exists, and `usermod -a -G` the app user into it -- every start,
    # nothing hardcoded. cont-init.d completes before services.d, so the account database
    # s6 reads at exec time already carries the group. The service is then started with
    # `exec s6-setuidgid plex ...`, and `s6-setuidgid` REPLACES the process group list from
    # /etc/group rather than appending to it: `Plex Media Server` runs with
    # `Uid: 1000  Gid: 1000  Groups: 100 1000` -- root's gid 0, which PID 1 (s6-svscan) does
    # have, is GONE. Anything Kubernetes puts in supplementalGroups reaches s6-svscan and is
    # then discarded before the app ever starts. MOUNTING THE DEVICE IS SUFFICIENT; the
    # group handling is the image's job and it does it correctly. A supplementalGroups entry
    # here would be inert AND would mislead whoever debugs this next.
    hw_gpu = (s.get("hardware") or {}).get("gpu")
    if hw_gpu == "intel-quicksync":
        volume_mounts.append({"name": DRI_VOLUME_NAME, "mountPath": DRI_DEVICE_DIR})
        volumes.append(
            {
                "name": DRI_VOLUME_NAME,
                "hostPath": {"path": DRI_DEVICE_DIR, "type": "DirectoryOrCreate"},
            }
        )
    if volume_mounts:
        container["volumeMounts"] = volume_mounts
@@ -2579,83 +2530,6 @@ def _selftest() -> int:
        "an SD without podSecurityContext must not gain a pod securityContext",
    )
    # 6b) hardware.gpu: intel-quicksync -> a /dev/dri hostPath volume and a read-write mount
    #     on the app container, and NOTHING ELSE. The SD declares the CAPABILITY; these
    #     assertions pin the MECHANISM that currently satisfies it -- including what it
    #     deliberately does NOT emit -- so a change to it is a deliberate, visible edit.
    hw_docs = render(
        {
            "name": "hw",
            "source": {"image": "a:1"},
            "hardware": {"gpu": "intel-quicksync"},
        }
    )
    hw_pod = next(
        d[1] for d in hw_docs if d[1]["kind"] == "Deployment"
    )["spec"]["template"]["spec"]
    check(
        {
            "name": "dri",
            "hostPath": {"path": "/dev/dri", "type": "DirectoryOrCreate"},
        }
        in hw_pod.get("volumes", []),
        "hardware.gpu=intel-quicksync must render a /dev/dri hostPath volume "
        "with type DirectoryOrCreate (a GPU-less node must degrade, not fail the mount)",
    )
    hw_mounts = hw_pod["containers"][0].get("volumeMounts", [])
    check(
        {"name": "dri", "mountPath": "/dev/dri"} in hw_mounts,
        "hardware.gpu=intel-quicksync must mount /dev/dri READ-WRITE on the app container",
    )
    # The device mount is the WHOLE feature: no pod securityContext is invented. The image's
    # own cont-init hook adds the app user to the device's group inside the container, and
    # s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway (see the
    # emission site). Asserting the absence is what stops someone adding an inert one back.
    check(
        "securityContext" not in hw_pod,
        "hardware.gpu must NOT invent a pod securityContext (supplementalGroups is inert "
        "under s6-setuidgid; the image's cont-init hook handles the device group)",
    )
    # Nothing privileged, and no node pinning: Kyverno restrict-host-namespaces (Enforce)
    # rejects `privileged: true` in an app namespace, and placement is a separate concern
    # with its own justification -- this field is about the DEVICE only.
    check(
        "privileged" not in hw_pod.get("securityContext", {})
        and "securityContext" not in hw_pod["containers"][0]
        and "nodeSelector" not in hw_pod,
        "hardware.gpu must emit no privileged securityContext and no nodeSelector",
    )
    # The regression that matters: absence renders NOTHING. Mirrors the podSecurityContext
    # negative above -- an SD that does not opt in must be byte-identical to before.
    nohw_pod = next(
        d[1] for d in render({"name": "nohw", "source": {"image": "a:1"}})
        if d[1]["kind"] == "Deployment"
    )["spec"]["template"]["spec"]
    check(
        "volumes" not in nohw_pod
        and "volumeMounts" not in nohw_pod["containers"][0]
        and "securityContext" not in nohw_pod,
        "an SD without `hardware` must gain no dri volume, no mount and no securityContext",
    )
    # An SD that declares BOTH keeps its own podSecurityContext exactly as written -- the
    # hardware field neither adds to it nor rewrites it.
    hwm_pod = next(
        d[1]
        for d in render(
            {
                "name": "hwm",
                "source": {"image": "a:1"},
                "hardware": {"gpu": "intel-quicksync"},
                "podSecurityContext": {"fsGroup": 1001, "supplementalGroups": [44]},
            }
        )
        if d[1]["kind"] == "Deployment"
    )["spec"]["template"]["spec"]
    check(
        hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]},
        "hardware.gpu must leave the SD's own podSecurityContext untouched",
    )
    # 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that
    #     fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged.
    fa_docs = render(
@@ -1,5 +1,10 @@
 #!/usr/bin/env bash
-# ci/render-golden.sh — Stage: render==golden reproducibility gate.
+# ci/render-golden.sh — Stage: the renderer is trustworthy.
 #   (1) catalog/render.py passes its OWN output-contract self-test, and
 #   (2) every committed apps/ golden reproduces byte-for-byte from the canonical render.
 # Two independent assertions about one subject, catalog/render.py. (1) was added
 # 2026-10-01 because nothing in ci/ or .gitea/ had ever invoked `--selftest`; see the
 # block above the invocation for why it lives here and not in a conformance check.
 #
 # WHY: the committed apps/ tree is a set of GOLDENS produced by ONE canonical
 # invocation of catalog/render.py. If a golden is hand-edited, or a catalog
@@ -54,6 +59,54 @@ if [ ! -f "$RENDER_PY" ]; then
  summary "- render-golden: **SKIPPED** (renderer missing)"
  exit 3
 fi
 # --- The renderer's OWN self-test, before anything is compared ---------------
 # `catalog/render.py --selftest` is ~106 executed assertions on the renderer's output
 # contract (default-deny covers both directions, the Kyverno namespace hook label, a valid
 # auth-mode, the oauth2-proxy sidecar's cookie-name/-domain/-refresh flags, the
 # placeholder-substitution guard, the observability refusals...). Until now NOTHING in ci/
 # or .gitea/ invoked it — measured by grep on 2026-09-30 and recorded as the top follow-up
 # of decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md — so every one of those
 # assertions was dev-time only and a merge gate could not tell a sound renderer from a
 # gutted one.
 #
 # WHY HERE, AND NOT IN A CONFORMANCE CHECK. The lead was
 # conformance/tests/l0-static/auditability-proofs-producible/check.sh, which drives
 # `conformance/lib/{record_run,status}.py --selftest` "because nothing else drives them".
 # That loop is NOT the right hook: that check serves R-SEC-03 and its subject is the
 # EVIDENCE CHAIN (can a forged or stale run become a proof?). The renderer is not part of
 # that chain, and hanging it there would make the check's name and its requirement mapping
 # a lie. This stage's subject already IS catalog/render.py — it exists to assert that the
 # renderer's output is trustworthy — so the renderer's own contract test belongs in it.
 #
 # It runs BEFORE the catalog/empty-catalog exit-3 below on purpose: the self-test is
 # deterministic and catalog-independent (it renders synthetic ServiceDefinitions in
 # memory), so an empty catalog must not be able to skip it.
 #
 # render-golden and the self-test are INDEPENDENT assertions, which is why both are needed:
 # render-golden catches golden-vs-renderer drift but is satisfied by ANY renderer whose
 # output was re-rendered, including one that just lost a contract invariant and had all ten
 # goldens regenerated in the same commit. The self-test catches exactly that.
 #
 # FAILURE IS A HARD FAIL, NOT A SKIP: the renderer is source in this repo, so "could not
 # run its self-test" has no legitimate environmental cause once python3 and render.py are
 # both present (both asserted above). ci/gate.sh maps this stage's rc 1 to BLOCKED.
 log "Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)"
 selftest_out="$(mktemp)"
 if "$PYTHON" "$RENDER_PY" --selftest >"$selftest_out" 2>&1; then
  ok "render.py --selftest passed"
  summary "- render-selftest: **passed** (\`catalog/render.py --selftest\`)"
 else
  err "catalog/render.py --selftest FAILED — the renderer violates its own output contract."
  err "  Every golden under apps/ is produced by this renderer, so the goldens below cannot"
  err "  be trusted even if they reproduce byte-for-byte. Reproduce with:"
  err "    ${PYTHON} catalog/render.py --selftest"
  sed 's/^/    /' "$selftest_out" >&2 || true
  summary "- render-selftest: **FAILED** (\`catalog/render.py --selftest\` — renderer contract violated)"
  rm -f "$selftest_out"
  exit 1
 fi
 rm -f "$selftest_out"
 if [ ! -d "$CATALOG_DIR" ] || ! compgen -G "${CATALOG_DIR}/*.yaml" >/dev/null 2>&1; then
  skip "no catalog ServiceDefinitions under '${CATALOG_DIR}' — nothing to render."
  summary "- render-golden: **SKIPPED** (empty catalog)"
@@ -1,104 +0,0 @@
 ---
 # Flux Kustomization: Intel GPU device plugin — reconciles ./platform/intel-gpu-plugin.
 # Makes the three prod nodes advertise `gpu.intel.com/i915`, which is the resource a later PR will
 # have `hardware.gpu: intel-quicksync` request in place of today's non-working /dev/dri hostPath.
 #
 # WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
 #
 # The cheap option was a line in the shared `platform` tier plus a new platform/ directory, and
 # neither path is sensitive in infra/gitea/approval-policy.json, so it would have needed no human
 # reviewer on the policy's terms. It is still the wrong place, for the reason the repo already
 # wrote down in platform/kustomization.yaml when Cilium was split out of that tier:
 #
 #   * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot tell
 #     you "the GPU plugin is wedged" from "longhorn is slow".
 #   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
 #     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device
 #     plugin.
 #   * The tier's 10m interval / 5m timeout are explicitly called out there as WRONG for a 3-node
 #     DaemonSet rollout. This is a 3-node DaemonSet rollout.
 #
 # And the consequence is worse here than for an average component, because of what sits downstream:
 # `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates apps-wave-1.
 # A plugin DaemonSet that never went healthy inside that tier would stall every app on "dependency
 # not ready" — i.e. a component added to fix a SILENT NO-OP (software transcoding) would have
 # acquired the ability to cause a REAL OUTAGE. That trade is the whole argument.
 #
 # The usual objection to this placement is the approval cost, and it does not apply: Gitea enforces
 # `required_approvals: 1` with `block_admin_merge_override: true` on EVERY PR here regardless of
 # path, and authors cannot self-approve. So this change needed the operator's approval either way;
 # touching clusters/** costs nothing extra and buys a blast radius of exactly one component.
 #
 # dependsOn: [cilium], WHICH IS THE MINIMUM AND ALSO THE EARLIEST
 #
 # Not platform-config (what headlamp/crafty/homepage/oauth2-proxy use) — those are leaf UIs that
 # genuinely need Keycloak + wildcard TLS + split-horizon DNS + Gateway. This plugin needs none of
 # it. It never uses the network and never talks to the API server; it talks to the local kubelet
 # over a unix socket. Its only real prerequisite is a working CNI, so its pod can get a sandbox at
 # all, and `cilium` already depends on `flux-system` (the root Kustomization that applies the
 # cluster-vars ConfigMap this one substitutes from), so that edge is transitively satisfied too.
 #
 # Depending on `cilium` rather than `platform` also gets the resource advertised EARLIEST — this
 # reconciles in parallel with the `platform` tier instead of behind all eleven of its components.
 # That matters on a cold start once the consumer PR lands: a Plex pod requesting a resource no node
 # advertises is Unschedulable, and apps-wave-0 is wait:true. Reconciling early shrinks that window
 # instead of widening it. It is not a GUARANTEE — see the ordering note below, which is a real
 # constraint on the next PR and not a formality.
 #
 # Kyverno (in `platform`) is deliberately NOT a dependency: every policy that could match this
 # workload is scoped to namespaces labelled app-namespace=true, and this namespace is labelled
 # "false" (platform/intel-gpu-plugin/namespace.yaml). So there is no admission race to lose.
 #
 # wait: true — A HEALTH SIGNAL, NOT A GATE. This is the one place this component departs from the
 # headlamp/crafty leaf template, and the departure is deliberate. Those run wait:false because a
 # leaf UI can be legitimately not-Ready for a long time during a cold start while it waits on
 # Keycloak/DNS/TLS, so its health is not a useful signal. This DaemonSet has no such dependency: it
 # reads sysfs and opens a socket, so on healthy nodes it should be Ready in seconds, and
 # not-Ready genuinely means BROKEN. wait:true turns `flux get kustomization intel-gpu-plugin` into
 # a one-line answer to "did the plugin come up on all three nodes", which is exactly the question
 # the operator must answer before the consumer PR may merge.
 #
 # NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
 # plugin is this Kustomization's own Ready condition and nothing else. Anyone later adding
 # `dependsOn: [intel-gpu-plugin]` is choosing to make plugin health gate their thing; for the app
 # waves that would hand a device plugin the power to stall every app, which is the thing this file
 # exists to prevent. Think before adding that edge.
 #
 # ORDERING — DO NOT REORDER THE STACK. The renderer change (catalog/render.py emitting
 # `gpu.intel.com/i915: 1` for `hardware.gpu: intel-quicksync` instead of the hostPath) must NOT
 # land until this plugin is merged AND confirmed advertising the resource on all three nodes. If
 # the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is
 # wait:true and apps-wave-1 dependsOn it, so that stalls the whole app tier — turning today's silent
 # no-op into a real outage. This PR's only job is to make the resource exist.
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
  name: intel-gpu-plugin
  namespace: flux-system
 spec:
  # Static vendored manifests with nothing to re-render; kustomize-controller is source-triggered,
  # so this is only the drift-correction floor.
  interval: 10m
  retryInterval: 2m
  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
  # exceed 5m on a cold start (image pull on three nodes), and the Cilium split-out comment calls
  # out that exact mismatch as one of its reasons for leaving the shared tier.
  timeout: 10m
  prune: true
  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
  wait: true
  dependsOn:
    - name: cilium   # the CNI — the only real prerequisite; transitively gives us flux-system too
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./platform/intel-gpu-plugin
  # Retained for parity with every other tier / future overlay knobs. This component references no
  # ${VARS} today, and its manifests contain no `$` literals for envsubst to eat (unlike
  # clusters/base/runbooks.yaml, which omits substitution for exactly that reason).
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
  # No SOPS decryption patch: this component carries no Secret, so clusters/dev adds nothing for it
  # (same as homepage/runbooks, unlike headlamp/crafty/oauth2-proxy/backup).
@@ -12,15 +12,6 @@ resources:
  # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
  # freeze the other eleven platform components. See clusters/base/cilium.yaml.
  - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
  # Intel GPU device plugin — its OWN Kustomization (dependsOn cilium, wait:true), NOT a line in
  # platform/kustomization.yaml. Same reasoning as the Cilium split-out, applied to a leaf: inside
  # that wait:true tier a wedged 3-node DaemonSet would be indistinguishable from eleven unrelated
  # components AND would stall platform-config + every apps wave on "dependency not ready" — so a
  # component added to fix a silent no-op (Plex software transcoding) could cause a real outage.
  # wait:true here is a HEALTH SIGNAL, not a gate: nothing in the graph dependsOn it.
  # It advertises `gpu.intel.com/i915`; the renderer change that REQUESTS that resource must land
  # strictly later. See clusters/base/intel-gpu-plugin.yaml.
  - intel-gpu-plugin.yaml # Flux Kustomization -> ./platform/intel-gpu-plugin (dependsOn cilium)
  - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
  - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
  # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
@@ -1,174 +0,0 @@
 ---
 date: 2026-09-30
 branch: feat/plex-quicksync
 title: Plex opts into `hardware.gpu: intel-quicksync` — the device mount is GitOps, the Plex-side toggle is a documented manual step, and hardware transcoding has NOT been observed working
 ---
 ## What changed
 Three things, and the third is the point of the branch.
 1. **`catalog/plex.yaml`** — added the four lines the whole feature reduces to:
   ```yaml
   hardware:
     gpu: intel-quicksync
   ```
   placed between `env` and `exposure` to match the schema's own property order, with ~48 lines of
   comment in the file's established style. The comment records: the field is **intent, not
   mechanism** (a later move to the Intel device plugin is a `render.py` change and this block does
   not move); that `/dev/dri/renderD128` is present on **all three** prod nodes with `i915` bound;
   **why a `supplementalGroups` entry would be inert** (the image's cont-init hook does the group
   work inside the container and `s6-setuidgid` replaces the process group list from `/etc/group`);
   why `DirectoryOrCreate` is the load-bearing safe failure mode (`apps-wave-0` is `wait: true` with
   `apps-wave-1` depending on it, so a failed mount would stall the whole app tier for a feature
   that is not load-bearing); and that the Plex-side toggle is a **separate manual operator step**,
   pointing at the new runbook.
 2. **`apps/wave-0/plex.yaml`** — re-rendered. Six added lines, nothing else, in the whole `apps/`
   tree:
   ```
           - name: dri
             mountPath: /dev/dri
         - name: dri
           hostPath:
             path: /dev/dri
             type: DirectoryOrCreate
   ```
 3. **`runbooks/plex-quicksync.md`** (new), registered in `runbooks/kustomization.yaml`'s
   `configMapGenerator` as `- plex-quicksync=plex-quicksync.md` and indexed in `runbooks/README.md`
   under "Operator procedures". **The file alone would have been a 404** — the pod serves a
   ConfigMap, not the git tree, and `conformance/tests/l0-static/alert-runbooks-resolvable` exists
   precisely because "the file is in git" and "the URL resolves" are different claims. The page
   covers: the GitOps/manual boundary as a table; the toggle (Settings → Transcoder → "Use hardware
   acceleration when available", preference key `HardwareAcceleratedCodecs`); the real neighbouring
   keys and the widely-cited one that **does not exist**; that the only proof is `Transcode (hw)` on
   the Plex dashboard **and why no alert is possible**; the device-cgroup failure to expect first
   with a concrete open() diagnostic and the Intel-device-plugin fallback; and the floating-`:latest`
   caveat with a digest check.
 **Stacked on PR #27 (`feat/sd-hardware-capability`), which is open and unmerged.** This branch was
 cut from it rather than from `main`, so its one commit (`d9baa2a`, the schema + renderer + contract)
 is in this PR's range. That is deliberate: merging #27 is not an agent's action (CLAUDE.md §1), and
 duplicating its changes here would have been worse than showing them. **This must merge after #27.**
 Not done, deliberately: no `--emit-waves` (it writes into `clusters/base/`, a sensitive path that
 would force a human reviewer, and it is a no-op unless the *set* of files under `apps/` changes — it
 does not here). No `podSecurityContext`. No `nodeSelector`. No alert rule. No `kubectl apply`.
 ### Two follow-ups left unclaimed, with the reasoning
 - **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
  `TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
  overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
  transcode scratch performance changes silently on reschedule, with nothing reporting it. Both
  candidate fixes need a **schema** change and so neither belongs in this PR: pin Plex to prod-2
  (needs a `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory`
  emptyDir and transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not
  have). **The RAM option is the better one** — it removes the node asymmetry outright instead of
  working around it — which is exactly why it deserves its own PR and its own review rather than
  being smuggled in behind a four-line catalog change. Documented in the runbook's "Known gaps".
 - **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
  quality only; decode and this feature are unaffected. Changing it needs a kernel parameter plus a
  **serialized rolling reboot** — an operator action under CLAUDE.md §5, on a cluster with a
  documented history of that going badly (a prod node hung on a warm reboot on 2026-09-03). Kept
  well clear of this PR.
 ### Rebase note (2026-10-01, Ruth Okonkwo) — resolution only, no redesign
 Rebased onto `main` @ `08deb20` (was cut from `0fbf544`), restacked on the rebased
 `feat/sd-hardware-capability` (PR #27) so the stack stays coherent — this branch still carries
 exactly PR #27's commit plus its own, with no duplicate of anything already in `main`. The branch
 ref was force-updated with `--force-with-lease`, so PR #32 is the same PR with a new head.
 **The `catalog/render.py` conflict disappeared** once PR #27 was rebased; it was never this
 branch's own conflict. **One conflict remained, in `runbooks/README.md`, purely textual.** PR #30
 (`fix/jackett-oauth2-proxy-session`, merged as `08deb20`) appended an `oauth2-proxy-session` index
 entry under *Operator procedures* at the same point this branch appended its `plex-quicksync` entry.
 **Both entries were kept, in full.** Ordering follows this file's existing convention, which is
 grouped/narrative rather than alphabetical (the *Nodes* section runs down/io/memory/disk/cpu):
 `plex-quicksync` sits next to `plex-claim` as the other Plex page, leaving `oauth2-proxy-session`
 adjacent to `crafty-admin-credential`, which cross-references the same sidecar gate. The change to
 `runbooks/README.md` is an 8-line pure insertion, so PR #30's entry is untouched.
 `runbooks/kustomization.yaml` auto-merged and **both** `configMapGenerator` entries are present.
 Re-verified after the resolution, on the rebased stack: `--selftest` PASS (rc 0); a full re-render
 reproduces every committed golden byte-identically; and the **only** `apps/` diff against the new
 `main` is `apps/wave-0/plex.yaml`, confined to the expected six lines (the `dri` mount and its
 `DirectoryOrCreate` hostPath volume). No other service's golden moved — PR #28 added ~176 lines of
 oauth2-proxy sidecar rendering, and this confirms by measurement rather than assumption that Plex
 (`auth: native`, no sidecar) is unaffected by it.
 ## What I verified
 - **The render is exactly the intended change, and nothing else moved.** After
  `python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` (exit 0,
  silent): `git diff --stat apps/` = `apps/wave-0/plex.yaml | 6 ++++++`, and
  `git status --porcelain apps/` = a single ` M apps/wave-0/plex.yaml`. Read line by line, the diff
  is the `dri` volumeMount and the `dri` hostPath volume, and nothing more. **Specifically checked:**
  the same invocation also rewrites `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, which
  are git-tracked but excluded from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in
  `ci/render-golden.sh` — a spurious diff there would have landed **ungated**. Neither file shows a
  diff; whole-repo `git status --porcelain` lists only `apps/wave-0/plex.yaml` and `catalog/plex.yaml`
  plus my new files.
 - **The rendered output matches what the contract promises** — `schemas/renderer-contract.md`
  `### hardware` says exactly two things (a `/dev/dri` hostPath with `type: DirectoryOrCreate` and a
  read-write mount on the app container) and exactly two things appeared. No `securityContext`, no
  `nodeSelector`, no resources in the diff.
 - **The runbook is projected into the server, not merely committed.** `runbooks/kustomization.yaml`
  carries `- plex-quicksync=plex-quicksync.md`, whose shape matches the generator-entry regex the
  L0 check parses; the extensionless key is what makes `/runbooks/plex-quicksync` a plain static
  lookup. Also indexed in `runbooks/README.md`.
 - **The runbook clears the L0 quality floor** that `alert-runbooks-resolvable` applies to
  alert-linked pages, even though nothing links to this one (it is advisory for an unreferenced
  page): all four required sections (`## What this means here`, `## First commands`,
  `## Likely causes, most likely first`, `## When to escalate or stop`), well over the 1500-byte
  floor, `dev.sh` used throughout, and `--context k3s-prod` on every command.
 - **The gate.** See the run recorded in the PR description for the per-stage table and the real rc.
 - **No secret was printed.** Every `Preferences.xml` read in the runbook greps one named key, and the
  page states explicitly why: the file holds `PlexOnlineToken`. Nothing was `sops -d`'d.
 ## What I did NOT verify
 **Hardware transcoding has NOT been observed working.** Nothing in this branch is evidence that it
 does. A green gate here proves the manifest renders and the golden matches — it proves nothing about
 runtime behaviour, and `vars.CLUSTER_AVAILABLE` is unset so L2/L3 conformance did not run at all.
 - **The device-cgroup question is OPEN, and it is the largest remaining technical risk.** Whether
  containerd on this k3s build permits a **non-privileged** pod to `open()` a bind-mounted character
  device is **UNVERIFIED**. It can only be settled by deploying and trying. The container inits as
  uid 0, which settles *file* permissions, but uid 0 does **not** bypass the cgroup device gate — and
  the two failures look identical from the outside. If it is refused, the fallback is the Intel
  Device Plugin (`gpu.intel.com/i915`), which is a `catalog/render.py` change plus a re-render with
  **no ServiceDefinition edit**; `privileged: true` is not available (Kyverno
  `restrict-host-namespaces` is Enforce). The runbook has the diagnostic sequence.
 - **The Plex-side toggle was still unset at the time of writing.** `HardwareAcceleratedCodecs` was
  **absent** from `Preferences.xml` — the toggle has never been enabled on this server. Until an
  operator ticks it in the UI, this branch changes the pod spec and **nothing about transcoding**.
  That is the whole reason the runbook exists.
 - **No metric, so no alert, so no automated detection — permanently.** `catalog/plex.yaml` declares
  no `metricsEndpoint`, so Plex renders no ServiceMonitor; there is no transcode-mode signal for a
  rule to evaluate. A silent degradation to software transcoding (a node losing its iGPU, the
  `DirectoryOrCreate` path) will **not** be reported by anything. I deliberately did not add an
  alert: with no metric behind it, it could never fire, and this repo has been burned at scale by
  exactly that. Closing the gap honestly means an exporter first.
 - **I did not re-derive this session's node or in-container measurements.** The three-node
  `renderD128`/`i915`/GID-992 facts, the cont-init hook behaviour, the `s6-setuidgid` group
  replacement, the `PLEX_PREFERENCE*` absence, the six `setPref` keys and the
  `HardwareAcceleratedCodecs` string were established earlier in this session and are cited, not
  re-measured, by me. Per CLAUDE.md §2b, treat this file as a record of belief and re-measure against
  the cluster before relying on any of it.
 - **The image is a floating tag.** Everything known about the hook came from digest
  `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), while the Deployment runs
  `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. A silent pull can invalidate the
  "no `supplementalGroups` needed" conclusion **with no commit in this repo**. Not guarded against —
  the runbook tells the reader to re-check the hook after an image bump, which is documentation, not
  enforcement.
 - **I did not verify the live Gitea gate state** (branch protection, required contexts, runner
  online) for this PR. Per §2b that must be queried, not read out of `CLAUDE.md`'s table; I did not
  query it.
 - **`/runbooks/plex-quicksync` was not fetched.** The L0 check asserts every static precondition for
  an HTTP 200; it does not make the request, and I did not either.
@@ -1,204 +0,0 @@
 ---
 date: 2026-09-30
 branch: feat/sd-hardware-capability
 title: ServiceDefinition gains a `hardware` capability field — `gpu: intel-quicksync` mounts /dev/dri and nothing else; the enum is what keeps the mechanism swappable
 ---
 ## What changed
 A new optional top-level ServiceDefinition property, `hardware`, with one optional member,
 `gpu`, whose enum today has exactly one value: `intel-quicksync`. **No catalog entry uses it.**
 This PR delivers the capability; consuming it (Plex) is separate work on top.
 - `schemas/servicedefinition.schema.json` — adds `hardware` (object, `additionalProperties: false`,
  `minProperties: 1`) with `gpu` (enum `["intel-quicksync"]`). Schema version bumped **1.7.0 →
  1.8.0**, changelog entry prepended to the schema `description` per that file's own convention.
 - `catalog/render.py` — two module-level constants (`DRI_VOLUME_NAME`, `DRI_DEVICE_DIR`) and the
  emission in `r_deployment`.
 - `schemas/renderer-contract.md` — new `### hardware` field-mapping section, plus cross-field
  invariant **8** in that file's numbered list.
 - `schemas/README.md` — its `**Schema version:**` line said `1.0.0`; it had been stale since 1.1.0.
  Synced to 1.8.0 and annotated that the schema `description` is the changelog of record. Added one
  bullet to "Cross-field rules" for the new enum. That was the whole of the staleness; nothing
  ballooned.
 **What the renderer emits** for `hardware.gpu: intel-quicksync`, on the app container only (never a
 sidecar), and it is exactly two things:
 1. a hostPath volume `dri` → `/dev/dri`, `type: DirectoryOrCreate`;
 2. a read-write `volumeMount` of `dri` at `/dev/dri` on the app container (deliberately not
   `readOnly` — VA-API opens the render node read-write).
 **Mounting the device is sufficient.** It emits **no** `privileged: true` (Kyverno
 `restrict-host-namespaces` is Enforce on app namespaces and it is not needed), **no** `nodeSelector`
 (placement is a separate concern with its own justification), **no** resource requests (there is no
 app-container resource render path), and — the one that is worth spelling out — **no
 `podSecurityContext.supplementalGroups`**.
 **Why no `supplementalGroups`, when `/dev/dri/renderD128` is `root:render 0660`.** An earlier
 revision of this branch merged the host `render` GID into `supplementalGroups`. **It was measured to
 be a no-op and has been removed.** In the running container on prod-2, 2026-09-30:
    PID 1   s6-svscan            Uid: 0     Gid: 0     Groups: 0
    pid 246 Plex Media Server    Uid: 1000  Gid: 1000  Groups: 100 1000
 `/etc/services.d/plex/run` ends in `exec s6-setuidgid plex …`, and the child's group list is exactly
 the `/etc/group` account-database list — root's gid `0`, which PID 1 holds, has **vanished**. That is
 a `setgroups()` that *replaces* rather than appends, so anything Kubernetes puts in
 `supplementalGroups` reaches `s6-svscan` and is discarded before the app ever starts. What actually
 works is the image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner`, which `find`s
 every char device under `/dev/dri`, `stat -c '%g'`s its GID, `groupadd`s a matching group inside the
 container when none exists, and `usermod -a -G`s the app user into it — every start, nothing
 hardcoded; `cont-init.d` completes before `services.d`, so the database is correct at exec time.
 Shipping an inert `supplementalGroups` would have been worse than shipping nothing: it would read as
 the mechanism to whoever debugs this next. The reasoning lives in a comment **at the emission site**
 and in `renderer-contract.md`, and `render.py --selftest` now asserts the *absence* of a
 `securityContext`, so it cannot be "helpfully" added back without a test failing. A pleasant
 corollary: because the hook re-`stat`s the device every start, host GID drift is **self-healing**,
 and no GID is pinned anywhere in this repo.
 **The design decision worth defending: the field is an enum of intent, not a device path.** The SD
 says *what class of host device the workload needs*; the hostPath and the mount exist only in
 `catalog/render.py`. Migrating to the Intel GPU device plugin (`gpu.intel.com/i915` resource
 requests, no hostPath at all) is then a renderer change with zero catalog edits. Both the schema
 `description` and the contract section say this explicitly, because the obvious "simplification" —
 letting an SD name `/dev/dri` — would push a platform implementation detail into the operator-facing
 contract and make that migration a catalog-wide edit.
 **`DirectoryOrCreate` is load-bearing.** A node that comes back without the render device (rebuild,
 BIOS change, driver regression, hardware swap) must degrade the workload to software transcoding,
 not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so a failed
 hostPath mount stalls the entire app tier for the sake of a feature that is not load-bearing.
 `/dev` is tmpfs, so the empty directory the kubelet creates on such a node leaves nothing durable
 behind. Do not "tighten" it to `Directory`. The reasoning is recorded at the emission site.
 ### Rebase note (2026-10-01, Ruth Okonkwo) — resolution only, no redesign
 Rebased onto `main` @ `08deb20` (was cut from `0fbf544`). Re-landed as a single cherry-pick onto
 current `main`; the branch ref was force-updated with `--force-with-lease`, so PR #27 is the same
 PR with a new head.
 **One conflict, in `catalog/render.py`, purely textual.** PR #28
 (`fix/oauth2-proxy-cookie-collision-fleet`, merged as `6caa5f2`) inserted its own constants blocks
 (`OAUTH2_ACCESS_TOKEN_LIFETIME_S` / `OAUTH2_COOKIE_REFRESH`, and `OAUTH2_PROXY_METRICS_PORT`) at the
 same insertion point immediately after `KEYCLOAK_ISSUER` that this branch used for
 `DRI_VOLUME_NAME` / `DRI_DEVICE_DIR`. The two blocks are unrelated and do not interact, so **both
 sides were kept in full**, oauth2-proxy's first and the hardware block after it. Nothing was
 dropped and nothing was rewritten: the rebased commit adds the **same 126 `render.py` lines** as the
 pre-rebase commit (verified by comparing the added-line sets) and deletes none. In particular the
 `supplementalGroups`-is-deliberately-not-emitted and `DirectoryOrCreate`-is-load-bearing comments
 survive verbatim.
 Re-verified after the resolution: `--selftest` PASS (rc 0), and a full re-render still produces a
 **zero** diff in `apps/` — this PR remains inert, as it must be while no ServiceDefinition opts in.
 ## What I verified
 Measured, in this worktree, on this branch:
 - **`python3 catalog/render.py --selftest` → `render.py selftest: PASS`, rc 0.** New cases in the
  existing `check(...)` style: the positive (volume + mount, and *no* `securityContext`, *no*
  privileged, *no* nodeSelector), the negative that matters (an SD without `hardware` gains no
  volume, no volumeMount and no `securityContext` — mirroring the existing `podSecurityContext`
  negative), and a pass-through case (an SD declaring both `hardware` and its own
  `podSecurityContext: {fsGroup: 1001, supplementalGroups: [44]}` keeps it **exactly** as written).
 - **The self-tests actually bite.** I mutated the guard (`if hw_gpu == "intel-quicksync":` → `if
  True:`) in a scratch copy and re-ran: **rc 1**, failing on "an SD without `hardware` must gain no
  dri volume, no mount and no securityContext". A test that cannot fail is not a test; this one can.
 - **Re-render is a zero diff.**
  `python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` then
  `git diff --stat apps/` → **empty**, and `git status --porcelain apps/` → **empty** (so the two
  not-onboarded by-products, `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, were
  rewritten byte-identically and needed no revert). The change is inert until a service opts in.
 - **Schema validity and catalog conformance.** `Draft202012Validator.check_schema` passes; all
  **12** `catalog/*.yaml` validate with **0** errors. On the new field specifically: `gpu:
  intel-quicksync` accepted; `gpu: /dev/dri` rejected (enum); `hardware: {gpu: …, path: …}` rejected
  (`additionalProperties: false`); `hardware: {}` rejected (`minProperties: 1`).
 - **Hand-checked the emitted YAML** by rendering a throwaway SD carrying `hardware`, a Longhorn
  volume and its own `podSecurityContext: {runAsUser: 1000, supplementalGroups: [44]}`. Result:
  `volumeMounts` = config then `{name: dri, mountPath: /dev/dri}`; `volumes` = the PVC then
  `{name: dri, hostPath: {path: /dev/dri, type: DirectoryOrCreate}}`; `securityContext` =
  `{runAsUser: 1000, supplementalGroups: [44]}` — i.e. **unchanged**, nothing appended. That toy SD
  was never written into `catalog/`.
 - **GATE: `rc 3` — UNDETERMINED, NOT green.** Run in the pinned toolbox from this worktree with
  both CLAUDE.md §3 workarounds (`DEV_KUBECONFIG=/dev/null`, and an `ENGINE` shim bind-mounting the
  main checkout's `.git` at its own absolute path). Diff base `origin/main @ 0fbf544d5e2f`; touched
  requirements `R-DNS-03 R-GITOPS-02 R-IDP-02`. Run **four** times on this branch with an identical
  result each time, most recently against tree `c132893045b1` — the final commit differs from that
  run only by this paragraph. Per stage:
  | stage | result |
  |---|---|
  | spec-immutability | **SKIPPED** — sibling spec repo not at `/work/../home-network-k3s`. Expected from a worktree; tolerated by §3. |
  | decisions-log | pass |
  | lint | pass |
  | l0 | pass — 19 offline cases, `0 failure(s), 0 skip(s), 0 unwired` |
  | render-golden | pass — *"all 10 onboarded service golden(s) reproduce from the canonical render"* |
  | l1 | pass — 3 offline checks, `0 failure(s), 0 skip(s), 0 unwired`; coverage 51/51 requirements scaffolded |
  | approval-policy | pass (`event='none'`, not a pull_request event — N/A locally; it will evaluate for real on the PR) |
  | gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset. Expected; branch protection and runner registration are therefore UNVERIFIED by this run. |
  | conformance (L2/L3) | **SKIPPED** — `CLUSTER_AVAILABLE!=true`. Phase-0 standing skip. |
  The gate's own verdict: *"Stages that did not run: spec-immutability gitea-settings"*. **No stage
  failed, and no stage skipped for a reason caused by this change.** rc 3 is reported as rc 3; I am
  not calling this green.
 - **Blast radius of the schema change:** `hardware` is absent from every catalog entry, so
  `apply_defaults` never materializes it and `s.get("hardware")` is `None` on all 12 — which is what
  the zero diff above demonstrates rather than asserts.
 - **Touched no sensitive path.** Changes are confined to `schemas/`, `catalog/`, `decisions/`; per
  `infra/gitea/approval-policy.json` none of those are sensitive. `conformance/**` and `clusters/**`
  are untouched. I did not run `--emit-waves` (it writes `clusters/base/`, and the set of files
  under `apps/` did not change).
 ## What I did NOT verify
 - **Whether containerd's device cgroup permits a non-privileged pod to `open()` a bind-mounted char
  device. Still unverified, and it is the single largest technical risk in this workstream.**
  containerd installs a per-container **device cgroup** allow-list, and whether
  `/dev/dri/renderD128` arriving as a bind-mounted device node is permitted by it on this
  k3s/containerd version is **unmeasured**. Settling it requires applying a pod, which is an
  operator action, not an agent one. The container's PID 1 running as uid 0 helps on the DAC side
  but does **not** bypass the cgroup gate. If it blocks, the mount will succeed, the image's group
  hook will do its job, and the `open()` will still fail — and the fallback is the Intel GPU device
  plugin, which *sets* the cgroup entry. Note that that fallback is a **renderer-only change**: no
  ServiceDefinition moves, because the authoring surface is an intent enum. That is the design
  justifying itself, and it is the reason to keep the enum even if the hostPath mechanism loses.
 - **That a missing device degrades silently — and nothing will tell you.** This is the accepted cost
  of `DirectoryOrCreate`. The workload will software-transcode and **no signal reports it**: media
  apps in this catalog declare no `metricsEndpoint`, so no ServiceMonitor exists, and there is no
  metric to alert on. I deliberately did **not** propose an alert, because an alert that cannot fire
  is worse than none — see the 2026-09-13 correction in `renderer-contract.md` §`observability`,
  where twelve monitors discovered zero targets. The only proof of hardware transcode is an operator
  seeing it in the application's own dashboard.
 - **That "mounting the device is sufficient" generalizes beyond this one image.** It does not, and
  I have not verified that it does. The claim was measured for an image that resolves the device
  group itself in `cont-init.d`. A future consumer of `hardware.gpu` that runs non-root with no
  equivalent hook would still need the group solved — and the fix for that case is a **renderer**
  change (per-image handling, or the device plugin), never a new SD field. I scoped the claim in
  `renderer-contract.md` rather than letting it read as general; the first such consumer must
  re-measure.
 - **The feature now depends on a hook inside an image that floats.** The `cont-init.d` behaviour
  above was read from digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), but the
  deployment runs `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. **A silent pull can
  change the behaviour this feature depends on, with no commit anywhere in this repo to show for
  it** — and the failure would be the silent software-transcode described above. I have not fixed
  that here (it is the Plex wiring's tag/digest decision, not the capability's), but it should not
  be forgotten.
 - **GID drift is NOT on this list any more, deliberately.** An earlier revision pinned
  `INTEL_RENDER_GID = 992` and listed "a node rebuild could move it" as an open risk. The hook
  re-`stat`s the device on every start, so drift is self-healing and no GID is pinned anywhere. The
  risk is retired, not overlooked.
 - **Anything on the live cluster.** I ran no `kubectl` and I opened no shell in any container. The
  node facts (device present, `root:render` 0660, `i915` bound, no device plugin, no
  `gpu.intel.com/*` advertised), the in-container measurements quoted above, and the Kyverno
  posture (`restrict-host-namespaces` Enforce; no hostPath or `supplementalGroups`
  restriction; no PSA labels on app namespaces) are **prior recon handed to me, not re-measured by
  me**. Per CLAUDE.md §2b, treat them as second-hand until re-queried.
 - **That CI exercises the self-tests. It does not.** `git grep -- --selftest ci/` returns nothing —
  no gate stage invokes the renderer's self-test. They are the executable spec and house style, and
  I ran them by hand, but a future regression in this code path **will not be caught by the gate**.
  Wiring `--selftest` into a gate stage is a real gap and is not part of this PR (`ci/` is a
  sensitive path and would drag in a human reviewer for a change unrelated to this one).
 - **Runtime behaviour of any kind.** The gate proves the L0/L1 offline subset only;
  `vars.CLUSTER_AVAILABLE` is unset, so L2/L3 conformance does not run in CI.
@@ -1,198 +0,0 @@
 ---
 date: 2026-10-01
 branch: feat/intel-gpu-device-plugin
 title: Intel GPU device plugin, in its own Kustomization — the resource exists; nothing has run yet
 ---
 ## What changed
 Deployed the **Intel Device Plugin for Kubernetes (GPU)**, standalone DaemonSet, pinned at
 **v0.37.0**, so the prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the
 Plex hardware-transcoding blocker: the `/dev/dri` hostPath merged on `main` @ `cdf3e14` makes the
 device node *visible* and not *openable* (`EPERM` from the device cgroup at both uid 0 and uid
 1000; the container's runc `config.json` carries one deny-all rule and `linux.devices: None`). A
 hostPath is a filesystem operation and adds no cgroup allow rule. A device plugin's `Allocate`
 response is what makes kubelet populate `linux.devices` *and* the matching cgroup allow rule, and
 it is the only non-privileged way to get there — `privileged: true` is blocked by Kyverno
 `restrict-host-namespaces` under Enforce in app namespaces, and `plex` is labelled
 `app-namespace=true`.
 New files:
 - `platform/intel-gpu-plugin/intel-gpu-plugin.yaml` — the DaemonSet, **vendored** as
  `kubectl kustomize` output of upstream `deployments/gpu_plugin/base` at `v0.37.0`, under a
  provenance header carrying the exact re-vendor command and both sha256s (generated body
  `80ebd070…`, raw upstream source `c8773918…`). Generated rather than byte-copied because
  upstream mixes YAML sequence-indentation styles within the one document and `ci/yamllint.yaml`
  sets `indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate.
  That is not a guess: the first gate run of this branch returned **rc 1 BLOCKED** on exactly that
  error, and regenerating through kustomize is the fix (and is also the re-vendor idiom
  `platform/external-snapshotter` documents). Vendored at all, rather than referenced as a remote
  kustomize base, for the other reason `external-snapshotter` records: a remote base is
  `git fetch`ed on every reconcile and those fetches have flaked a whole tier.
 - `platform/intel-gpu-plugin/namespace.yaml` — a PLATFORM namespace, explicitly
  `homelab.keithjackson.dev/app-namespace: "false"` (following `vpn-egress`/`kyverno`).
 - `platform/intel-gpu-plugin/kustomization.yaml` — namespace scoping plus the image pin.
 - `platform/intel-gpu-plugin/README.md` — the measurement, the placement argument, the
  non-choices, and the post-merge verification sequence.
 - `clusters/base/intel-gpu-plugin.yaml` — **its own Flux Kustomization**: `dependsOn: [cilium]`,
  `wait: true`, `interval: 10m`, `timeout: 10m`, `prune: true`.
 - One wired line (plus its rationale comment) in `clusters/base/kustomization.yaml`.
 **Placement: its own Kustomization, not a line in the shared `platform` tier.** The shared tier was
 cheaper and needed no human reviewer on `approval-policy.json`'s terms, but `platform` is
 `wait: true` over eleven-plus components and the repo already argued this case when Cilium was
 split out: its Ready condition cannot distinguish "the plugin is wedged" from "longhorn is slow",
 the only emergency valve is freezing the entire tier, and the tier's 10m/5m interval/timeout are
 called out there as *wrong for a 3-node DaemonSet rollout* — which is exactly what this is. The
 consequence is worse than average here, because `platform` gates `platform-config` gates
 `apps-wave-0` (`wait: true`) gates `apps-wave-1`: a plugin that never went healthy inside that tier
 would stall every app, so a component added to fix a **silent no-op** would have gained the ability
 to cause a **real outage**. The approval cost of touching `clusters/**` is nil, because
 `required_approvals: 1` with `block_admin_merge_override: true` applies to every PR here regardless
 of path and authors cannot self-approve.
 `dependsOn: [cilium]` is the minimum *and* the earliest: the plugin never uses the network and
 never talks to the API server (it talks to the local kubelet over a unix socket), so a working CNI
 is its only real prerequisite, and `cilium` already depends on `flux-system`, which applies the
 `cluster-vars` ConfigMap this Kustomization substitutes from. Depending on `cilium` rather than
 `platform` also means it reconciles *in parallel with* the platform tier rather than behind all
 eleven of its components, which shrinks the cold-start window in which a node has not yet
 advertised the resource.
 `wait: true` is a deliberate departure from the `headlamp`/`crafty` leaf template. Those use
 `wait: false` because a leaf UI can be legitimately not-Ready for a long time during a cold start,
 making its health a useless signal. This DaemonSet reads sysfs and opens a socket, so not-Ready
 genuinely means broken — and since **nothing in the graph `dependsOn` it** (verified against the
 rendered graph), `wait: true` gates nothing while turning
 `flux get kustomization intel-gpu-plugin` into a one-line answer to the question that must be
 answered before the consumer PR may merge.
 Deliberate non-choices, each argued in the README: standalone DaemonSet rather than the operator
 (no CRD, no controller-manager, no cert-manager webhook, no second namespace to keep healthy);
 `--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler
 believes — there is no GPU-side partitioning behind it, so it converts a loud "Unschedulable" into
 a quiet "transcodes are slow"); `--health-management` left OFF, because it is the one flag that
 *would* open a device and would reintroduce the exact `EPERM` this fixes; no NFD overlay (all three
 nodes are identical); no `supplementalGroups` anywhere, because the `pms-docker` cont-init hook
 already adds the `plex` user to each device's group and that works identically on a
 plugin-injected device.
 **Scope held deliberately:** `catalog/render.py`, `catalog/plex.yaml` and `apps/wave-0/plex.yaml`
 are untouched. Switching Plex from the hostPath to a `gpu.intel.com/i915: 1` request is a strictly
 later PR, and **it must not land until this plugin is merged AND confirmed advertising the resource
 on all three nodes** — a request for a resource no node advertises makes the Plex pod
 Unschedulable, and `apps-wave-0` is `wait: true` with `apps-wave-1` depending on it, so that would
 stall the whole app tier and turn today's silent no-op into a real outage.
 ## What I verified
 Mechanism, read from the v0.37.0 source rather than assumed:
 - **The plugin only lists devices; it never opens one.** The scan path uses `os.ReadDir`,
  `os.ReadFile`, `os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. `grep` for
  `os.Open|OpenFile|ioctl|syscall|unix\.` across `cmd/gpu_plugin/gpu_plugin.go` (908 lines) returns
  **nothing**. Listing and `stat()` are what demonstrably work today, so the plugin needs no cgroup
  grant and no extra privilege — it does not need more than the upstream manifest gives it.
 - **The fix mechanism is real:** `createDeviceSpecsFromDrmFiles` emits
  `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, which is the
  thing kubelet turns into `linux.devices` plus a cgroup allow rule.
 - **The resource name is `gpu.intel.com/i915` by derivation, not by hope:** `scan()` calls
  `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the kernel driver name, and these
  nodes have `i915` bound.
 - `--shared-dev-num` default is `1` (`flag.IntVar(..., 1, ...)`); `--health-management` default is
  `false` and `setupLevelZeroService` returns early unless it is set.
 - Missing `/dev/dri/by-path` is non-fatal (`newDevicePlugin` warns, sets `bypathFound = false`).
 Image pin, verified against the registry:
 - `docker.io/intel/intel-gpu-plugin:0.37.0` exists; manifest digest
  `sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1` (single amd64
  manifest, not an index), fetched from `registry-1.docker.io` on 2026-10-01. `v0.37.0` is the
  latest upstream release (2026-09-16) and is not a prerelease.
 - The Docker Hub tag list carries `0.35.0 / 0.36.0 / 0.37.0` — **image tags have no `v` prefix**
  while GitHub release tags do.
 Policy scope, read from the policies rather than assumed:
 - All four app-scoped Kyverno policies (`restrict-host-namespaces`, `require-egress-class-label`,
  `require-auth-mode-label`, `require-default-deny-baseline`) are gated on
  `namespaceSelector`/`selector` `matchLabels: homelab.keithjackson.dev/app-namespace: "true"`.
  The new namespace is labelled `"false"`, so none of them match it.
 - Measured on prod: `plex` **is** `app-namespace=true`, confirming `privileged: true` is genuinely
  unavailable there; `kyverno` and `vpn-egress` are the two namespaces carrying the label as
  `"false"`; the other platform namespaces leave it unset. No `intel*` namespace exists, so no
  collision.
 Cluster before-state (read-only, `kubectl` in the pinned toolbox with the prod kubeconfig
 bind-mounted read-only — never copied into the worktree):
 - Three nodes `k3s-prod-{1,2,3}`, each `v1.36.4+k3s1`, Debian 13 trixie, `containerd://2.3.4-k3s1.36`.
 - Every extended resource currently advertised is `devices.kubevirt.io/{kvm,sev,tun,vhost-net}` on
  all three nodes. **No node advertises `gpu.intel.com/*`** — queried explicitly, returned
  "NONE". That is the before-state this change is meant to move.
 - Those `devices.kubevirt.io/*` resources are themselves advertised by a device plugin, which is
  measured evidence that the device-plugin API and the default kubelet socket path
  (`/var/lib/kubelet/device-plugins`) work on these nodes.
 Offline rendering (the main proof available to a PR):
 - `kubectl kustomize platform/intel-gpu-plugin` renders two objects: `Namespace/intel-gpu-plugin`
  carrying `app-namespace=false`, and `DaemonSet/intel-gpu-plugin` in namespace
  `intel-gpu-plugin`, with the image resolved to `docker.io/intel/intel-gpu-plugin:0.37.0`.
 - `kubectl kustomize clusters/base` renders, and the new Flux Kustomization appears as
  `intel-gpu-plugin  ./platform/intel-gpu-plugin  wait=true  dependsOn=cilium`. Both overlays
  (`clusters/prod`, `clusters/dev`) also render and carry it.
 - Across the whole rendered graph, **no Kustomization lists `intel-gpu-plugin` in `dependsOn`** —
  so `wait: true` gates nothing.
 - The vendored DaemonSet is a **formatting-only** transform of upstream, proven rather than
  asserted: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file `diff`
  clean, so passing it through kustomize changed no content. The committed body's sha256 matches
  the one recorded in its own provenance header (`80ebd070…`), so the file has not drifted from
  what was generated.
 Gate (pinned toolbox, from this worktree, with the `ENGINE` shim bind-mounting the main checkout's
 gitdir and `DEV_KUBECONFIG=/dev/null`): see the PR description for the per-stage table. **`rc 3`
 (UNDETERMINED), with exactly the three expected skips** — `spec-immutability` (sibling spec repo is
 not beside a worktree), `gitea-settings` (no CI token), `conformance` (no cluster in CI). Every
 other stage passed. **rc 3 is not green** and is not reported as such.
 ## What I did NOT verify
 **Nothing about this component's runtime behaviour. It has never run.**
 - **The plugin has never executed.** No pod has been scheduled, no image has been pulled, no scan
  has happened. It cannot run until the operator merges this and Flux reconciles — `kubectl apply`
  and `flux reconcile` are operator actions (CLAUDE.md §5), so this is expected, not an oversight.
 - **The resource has never been advertised.** `gpu.intel.com/i915` does not exist on any node and
  I have not seen it exist. The claim that it *will* be `i915` rather than `xe` is derived from
  source plus the established fact that `i915` is bound — it is a derivation, not an observation.
 - **The DaemonSet has never been admitted.** That the Kyverno policies do not match this namespace
  is read off the policy manifests; no admission request has actually been evaluated. Nor has the
  `kubectl kustomize` output ever been server-side dry-run against the API — rendering is not
  admission.
 - **The `open()` test has NOT been re-run.** The `EPERM` is exactly as it was. Nothing in this PR
  touches Plex, so Plex is still software-transcoding and will continue to until the later renderer
  PR lands. The decisive evidence is an `open()` as **uid 1000 through `s6-setuidgid plex`**;
  passing as uid 0 is weaker and must not be accepted as proof.
 - **A green gate proves the manifests render, nothing more.** And this gate was not green: it was
  `rc 3`. Note also that the Gitea `gate` job runs `./ci/gate.sh || [ "$?" -eq 3 ]`
  (`.gitea/workflows/gate.yml:138`), so a green tick on this PR cannot distinguish rc 0 from rc 3,
  and `vars.CLUSTER_AVAILABLE` is unset so no L2/L3 conformance runs in CI.
 - **`seLinuxOptions: type: container_device_plugin_t`** is carried verbatim from upstream. These
  nodes are Debian 13 and I did not measure whether SELinux is enabled or whether that type exists
  there. Expected to be inert (runc skips labelling when SELinux is disabled), but unverified — if
  the pods fail to start with an SELinux error, that line is the first suspect.
 - **Renovate has not been run** against this pin. The inline-marker idiom is copied from
  `platform/kubevirt` / `platform/external-snapshotter`, but whether Renovate normalises the
  `v`-prefix mismatch (release `v0.37.0` vs image `0.37.0`) for this dep is unknown. If it proposes
  `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in `renovate.json`.
 - **Resource limits unexamined under load.** The upstream 115Mi/100m limits are taken as-is; I have
  no measurement of this plugin's footprint on these nodes. Worth a glance given
  `keycloak-operator` is OOMing at 91% of its limit elsewhere in this cluster.
 - **`clusters/dev` renders it, and there is no dev cluster** (the operator confirmed the
  `k3s-dev-*` nodes are gone). That is inert rather than wrong — the unified graph (R-ENV-02) has
  both overlays render the same base — but it is not a thing I could test.
 - **Branch protection / required-checks state not re-measured** in this session. CLAUDE.md §2b
  says re-measure before relying on it; I did not, so treat the approval/gating claims in §2 as
  unconfirmed-by-me.
@@ -0,0 +1,205 @@
 ---
 date: 2026-10-01
 branch: fix/render-selftest-runs-in-gate
 title: catalog/render.py's 106 assertions were dev-time only; the gate now runs them and blocks on them
 ---
 ## What changed
 Single subject, one file: **`ci/render-golden.sh` now invokes `catalog/render.py --selftest`**
 and exits 1 when it fails. Nothing else. No renderer change, no new conformance test, no new
 gate stage.
 This closes the top follow-up of `decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md`:
 > Nothing in `ci/` or `.gitea/` runs `catalog/render.py --selftest` — I grepped for it. So the
 > twenty-odd assertions in that selftest, including the six I just added, are dev-time only: a
 > future edit that drops either flag *and* re-renders the goldens would pass the entire merge
 > gate.
 Re-measured here and confirmed: before this commit, `grep -rl -- --selftest` over `ci/` and
 `.gitea/` returned **nothing** for the renderer. The selftest executes **106** assertions
 (`check()` invocations counted at runtime via `sys.settrace`, without editing `render.py`; 109
 static call sites, three of them on failure-only branches). All 106 were unexercised by the
 merge gate.
 ### The lead was WRONG, and here is why
 The brief named `conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` as the
 one-line fix. That line is real and it does drive selftests:
    for tool in ("record_run.py", "status.py"):
        r = run([sys.executable, str(LIB / tool), "--selftest"])
 But it is **the wrong hook**. That loop is section (d) of a check that serves **R-SEC-03**, and
 its subject is the *evidence chain* — can a self-skipped run, or a stale ledger row, become an
 audit proof? `record_run.py` and `status.py` are **in** that chain; `catalog/render.py` is not.
 Hanging the renderer there would have made the check's name, its `intent` annotation and its
 requirement mapping (`R-SEC-03`) all untrue, and a future decision about the renderer could then
 only be expressed by weakening a *security-auditability* check. That is the precise coupling the
 cookie-collision entry refused for its own check, and it applies here.
 The second option that entry offered — a new scope-declaring L0 test — is the supported mechanism
 for a **whole-repo invariant** (`ci/l0.sh` Stage 3b's `--include-scope whole-repo` union, ratcheted
 by `tests/l0-static/whole-repo-invariants-wired`). It is the right shape for an invariant
 quantified over the tracked tree. It is the wrong shape for *"a script in this repo must pass its
 own unit test"*: it would need to declare requirements it does not serve, and it would be two new
 files of several hundred lines by this tree's conventions for what is a six-line invocation.
 **The honest home is `ci/render-golden.sh`** — the gate stage whose subject *already is*
 `catalog/render.py`. It already resolves `$PYTHON` and `$RENDER_PY`, already reports rc 3 when
 either is missing, already exits 1 on renderer error, and `ci/gate.sh` already maps it through
 `for s in lint l0 render-golden l1` where a FAIL is BLOCKED and a SKIP is UNDETERMINED. The
 stage's two assertions are independent and both are needed:
 - `render-golden` catches golden-vs-renderer drift, but is satisfied by **any** renderer whose
  output was re-rendered — including one that just lost a contract invariant and had all ten
  goldens regenerated in the same commit. That is exactly the hole named in the quote above.
 - `--selftest` catches that, and is blind to golden drift.
 The file header and the stage name were updated so neither is a lie: the stage is now "the
 renderer is trustworthy", with two numbered assertions, and the step summary emits its own
 `- render-selftest:` line so the gate output still NAMES what ran.
 The selftest runs **before** the empty-catalog `exit 3` on purpose: it renders synthetic
 ServiceDefinitions in memory and is catalog-independent, so an empty catalog must not be able to
 skip it.
 `catalog/render.py` itself was **not touched** (owned by a concurrent branch this session), and
 neither was any file under `conformance/`.
 ## What I verified
 - **The selftest runs and passes: rc 0, `render.py selftest: PASS`, 106 assertions executed.**
  Counted at runtime, not from the source: a `sys.settrace` hook counting `check()` calls, in a
  throwaway script outside the repo, with no edit to `render.py`.
 - **The stage passes with it wired in**, host run of the single stage:
      [gate] Stage: render==golden reproducibility (canonical render vs committed apps/)
      [gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
      [ ok ] render.py --selftest passed
      [ ok ] render==golden: all 10 onboarded service golden(s) reproduce from the canonical render
  rc **0**. The golden comparison still reports all 10 goldens, so the new invocation did not
  displace what the stage already asserted.
 - **IT CAN FAIL — provoked twice, against a mutated COPY of the renderer, never against the
  tree.** `ci/render-golden.sh` already honours `RENDER_PY`, so each mutant lived in
  `/tmp/claude-1000/desmond-selftest/` and the repo's `catalog/render.py` was never modified —
  which matters, because that file belongs to a concurrent branch this session.
  - **Mutation 1 — the regression this selftest was written for.** Deleted the
    `f"--cookie-name=_oauth2_proxy_{s['name']}"` emission from `_oauth2_sidecar_container()`,
    i.e. reverted the fleet-wide cookie-collision fix of PR #28. Stage rc **1**. Note the shape
    of the failure, because it is a real (minor) weakness and is listed as a follow-up: the
    selftest died with an uncaught `StopIteration` out of `next(a for a in container["args"] if
    a.startswith("--cookie-name="))` at `render.py:2641`, rather than printing its collected
    `SELFTEST FAIL:` lines. It is correctly non-zero and the stage correctly blocks, but the
    operator gets a traceback instead of the assertion message.
  - **Mutation 2 — a clean, named assertion failure.** `OAUTH2_COOKIE_REFRESH = "2m30s"` ->
    `"10m"`, widening the refresh past the measured 300s access-token lifetime. Stage rc **1**:
        [fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
        [fail]   Every golden under apps/ is produced by this renderer, so the goldens below cannot
        [fail]   be trusted even if they reproduce byte-for-byte. Reproduce with:
        [fail]     python3 catalog/render.py --selftest
            SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). A
            refresh at or past the access-token lifetime never renews in time, which is the same
            logout it was added to stop
    This is the mutation that proves the wiring's value most directly: mutation 2 would **not**
    be caught by `render-golden`'s diff if the goldens were regenerated in the same commit, and
    it is one of the two flags the cookie-collision entry explicitly recorded as **ungated**.
 - **THE NON-ZERO PROPAGATES ALL THE WAY TO `gate.sh`'s rc — provoked, in the pinned toolbox, not
  reasoned about.** `RENDER_PY=/desmond-selftest/render-mutant2.py ./ci/gate.sh`:
      [ ok ] stage spec-immutability: pass
      [ ok ] stage decisions-log: pass
      [ ok ] stage lint: pass
      [ ok ] stage l0: pass
      [gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
      [fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
          SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). ...
      [fail] stage render-golden: FAIL
      [ ok ] stage l1: pass
      ...
      [fail] render-golden: FAILED -> blocks merge
      [fail] MERGE GATE: BLOCKED
  `gate.sh` rc **1**. Three links in the chain, each measured: selftest rc 1 -> stage rc 1 ->
  gate rc 1 / `MERGE GATE: BLOCKED`. This mattered enough to provoke rather than assume: this
  repo has a documented habit of exit codes getting flattened, and
  `.gitea/workflows/gate.yml:138` runs `./ci/gate.sh || [ "$?" -eq 3 ]`, which tolerates rc 3 —
  so a stage that degraded a failure to a skip would be swallowed by CI. rc 1 is not tolerated
  there, and rc 1 is what this produces.
 - **`spec-immutability` genuinely RAN and passed, it did not skip.** The two-mount worktree shim
  plus `-e SPEC_REPO=/home-network-k3s` (the fix recorded in the 2026-09-30 cookie entry) works:
  `stage spec-immutability: pass`, not SKIPPED. R-TEST-06 is therefore actually covered by these
  runs.
 - **No collision with PR #29.** The brief warned that `feat/oauth2-proxy-auth-failure-visible`
  had added checks under `conformance/tests/l0-static/`. Measured —
  `git diff origin/main...origin/feat/oauth2-proxy-auth-failure-visible --name-only` — it touches
  **no `conformance/` path at all**: `decisions/`, `platform/oauth2-proxy/` (a promtool test, an
  alert rule, its kustomization), `runbooks/`. Disjoint from this branch either way, since this
  branch touches only `ci/render-golden.sh` and this file.
 - **Priya's files were not touched.** `catalog/render.py`, `schemas/servicedefinition.schema.json`,
  `apps/wave-*` and `conformance/tests/l1-rendering/` are all untouched by this branch — confirmed
  by `git diff --name-only`, which lists exactly two paths.
 - **`catalog/render.py --selftest` was the LAST unwired selftest in the tree.** Measured:
  `grep -rl -- --selftest` across the repo finds selftest-capable artifacts in exactly four
  places, and after this commit every one is driven by the gate — `conformance/lib/record_run.py`
  and `conformance/lib/status.py` by `auditability-proofs-producible/check.sh:291`, the Keycloak
  client-secret reconciler (embedded in `platform/keycloak/client-secret-reconciler.yaml`) by
  `keycloak-client-secret-reconciler/check.sh:633`, and `catalog/render.py` by this change.
 ## What I did NOT verify
 - **Local gate: rc 3 (UNDETERMINED), NOT 0. This is not green and must not be reported as
  green.** Measured in the pinned toolbox on `eeb6211`:
      [ ok ] spec-immutability: green
      [ ok ] approval-policy: green
      [ ok ] decisions-log: green
      [ ok ] lint: green
      [ ok ] l0: green
      [ ok ] render-golden: green
      [ ok ] l1: green
      [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
      [fail]   Stages that did not run: gitea-settings
  Nothing FAILED. The single cause is `gitea-settings` **SKIPPED** — `SETTINGS_API_TOKEN` is not
  set in my environment, so branch protection and runner registration were not read; CI has that
  token and the stage is expected to run there. **L2/L3 `conformance`** also skipped
  (`CLUSTER_AVAILABLE` unset) but is a *declared* gap that `gate.sh` treats as not-applicable
  rather than undetermined, so it is not what produced the 3. Touched requirements
  **R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06** are therefore **UNVERIFIED at L2/L3**, all four.
  The stage this branch changes, `render-golden`, is green **and** was independently proven able
  to turn the whole gate red (above).
 - **`approval-policy` passed LOCALLY only because there is no pull_request event**
  (`not a pull_request event (event='none') — approval policy N/A`). That is not evidence that it
  passes. This diff touches `ci/**`, a sensitive path, so in CI the stage is expected to **FAIL**
  until a non-author approving review exists. That is §1 working, fail-closed, not a defect.
 - **The selftest's own quality is unchanged and ungated.** I wired its *invocation*; I did not
  review its 106 assertions, and nothing ratchets them. Deleting the body of `_selftest()` would
  still pass this stage, because an empty selftest returns 0. The repo already has the right
  answer for that — `conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py`
  injects regressions one at a time and requires `--selftest` to catch every one — and applying
  that pattern to `catalog/render.py` is the follow-up, deliberately not done here: it is a
  second subject, it touches `conformance/`, and it would need to live beside the renderer whose
  file another branch owns this session. **Until it exists, "the selftest runs" is proven and
  "the selftest is falsifiable" rests on my two hand-run mutations, which are not in the gate.**
 - **Mutation 1's traceback is not fixed.** `render.py:2641`'s `next(...)` has no default, so the
  absence of `--cookie-name` raises `StopIteration` and the selftest's collected `SELFTEST FAIL:`
  lines are never printed. Correct verdict, poor diagnostics, and it suppresses the other
  assertions that mutation also broke. The one-line fix (`next(..., None)` plus a `check()`) is in
  `catalog/render.py`, which is another branch's file this session, so it is a follow-up and not
  a thing I touched.
 - **Nothing about the live cluster.** No `kubectl` of any kind was run on this branch — not even
  read-only; this change is offline CI wiring and needed none. No `apply`/`edit`/`patch`, no
  `flux reconcile`. Merging this deploys nothing: it changes only what the gate refuses.
 - **CI's own run of this branch.** Quoted in the PR body from the commit's `/statuses`, with
  descriptions read rather than colours. A *skipped* Gitea job reports `success` with
  "Has been skipped" and certifies nothing.
 - **`vars.CLUSTER_AVAILABLE` is still unset in CI**, so even a fully green PR proves the L0/L1
  offline subset and nothing about runtime behaviour.
 - **I did not re-measure branch protection, required approvals, or the runner.** Those are
  CLAUDE.md §2 claims and §2b says re-measure before relying on them; this branch relies on none
  of them beyond the PR flow itself, and I read the gate's own output rather than the API.
@@ -1,225 +0,0 @@
 # platform/intel-gpu-plugin
 **Intel Device Plugin for Kubernetes (GPU)** — the standalone GPU plugin DaemonSet from
 [`intel/intel-device-plugins-for-kubernetes`](https://github.com/intel/intel-device-plugins-for-kubernetes),
 pinned at **v0.37.0**. It makes every node advertise **`gpu.intel.com/i915`** so a workload can
 *request* a GPU instead of hoping a `hostPath` is enough.
 - **Serves:** hardware transcoding for Plex (`hardware.gpu: intel-quicksync`), and any later
  consumer of an Intel iGPU.
 - **Tier:** its **own** Flux Kustomization
  ([`clusters/base/intel-gpu-plugin.yaml`](../../clusters/base/intel-gpu-plugin.yaml),
  `dependsOn: cilium`, `wait: true`) — **not** a line in `platform/kustomization.yaml`. See
  *Placement* below.
 - **Namespace:** `intel-gpu-plugin` — a PLATFORM namespace, labelled
  `homelab.keithjackson.dev/app-namespace: "false"`, so the app-scoped Kyverno policies
  (default-deny generation, auth-mode, egress-class, host-namespaces) do not apply.
 - **No RBAC, no ServiceAccount, no CRDs, no webhook, no API-server access.** A device plugin talks
  to the *local kubelet* over a unix socket.
 ## What this component owns
 | File | Purpose |
 |---|---|
 | `namespace.yaml` | `intel-gpu-plugin` ns, explicitly `app-namespace=false` |
 | `intel-gpu-plugin.yaml` | the DaemonSet, **vendored** (machine-generated) from upstream v0.37.0 |
 | `kustomization.yaml` | namespace scoping + the pinned image (`images:` transformer) |
 ## The problem this fixes
 Plex was given a `/dev/dri` hostPath (merged on `main` @ `cdf3e14`). The device is **visible** in
 the container and **cannot be opened**. Measured in the running pod on `k3s-prod-2`:
 ```
 uid 0,    O_RDWR -> EPERM "Operation not permitted"
 uid 1000, O_RDWR -> EPERM "Operation not permitted"
 ```
 `EPERM` from the **device cgroup**, not `EACCES` from file permissions — proven three ways:
 1. root on the **host** opens the same device with `rc=0`;
 2. the node is unified `cgroup2fs`;
 3. the container's runc `config.json` reads
   `linux.resources.devices: [{'allow': False, 'access': 'rwm'}]` and `linux.devices: None` —
   one deny-all rule, no allow entry for major 226, and the runtime was never asked to grant a
   device.
 A hostPath is a **filesystem** operation. It makes the device *node* appear and adds no cgroup
 allow rule, because Kubernetes cannot infer one from a volume.
 **A device plugin is the mechanism that closes exactly this gap.** Verified in the v0.37.0 source
 (`cmd/gpu_plugin/gpu_plugin.go`, `createDeviceSpecsFromDrmFiles`), the `Allocate` response carries
 per DRM node:
 ```go
 specs = append(specs, pluginapi.DeviceSpec{
    HostPath: devPath, ContainerPath: devPath, Permissions: "rw",
 })
 ```
 and it is that `DeviceSpec` which makes kubelet populate `linux.devices` **and** the matching
 cgroup allow rule. It is the only **non-privileged** way to get there. `privileged: true` would
 also work and is not available: `platform/kyverno/policies/restrict-host-namespaces.yaml` blocks it
 under Enforce in app namespaces, and `plex` is labelled `app-namespace=true` (measured
 2026-10-01). Correctly so.
 ## Placement — why its own Kustomization
 The cheap option was one line in the shared `platform` tier. Neither that file nor a new
 `platform/<name>/` directory is a sensitive path in `infra/gitea/approval-policy.json`, so it would
 have needed no human reviewer *on the policy's terms*. It is still the wrong place, and the repo
 already argued the case when Cilium was split out (read the comment in
 `platform/kustomization.yaml`):
 * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot distinguish
  "the GPU plugin is wedged" from "longhorn is slow".
 * The only emergency valve would be `flux suspend kustomization platform` — freezing storage, TLS,
  DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device plugin.
 * The tier's 10m/5m interval/timeout are called out there as **wrong for a 3-node DaemonSet
  rollout**. This is a 3-node DaemonSet rollout.
 The consequence is worse than average here because of what sits downstream: `platform` gates
 `platform-config`, which gates `apps-wave-0` (`wait: true`), which gates `apps-wave-1`. A plugin
 that never went healthy inside that tier would stall **every app** on "dependency not ready" — so a
 component added to fix a *silent no-op* (software transcoding) would have acquired the ability to
 cause a *real outage*.
 The usual objection — that `clusters/**` needs a human approval — costs nothing: Gitea enforces
 `required_approvals: 1` with `block_admin_merge_override: true` on **every** PR here regardless of
 path, and authors cannot self-approve. The approval was required either way.
 ### `wait: true` is a health signal, not a gate
 This departs from the `headlamp`/`crafty` leaf template, deliberately. Those run `wait: false`
 because a leaf UI can be legitimately not-Ready for a long time during a cold start while it waits
 on Keycloak/DNS/TLS, making its health useless as a signal. This DaemonSet has no such dependency —
 it reads sysfs and opens a socket, so on healthy nodes it is Ready in seconds and **not-Ready
 genuinely means broken**. `wait: true` makes `flux get kustomization intel-gpu-plugin` a one-line
 answer to "did it come up on all three nodes", which is precisely the question that must be
 answered before the consumer PR may merge.
 Nothing in the graph `dependsOn` this, so `wait: true` gates nothing. **Think hard before adding
 `dependsOn: [intel-gpu-plugin]`** anywhere — for the app waves that would hand a device plugin the
 power to stall every app, which is the thing this placement exists to prevent.
 ## Deliberate non-choices
 **Standalone DaemonSet, not the operator.** The operator adds a CRD (`GpuDevicePlugin`), a
 controller-manager Deployment, cert-manager-backed admission webhooks and a second namespace —
 three more things that can be unhealthy — to buy declarative management of a DaemonSet that is
 already declared in git. For three identical nodes and one consumer that is strictly more blast
 radius for no gain.
 **It lists devices; it never opens them** — worth being certain about, since *opening* is exactly
 what fails today. Verified in v0.37.0 source: the scan path uses only `os.ReadDir`, `os.ReadFile`,
 `os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. There is no `os.Open`, no
 `OpenFile`, no `ioctl` and no raw syscall against a DRM node anywhere in
 `cmd/gpu_plugin/gpu_plugin.go`. Listing and `stat()` demonstrably work today — that is how the Plex
 probe found the device before failing to open it — so the plugin needs no cgroup grant of its own
 and no privilege. Accordingly `/dev/dri` and `/sys/class/drm` are mounted **readOnly**.
 * **`--health-management` stays OFF** (upstream default). It is the one flag that *would* need to
  open a device: it starts a Level-Zero sidecar (`setupLevelZeroService` returns early unless the
  flag is set). Enabling it would reintroduce the exact `EPERM` this component exists to fix.
 * `/dev/dri/by-path` being absent is non-fatal — `newDevicePlugin` warns and sets
  `bypathFound = false`.
 **`--shared-dev-num` left at the upstream default of 1** (i.e. not set). The flag multiplies each
 card into N schedulable device IDs, letting N containers share one GPU. Not raised, because:
 * Plex is the only consumer today and wants one device;
 * the count is a **lie the scheduler believes** — there is no GPU-side partitioning or quota behind
  it, so N containers get the same `renderD128` and contend for the same two media engines
  (`vcs0`/`vecs0`). Raising it converts "Unschedulable", which is loud and obvious, into
  "transcodes are mysteriously slow", which is neither;
 * it is a one-line, zero-migration change on the day a second consumer exists — and *then* the
  right number will be known rather than guessed.
 **No NFD overlay.** All three nodes are identical (HP EliteDesk 800 G5 Mini, i7-9700T, UHD Graphics
 630, PCI `8086:3e98`, `i915` bound), so a nodeSelector on a GPU feature label would gate on
 something true everywhere. If a GPU-less node ever joins, add upstream's `nfd` overlay *then*
 rather than relying on the plugin's behaviour on such a node.
 **No `supplementalGroups`, anywhere.** The `pms-docker` image's own cont-init hook stats each
 `/dev/dri` char device and adds the `plex` user to its group — measured live as
 `Groups: 44 100 992 1000` on the running Plex process, where GID 992 is the host `render` group.
 That mechanism is image-side and works identically on a plugin-injected device.
 ## Resource name
 `gpu.intel.com/i915`. Not a guess: `scan()` calls `devTree.AddDevice(devProps.driver(), ...)`, so
 the suffix is the **kernel driver name**. These nodes have `i915` bound (not `xe`), so the resource
 is `i915`. A future kernel moving CoffeeLake to `xe` would silently rename it to `gpu.intel.com/xe`
 and a consumer's request would stop matching — worth knowing before that kernel arrives.
 ## Versioning
 Pinned to **v0.37.0** via the `images:` transformer in `kustomization.yaml`; the DaemonSet itself is
 vendored as `kubectl kustomize` output of the upstream base (see its header for the exact re-vendor
 command and both sha256s). It is *generated* rather than a byte copy because upstream mixes YAML
 sequence-indentation styles within the one document and `ci/yamllint.yaml` sets
 `indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate
 (measured: it did). `kubectl kustomize` normalises the formatting, and the result was proven to be
 a formatting-only change: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file
 diff clean. This is also the re-vendor idiom `platform/external-snapshotter` documents.
 Vendored rather than referenced as a remote kustomize base for the same reason as
 `platform/external-snapshotter`: a remote base is `git fetch`ed on **every** reconcile, those
 fetches intermittently exceeded kustomize-controller's 27s timeout, and transient GitHub slowness
 then rendered as "kustomize build failed". A local file makes the build offline and deterministic —
 which also means the CI gate's `kustomize build` proves the manifests rather than proving GitHub
 was up.
 Renovate tracks it through the inline-marker idiom (`# renovate: datasource=github-releases
 depName=...` immediately above the version literal), because `renovate.json` sets
 `"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}` — an image reference in a
 plain manifest is tracked by nothing. **Caveat:** upstream *release* tags carry a `v` (`v0.37.0`)
 while *image* tags do not (`0.37.0`). Whether Renovate normalises that for this dep is unverified;
 if it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
 `renovate.json`, not a hand-edit.
 ## Ordering — do not reorder the stack
 The renderer change (`catalog/render.py` emitting `gpu.intel.com/i915: 1` for
 `hardware.gpu: intel-quicksync` instead of the hostPath) **must not land until this plugin is
 merged AND confirmed advertising the resource on all three nodes.** If the renderer requests a
 resource no node advertises, the Plex pod is **Unschedulable**; `apps-wave-0` is `wait: true` and
 `apps-wave-1` `dependsOn` it, so that stalls the whole app tier — turning today's silent no-op into
 a real outage.
 ## Post-merge verification
 Nothing about this component's runtime behaviour can be verified from a PR: it does not exist until
 the operator merges and Flux reconciles. Afterwards:
 ```bash
 D=./infra/devcontainer/dev.sh
 # 1. Flux applied it, and (wait:true) the DaemonSet went healthy.
 $D run bash -c 'flux get kustomization intel-gpu-plugin'
 $D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin get ds intel-gpu-plugin'
 #    want: DESIRED=3 CURRENT=3 READY=3 AVAILABLE=3
 # 2. The resource is advertised on ALL THREE nodes. This is the gate on the renderer PR.
 $D run bash -c 'kubectl --context k3s-prod get nodes -o json \
  | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
 #    want: three lines, each i915=1   (before this change: ABSENT on all three)
 # 3. If a node shows ABSENT, read that node's plugin logs.
 $D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin logs -l app=intel-gpu-plugin --tail=50 --prefix'
 ```
 ### The decisive test
 Resource-advertised is necessary but **not** sufficient — it proves the plugin registered, not that
 the cgroup allow rule reaches Plex. The decisive test is re-running the `open()` probe **as uid 1000
 through `s6-setuidgid plex`** inside the Plex pod, *after* the renderer PR has switched Plex to
 request the resource:
 ```bash
 $D run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
  s6-setuidgid plex python3 -c "import os; os.close(os.open(\"/dev/dri/renderD128\", os.O_RDWR)); print(\"rc=0 OK\")"'
 ```
 **Passing as uid 0 is weaker evidence and must not be accepted as proof** — Plex transcodes as uid
 1000, and the two differ in group membership, which is the half the image's cont-init hook supplies.
@@ -1,117 +0,0 @@
 ---
 # VENDORED, MACHINE-GENERATED, from intel/intel-device-plugins-for-kubernetes @ v0.37.0:
 #   deployments/gpu_plugin/base
 #
 # RE-VENDOR on a version bump (and bump `newTag` in kustomization.yaml to match):
 #   kubectl kustomize \
 #     "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/base?ref=vX.Y.Z" \
 #     > platform/intel-gpu-plugin/intel-gpu-plugin.yaml
 #   # then re-add this header, and READ THE DIFF — upstream has changed the securityContext
 #   # and the volume set between minor releases before.
 #
 # Generated 2026-10-01 at v0.37.0; sha256 of the body below (header excluded):
 #   80ebd070f983f3116f4004f719d9c872220997ec766b4ab7e7290585ac8ef842
 # The raw upstream source file reviewed to produce it
 # (deployments/gpu_plugin/base/intel-gpu-plugin.yaml) was sha256
 #   c8773918c2370e711d4ba84120a9971a290bf75ecf81dc520782bcccd7ef24b6
 # and the two were proven EQUIVALENT, not merely similar: `yq -P 'sort_keys(..)'` of each diffs
 # clean, so passing it through kustomize changed formatting only and no content.
 #
 # WHY GENERATED RATHER THAN A BYTE COPY of the upstream file: upstream mixes YAML sequence
 # indentation styles within the one document (`containers:` non-indented, `env:` indented), and
 # ci/yamllint.yaml sets `indentation: indent-sequences: consistent`, so a verbatim copy is a hard
 # lint ERROR and blocks the merge gate — measured, this file failed exactly that way first time
 # round. `kubectl kustomize` normalises the output, which is also precisely the re-vendor idiom
 # platform/external-snapshotter documents for the same reason.
 #
 # WHY VENDORED AT ALL rather than a remote kustomize base (`...gpu_plugin/base?ref=v0.37.0` left in
 # `resources:`): the same reason platform/external-snapshotter is vendored. A remote base is
 # `git fetch`ed by Flux's kustomize-controller on EVERY reconcile, those fetches intermittently
 # exceeded the controller's 27s timeout, and a transient GitHub slowness then rendered as
 # "kustomize build failed" on the whole tier. A local file makes the build OFFLINE and
 # deterministic — which also means `kustomize build` in the CI gate proves the real thing instead
 # of proving GitHub was up. (platform/kubevirt still carries the remote-base pattern and its latent
 # fragility; not this component's problem.)
 #
 # DO NOT HAND-EDIT THE BODY. Every local change belongs in kustomization.yaml as a transformer or
 # patch, so the next re-vendor stays a mechanical regenerate-and-diff instead of a three-way merge.
 # The image tag below is re-pinned (and fully qualified) by the `images:` transformer there. The
 # rationale for each thing deliberately NOT changed — no privileged, no supplementalGroups, no
 # -shared-dev-num, no NFD overlay, no health-management — is in README.md.
 apiVersion: apps/v1
 kind: DaemonSet
 metadata:
  labels:
    app: intel-gpu-plugin
  name: intel-gpu-plugin
 spec:
  selector:
    matchLabels:
      app: intel-gpu-plugin
  template:
    metadata:
      labels:
        app: intel-gpu-plugin
    spec:
      containers:
      - env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        image: intel/intel-gpu-plugin:0.37.0
        imagePullPolicy: IfNotPresent
        name: intel-gpu-plugin
        resources:
          limits:
            cpu: 100m
            memory: 115Mi
          requests:
            cpu: 40m
            memory: 45Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          seLinuxOptions:
            type: container_device_plugin_t
          seccompProfile:
            type: RuntimeDefault
        volumeMounts:
        - mountPath: /dev/dri
          name: devfs
          readOnly: true
        - mountPath: /sys/class/drm
          name: sysfsdrm
          readOnly: true
        - mountPath: /var/lib/kubelet/device-plugins
          name: kubeletsockets
        - mountPath: /var/run/cdi
          name: cdipath
      nodeSelector:
        kubernetes.io/arch: amd64
      volumes:
      - hostPath:
          path: /dev/dri
        name: devfs
      - hostPath:
          path: /sys/class/drm
        name: sysfsdrm
      - hostPath:
          path: /var/lib/kubelet/device-plugins
        name: kubeletsockets
      - hostPath:
          path: /var/run/cdi
          type: DirectoryOrCreate
        name: cdipath
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
@@ -1,99 +0,0 @@
 ---
 # Intel GPU device plugin for Kubernetes — makes the nodes advertise `gpu.intel.com/i915`.
 #
 # WHAT THIS FIXES, and why nothing cheaper works. Plex already has a `/dev/dri` hostPath mount
 # (merged on main @ cdf3e14). The device node is VISIBLE in the container and cannot be OPENED.
 # Measured in the running pod on k3s-prod-2:
 #     uid 0,    O_RDWR -> EPERM "Operation not permitted"
 #     uid 1000, O_RDWR -> EPERM "Operation not permitted"
 # EPERM from the DEVICE CGROUP, not EACCES from file permissions — proven three ways: root on the
 # HOST opens the same device rc=0; the node is unified cgroup2fs; and the container's runc
 # config.json carries one deny-all rule and nothing else:
 #     linux.resources.devices: [{'allow': False, 'access': 'rwm'}]
 #     linux.devices: None
 # No allow entry for major 226, and the runtime was never asked to grant a device. A hostPath is a
 # FILESYSTEM operation: it makes the device node appear, and adds no cgroup allow rule, because
 # Kubernetes cannot infer one from a volume.
 #
 # A device plugin is the mechanism that closes exactly that gap. Verified by reading the v0.37.0
 # source (cmd/gpu_plugin/gpu_plugin.go, createDeviceSpecsFromDrmFiles): the Allocate response
 # carries `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, and it
 # is that DeviceSpec which makes kubelet populate `linux.devices` AND the matching cgroup allow
 # rule. It is the only NON-PRIVILEGED way to get there. `privileged: true` would also work and is
 # not available: platform/kyverno/policies/restrict-host-namespaces.yaml blocks it under Enforce in
 # app namespaces, and `plex` is labelled app-namespace=true (measured 2026-10-01). Correctly so.
 #
 # STANDALONE DAEMONSET, NOT THE OPERATOR. Upstream ships both. The operator adds a CRD
 # (GpuDevicePlugin), a controller-manager Deployment, cert-manager-backed admission webhooks and a
 # second namespace — i.e. three more things that can be unhealthy — to buy declarative management
 # of a DaemonSet we are declaring in git anyway. For three identical nodes and one consumer that is
 # strictly more blast radius for no gain. The DaemonSet has no CRDs, no controller, no webhook, and
 # no API-server access at all: a device plugin talks to the LOCAL kubelet over a unix socket, which
 # is why there is no ServiceAccount or RBAC here (and therefore no `**/rbac.yaml` sensitive path).
 #
 # LISTING, NOT OPENING — the thing worth being sure about, since opening is precisely what fails
 # today. Verified in v0.37.0 source: the scan path uses only os.ReadDir / os.ReadFile / os.Stat /
 # os.Readlink over /sys/class/drm and /dev/dri. There is no os.Open, no OpenFile, no ioctl and no
 # syscall of any kind against a DRM node anywhere in cmd/gpu_plugin/gpu_plugin.go. Listing and
 # stat() demonstrably work today (that is how the Plex probe found the device before failing to
 # open it), so the plugin needs no cgroup grant of its own and no privilege.
 #   * `--health-management` is left OFF (upstream default). It is the one flag that WOULD need to
 #     open a device: it starts a Level-Zero sidecar (setupLevelZeroService returns early unless the
 #     flag is set). Turning it on would reintroduce the exact EPERM this component exists to fix.
 #   * /dev/dri and /sys/class/drm are mounted readOnly, consistent with list-and-stat only.
 #   * /dev/dri/by-path absence is non-fatal (newDevicePlugin warns and sets bypathFound=false).
 #
 # RESOURCE NAME: `gpu.intel.com/i915`. Not a guess — `scan()` does
 # `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the KERNEL DRIVER name. These nodes
 # have i915 bound (not xe) on UHD Graphics 630 / PCI 8086:3e98, so the resource is i915. A future
 # kernel moving CoffeeLake to xe would silently rename it to `gpu.intel.com/xe` and the consumer's
 # request would stop matching — worth knowing before that kernel arrives.
 #
 # `--shared-dev-num` LEFT AT THE UPSTREAM DEFAULT OF 1 (so: not set). The flag multiplies each
 # card into N schedulable device IDs (`for i := 0; i < sharedDevNum; i++`), letting N containers
 # share one GPU. Deliberately not raised:
 #   * Plex is the only consumer and wants one device.
 #   * The count is a LIE the scheduler believes. There is no GPU-side partitioning or quota behind
 #     it — N containers get the same renderD128 and contend for the same two media engines
 #     (vcs0/vecs0). Raising it converts "Unschedulable", which is loud and obvious, into
 #     "transcodes are mysteriously slow", which is neither.
 #   * It is a one-line, zero-migration change on the day a second consumer actually exists, and at
 #     that point the right number will be known rather than guessed.
 # No NFD overlay either: all three nodes are identical (established cluster fact), so a nodeSelector
 # on a GPU feature label would gate on something that is true everywhere. If a GPU-less node ever
 # joins, add upstream's nfd overlay THEN rather than relying on the plugin's behaviour there.
 #
 # NO supplementalGroups, anywhere, on purpose. The pms-docker image's own cont-init hook stats each
 # /dev/dri char device and adds the `plex` user to its group — measured live as `Groups: 44 100 992
 # 1000` on the running Plex process, where GID 992 is the host `render` group. That mechanism is
 # image-side and works identically on a plugin-injected device, so adding a supplementalGroup here
 # or on the consumer would duplicate a thing that already works.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 namespace: intel-gpu-plugin
 resources:
  - namespace.yaml
  - intel-gpu-plugin.yaml
 images:
  # PINNED, never `latest`. Upstream release v0.37.0 (2026-09-16); verified present on Docker Hub
  # 2026-10-01, manifest digest sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1
  # (single amd64 manifest, not an index). Recorded here rather than pinned by digest because this
  # repo pins upstream platform components by VERSION TAG (platform/kubevirt, external-snapshotter);
  # digest pinning is reserved for the conformance runner, where it is a security boundary.
  # `newName` fully qualifies the registry: upstream writes the bare `intel/intel-gpu-plugin`, which
  # relies on containerd's docker.io default rather than saying so.
  #
  # RENOVATE: the inline-marker idiom (customManagers in renovate.json), not the docker datasource —
  # renovate.json sets `"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}`, so an
  # image reference in a plain manifest is tracked by NOTHING. The marker must sit on the line
  # IMMEDIATELY above the version literal for the regex to match, which is why it is down here on
  # `newTag` and not up in this header.
  # CAVEAT, stated because it will bite someone: the GitHub release tags carry a `v`
  # (`v0.37.0`) and the IMAGE tags do not (`0.37.0` — confirmed against the Docker Hub tag list).
  # Whether Renovate normalises that for this dep is UNVERIFIED here (no Renovate run available).
  # If it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
  # renovate.json, not a hand-edit of this line.
  - name: intel/intel-gpu-plugin
    newName: docker.io/intel/intel-gpu-plugin
    # renovate: datasource=github-releases depName=intel/intel-device-plugins-for-kubernetes
    newTag: "0.37.0"
@@ -1,34 +0,0 @@
 ---
 # intel-gpu-plugin — namespace for the Intel GPU device plugin DaemonSet.
 #
 # PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and that distinction is load-bearing here rather
 # than cosmetic. Every Kyverno policy that could object to this DaemonSet is scoped by a
 # `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"` — verified by
 # reading the policies, not assumed:
 #   * platform/kyverno/policies/restrict-host-namespaces.yaml  (hostNetwork/PID/IPC + privileged)
 #   * platform/kyverno/policies/require-egress-class-label.yaml
 #   * platform/kyverno/policies/require-auth-mode-label.yaml
 #   * platform/kyverno/policies/require-default-deny-baseline.yaml (a *generate* rule, so an
 #     app-namespace also gets a default-deny NetworkPolicy minted into it)
 # Labelling this namespace "false" therefore puts it out of scope for all four. That is the same
 # mechanism restrict-host-namespaces' own header describes: "platform components that do [need host
 # access] live in platform namespaces that do NOT carry the app-namespace label".
 #
 # Measured on prod 2026-10-01: `kyverno` and `vpn-egress` are the two namespaces that carry the
 # label explicitly as "false"; the rest of the platform namespaces leave it unset. Both are out of
 # scope identically — an unset label does not match `"true"`. We set it EXPLICITLY, following
 # vpn-egress, because "someone chose this" and "nobody thought about it" should not look the same
 # in git, and because this component's correctness depends on the answer.
 #
 # NOTE this plugin does NOT actually need any of the things those policies forbid: it sets no
 # hostNetwork/hostPID/hostIPC and is not privileged (see intel-gpu-plugin.yaml). The namespace
 # placement is not a workaround for a policy — there is nothing to work around. It is here because
 # a node-level platform DaemonSet is not an application, and because the app-namespace fabric
 # (default-deny NetworkPolicy, egress-class, auth-mode) describes obligations that are meaningless
 # for a process that talks only to the local kubelet over a unix socket and never uses the network.
 apiVersion: v1
 kind: Namespace
 metadata:
  name: intel-gpu-plugin
  labels:
    homelab.keithjackson.dev/app-namespace: "false"
@@ -115,14 +115,6 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
  plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
  token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
  Includes why a changed Secret does not restart anything, and why an expired token is harmless.
 - **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod
  (`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware
  acceleration when available"**, and no environment variable can do it (the image supports no
  `PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`,
  read out of the binary), says which widely-cited key **does not exist**, and is explicit that the
  **only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no
  alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin
  fallback, and why a `:latest` pull can silently break it.
 - **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
  re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
  (the four lines that matter, including the upstream `-` that means the proxy answered a request the
@@ -95,12 +95,6 @@ configMapGenerator:
      # because the plex.tv claim window is short and is not a thing to go looking
      # for in git: /runbooks/plex-claim has to resolve the moment it is needed.
      - plex-claim=plex-claim.md
      # Operator procedure, not an alert page, and NO alert is possible for it: plex renders no
      # metricsEndpoint, so there is no ServiceMonitor, no transcode-mode metric and nothing a rule
      # could evaluate. Served because the feature it completes is INERT until a human ticks a box in
      # the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped
      # feature stays switched off for months.
      - plex-quicksync=plex-quicksync.md
      - pvc-capacity=pvc-capacity.md
      - targets-down=targets-down.md
      - update-rollback=update-rollback.md
@@ -1,283 +0,0 @@
 # plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did
 **Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**,
 not an incident page. It is indexed here because the feature it completes is **inert until a human
 clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
 to prevent.
 **Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the
 rendered golden — the `dri` hostPath volume and the `/dev/dri` mount), `catalog/render.py` (the
 mechanism), `schemas/renderer-contract.md` `### hardware` (the contract).
 **Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile.
 ## What this means here
 Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync
 moves that work off the CPU — which matters on a three-node cluster where the CPU is also running
 everything else. Two independent things have to be true before that happens, and **only one of them
 is in git**:
 | Half | Who does it | State |
 |---|---|---|
 | The render device is inside the container at `/dev/dri` | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders a `dri` hostPath volume (`/dev/dri`, `type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container. Flux applies it. | Done by merging that change. |
 | Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
 **There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
 grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`,
 `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin` — found **zero** `PLEX_PREFERENCE*` support.
 Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentifier`,
 `ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`,
 `TranscoderTempDirectory`), none of them the transcode toggle. No environment variable, no ConfigMap
 and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this;
 there isn't one, and the search has already been done.
 What the device mount does **not** do, so nobody reads more into a green gate than is there: it does
 not enable hardware transcoding, it does not prove the device can be *opened* from inside a
 non-privileged pod (see "Likely causes" #2), and it renders no `nodeSelector` — Plex can land on any
 of the three nodes and all three carry the device.
 ## Running the commands in this runbook
 `kubectl` does not work on the operator workstation; the toolchain lives only in the pinned dev
 container. Every command below runs as:
    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
 Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
 picks a current-context **per invocation**, and that default has already changed once. Do not run two
 `dev.sh` invocations at once.
 **Never `cat` `Preferences.xml`.** It contains `PlexOnlineToken`, a plex.tv account credential. Every
 read of that file below greps a single named key — do not widen one.
 ## First commands
 **1. Is the device mounted into the pod at all?** (Proves Flux applied the change, not that it works.)
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
     -o jsonpath="{.items[0].spec.containers[0].volumeMounts[*].mountPath}"; echo'
 ```
 `/dev/dri` in that list ⇒ the GitOps half landed. Absent ⇒ the running pod predates the merge; compare
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 \
     -o jsonpath="{.status.lastAppliedRevision}"; echo'
 ```
 against the merge commit before blaming anything else.
 **2. Is the device actually visible inside the container?**
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'
 ```
 Expect `renderD128` (usually `card0` too), mode `crw-rw----`, group `render`. An **empty directory**
 here is the `DirectoryOrCreate` degradation: the kubelet created an empty `/dev/dri` because the node
 had none. That is the designed-safe outcome — the app tier still reconciles — but it means software
 transcoding, silently. Find which node the pod is on and ask why that node lost its iGPU:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
 ```
 **3. Can Plex *open* it?** This is the question the manifest cannot answer — see "Likely causes" #2.
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
     bash -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
 ```
 `RW-OPEN-DENIED` (or `Operation not permitted` / `Permission denied`) with the device plainly visible
 in step 2 is the **device-cgroup** failure, not a file-permission failure. The two look identical from
 the outside and are not the same problem.
 **4. Is the toggle on?** Named key only — never the whole file:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
     grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
     "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'
 ```
 `ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on,
 and the remaining question is whether it *works* — go to "How to verify".
 ## The one manual step
 Plex web UI → **Settings → Transcoder → "Use hardware acceleration when available"** → tick → Save.
 (It sits under *Show Advanced*.) A transcode already in flight is unaffected; the next one picks it up.
 The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by
 extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from
 recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise
 them and not invent them:
 | Key | What it is |
 |---|---|
 | `HardwareAcceleratedCodecs` | the toggle this page is about |
 | `HardwareAcceleratedEncoders` | hardware **encode**, as distinct from decode |
 | `HardwareDevicePath` | how Plex is pointed at a *specific* render node — relevant only if a host ever exposes more than one |
 | `TranscoderToneMapping`, `TranscoderToneMappingAgorithm` | HDR→SDR tone mapping. **That misspelling of "Algorithm" is upstream's, in the binary. It is not a typo in this page — do not "fix" it** here or anywhere else; the corrected spelling is not a key Plex reads. |
 | `TranscoderHEVCEncoding` | HEVC output |
 **`HardwareAcceleratedCodecsEnableHEVC` does not exist.** It is widely cited and it is not in the
 binary. Recorded explicitly so nobody spends an evening hunting for it or, worse, writes it into
 `Preferences.xml` and concludes hardware transcoding is broken.
 Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn
 tier, it is in no git commit, and the UI writes the key correctly. Use the UI.
 ## How to verify — and why there is no alert
 Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a
 lower quality explicitly), then read Plex's own **Dashboard → Activity**:
 - **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.**
 - plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says.
 **That dashboard is the *only* proof available, and nothing else will ever tell you.**
 `catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
 there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work
 to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently
 inert, and this repo has already been burned at scale by rules that evaluated against nothing while
 looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that
 gap is ever worth closing, the honest order is a real exporter first and a rule second.
 Corroborating, not a substitute — the server log on the config PVC:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
     sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
     \"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'
 ```
 Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions.
 Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence
 of hardware. The dashboard remains the verdict.
 ## Likely causes, most likely first
 When the dashboard still says plain `Transcode`:
 1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists.
   First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
   investigating anything technical.
 2. **containerd's device cgroup is refusing the `open()` — UNVERIFIED, and the largest remaining
   technical risk in this feature.** Whether a non-privileged pod may `open()` a bind-mounted
   character device on this k3s/containerd build **has not been established**; it can only be settled
   by deploying and trying. The container inits as uid 0, which settles *file* permissions, but
   **uid 0 does not bypass the cgroup device gate.** Symptom: step 2 shows `renderD128` present and
   correctly owned, step 3 returns `RW-OPEN-DENIED`.
   **The documented fallback is the Intel Device Plugin for Kubernetes** (`gpu.intel.com/i915` as a
   resource request, no hostPath). Because the ServiceDefinition field expresses *intent* rather than
   a device path, that migration is a **`catalog/render.py` change plus a re-render — with no
   ServiceDefinition edit anywhere**; `catalog/plex.yaml` keeps saying exactly what it says today.
   **`privileged: true` is not an available escape hatch:** Kyverno `restrict-host-namespaces` is in
   **Enforce** mode and rejects it in app namespaces. Do not reach for it.
 3. **The pod landed on a node without the device.** Step 2 shows an empty `/dev/dri`. All three prod
   nodes carried `renderD128` with `i915` bound when this was measured (2026-09-30), so this means a
   node changed — a rebuild, a BIOS reset, or a driver regression. `DirectoryOrCreate` is why the app
   tier still came up; the cost is that the loss is silent. **Fix the node, do not tighten the mount:**
   `Directory` would stall `apps-wave-0` (`wait: true`, with `apps-wave-1` depending on it) for a
   feature that is not load-bearing.
 4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
   mechanism this feature leans on with no commit in this repo.
 5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so
   `Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as
   a hardware failure.
 6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which
   Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
   sessions show `(hw)` and others do not. Not a fault.
 ## The floating-image caveat
 Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
 user to the device's group, the absence of `PLEX_PREFERENCE*`, the six `setPref` keys, the
 `HardwareAcceleratedCodecs` string — was read from digest **`sha256:e0ab27395614…`**
 (`plexmediaserver 1.43.4.10903`).
 The deployment runs **`plexinc/pms-docker:latest` with `imagePullPolicy: Always`**. A silent pull can
 therefore change the behaviour this feature depends on **with no commit in this repo**, while the
 manifest still looks identical. Symptom to watch for: hardware transcoding that worked and quietly
 stopped, with no git change to point at.
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
     -o jsonpath="{.items[0].status.containerStatuses[0].imageID}"; echo'
 ```
 If that digest is not the one above, **re-verify the hook before trusting this page**:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- ls /etc/cont-init.d/'
 ```
 `45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one
 described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups`
 needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that
 hook exists.
 ## When to escalate or stop
 - **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be
  **inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's
  `cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its
  group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces**
  the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with
  `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
  discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
 - **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
  reject it. If the cgroup gate is the blocker, the answer is the Intel device plugin (cause #2).
 - **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles
  `main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating
  `kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change
  through a branch, a PR and a green gate.
 - **Do not edit `Preferences.xml` on the PVC**, never `cat` it, and never `sops -d` anything into a
  terminal. It holds `PlexOnlineToken`.
 - **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and
  watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
  lose the DB.
 - **Stop and get the operator** if the device cannot be opened from the pod (cause #2 confirmed). The
  next step is a platform change — the Intel device plugin — with its own PR, not more poking at this
  workload.
 ## Known gaps, deliberately not fixed here
 Recorded so they are findable, and so nobody assumes they were overlooked:
 - **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
  `TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
  overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
  transcode scratch performance changes silently whenever the pod is rescheduled. Both candidate fixes
  need a **schema** change and are therefore out of scope here: pin Plex to prod-2 (needs a
  `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory` emptyDir and
  transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not have). The RAM option
  is the better one, because it removes the node asymmetry outright instead of working around it.
 - **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
  quality only — decode, and the feature on this page, are unaffected. Changing it needs a kernel
  parameter and a **serialized rolling reboot**, which is an operator action (CLAUDE.md §5), and this
  cluster has a documented history of that going badly (a prod node hung on a warm reboot on
  2026-09-03). Not worth coupling to this change.
 ---
 Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
 The node measurements (all three prod nodes) and the in-container image measurements cited here were
 taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
 (`plexmediaserver 1.43.4.10903`). Hardware transcoding had **not** been observed working at the time
 of writing, and the Plex-side toggle was still unset.
@@ -15,7 +15,7 @@ docs/13 exactly. Do not diverge from it here — escalate spec problems, don't f
 | `servicedefinition.schema.json` | Formal JSON Schema (draft 2020-12) for a ServiceDefinition. Enforces enums, required fields, patterns, and cross-field invariants. |
 | `renderer-contract.md` | Authoritative field → rendered-resource mapping, with requirement IDs. The contract other workstreams consume. |
-**Schema version:** 1.8.0 (in the schema `description`, which carries the full per-version changelog — this line had said `1.0.0` since 1.1.0 and is now synced). Treat any change as an interface change:
+**Schema version:** 1.0.0 (in the schema `description`). Treat any change as an interface change:
 propose on the task board, update dependent renderers/tests in the same change, never silently break a
 consumer (coordination-protocol §2).
@@ -35,9 +35,6 @@ consumer (coordination-protocol §2).
 - `ingressRoutes[].host` must match `<label>.home.keithjackson.dev`.
 - `storage[].class ∈ {longhorn, bulk}`; `exposure ∈ {internal, public}`; `egress ∈ {internet, vpn}`;
  `auth.mode ∈ {oidc, forward-auth, native, none}`.
 - `hardware.gpu ∈ {intel-quicksync}` — a host **capability class**, never a device path. The enum
  is the contract: the mechanism that satisfies it lives only in `catalog/render.py` (see
  `renderer-contract.md` §`hardware`).
 - `additionalProperties: false` everywhere — unknown keys are rejected (catches typos in review/CI).
 ## Validate examples against the schema — L1, no cluster (R-TEST-02)
@@ -160,64 +160,6 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 - **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication),
  **R-STATE-04** (bulk tier). (docs/04)
 ### `hardware` (`gpu`) — host device capability
 - **The field is INTENT, never MECHANISM.** `hardware.gpu` is an **enum of capability classes**
  (today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
  ServiceDefinition says *what class of host device this workload needs*; how the platform hands it
  over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw
  `/dev/...` path.** The entire payoff is that migrating from the hostPath mechanism below to the
  Intel GPU device plugin (`resources.limits: gpu.intel.com/i915`) is a renderer change with **zero**
  catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely
  different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one.
 - `gpu: intel-quicksync` → on the **app container only** (never a sidecar), exactly two things:
  1. a **hostPath volume** `dri` → `/dev/dri`, **`type: DirectoryOrCreate`**;
  2. a **read-write volumeMount** of `dri` at `/dev/dri` on the app container (VA-API opens the
     render node read-write, so it is deliberately not `readOnly`).
  **Mounting the device is sufficient. That is the whole mapping.**
 - **`podSecurityContext.supplementalGroups` is deliberately NOT rendered, and adding it would be a
  regression rather than a hardening.** `/dev/dri/renderD128` is `root:render` `0660`, so the obvious
  move is to push the host `render` GID in — and it does not work. **Measured in the running
  container on prod-2, 2026-09-30:** the image's own
  `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` hook `find`s every char device under
  `/dev/dri`, `stat -c '%g'` each one's GID, `groupadd`s a matching group *inside the container* when
  none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded.
  `cont-init.d` completes before `services.d`, so the account database is already correct when the
  service execs. The service then starts via `exec s6-setuidgid plex …`, and `s6-setuidgid`
  **replaces** the process group list from `/etc/group` rather than appending to it:
  `Plex Media Server` runs as `Uid: 1000  Gid: 1000  Groups: 100 1000`, with root's gid `0` — which
  PID 1 (`s6-svscan`) does hold — **gone**. A Kubernetes `supplementalGroups` therefore reaches
  `s6-svscan` and is discarded before the app ever starts. It would be inert *and* it would mislead
  whoever debugs this next. The renderer asserts its own absence in `render.py --selftest`.
  - A corollary worth having: because the hook re-`stat`s the device on every start, the host GID
    **drifting** (a node rebuild renumbering the *systemd dynamic* `render` group) is self-healing.
    No GID is pinned anywhere in this repo.
  - **Scope of that measurement, stated so nobody over-reads it.** "Mounting the device is
    sufficient" was established for an image that resolves the device group itself. A FUTURE
    consumer that runs non-root and has no equivalent init hook would still need the group solved
    somehow — and the answer for that case is a **renderer** change (per-image handling, or the
    device plugin), not a new ServiceDefinition field. The enum is what keeps that option open. The
    first such consumer must re-measure rather than assume this bullet covers it.
 - **`DirectoryOrCreate` is load-bearing — do not tighten it to `Directory`.** A node that comes back
  without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the
  workload to **software transcoding**, not fail the mount. `apps-wave-0` is `wait: true` and
  `apps-wave-1` `dependsOn` it, so a failed hostPath mount stalls the **whole app tier** for the sake
  of a feature that is not load-bearing. `/dev` is tmpfs, so the empty directory the kubelet creates
  on such a node leaves nothing durable behind.
  - **The cost of that choice, stated plainly:** a missing device is **silent**. The workload
    software-transcodes and nothing reports it. There is no metric to alert on — media apps here
    render no `metricsEndpoint`, so no ServiceMonitor exists — and an alert that cannot fire is worse
    than none (see §`observability`'s 2026-09-13 correction). The only proof of hardware transcode is
    an operator observing it in the application's own dashboard.
 - **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno
  `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and the render node
  does not need it); no `nodeSelector` (placement is a separate concern with its own justification —
  this field is about the device, not about where the pod lands); no resource requests/limits (there
  is no app-container resource render path, and inventing one here would be scope creep).
 - Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical.
  Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`.
 - **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need).
 ### `ingressRoutes[]` (`host` / `port`) — HTTPRoute + Certificate + DNS
 - Each entry renders an **HTTPRoute** attaching `host` → the service `port` on the shared Gateway.
 - TLS: the Gateway terminates with the cert-manager **wildcard `*.home.keithjackson.dev`** certificate
@@ -344,11 +286,6 @@ platform must render, and cites the requirement each mapping satisfies. Changing
   `external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it.
   Derived from `exposure` alone; enforced on every gate run by
   `conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03).
 8. **`hardware` is inert until opted into, and its mechanism is renderer-local.** An SD with no
   `hardware` key renders no volume and no volumeMount; an SD *with* it gains no `securityContext`
   either. Both directions are proven by `render.py --selftest`. The `/dev/dri` hostPath and the
   `DirectoryOrCreate` type appear in `catalog/render.py` and nowhere else, so the capability's
   implementation can be replaced without touching a single ServiceDefinition.
 ## Requirement traceability (contract-level)
 Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02
@@ -2,7 +2,7 @@
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
  "title": "ServiceDefinition",
-  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 Line truncated
+  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 / R-IDP-02).",
  "type": "object",
  "additionalProperties": false,
  "required": ["name", "source"],
@@ -139,19 +139,6 @@
        "supplementalGroups": { "type": "array", "items": { "type": "integer" }, "description": "Additional GIDs added to the first process in each container." }
      }
    },
    "hardware": {
      "type": "object",
      "description": "Optional host HARDWARE CAPABILITY the workload needs, for the container kind (docs/13). This field declares INTENT (`what device class this service needs`), never MECHANISM (`which host path to bind-mount`, `which extended resource to request`). That separation is the whole point of the enum: the platform currently satisfies `intel-quicksync` with a /dev/dri hostPath volume and a matching mount on the app container, but a later migration to the Intel GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) must be a catalog/render.py change and NOTHING else - no ServiceDefinition may need editing, and no reviewer may have to re-derive per-service device wiring. Do NOT 'simplify' this into a raw device path, a hostPath, or a resource request; that would push a platform implementation detail into the operator-facing contract and make the migration a catalog-wide edit. Absent => no device wiring at all (existing apps render byte-identical). Ignored for kind=vm (a VM gets devices through KubeVirt's own domain.devices).",
      "additionalProperties": false,
      "minProperties": 1,
      "properties": {
        "gpu": {
          "type": "string",
          "description": "GPU capability class this service needs. `intel-quicksync` = the host's Intel Quick Sync Video render device (the iGPU's /dev/dri/renderD* node) for hardware video transcode/decode. Enum, not a free-form device path, so the satisfying mechanism stays swappable (see the parent description). Extend the enum when a genuinely different capability class appears (e.g. nvidia-nvenc), never to name a different path to the same one.",
          "enum": ["intel-quicksync"]
        }
      }
    },
    "exposure": {
      "type": "string",
      "description": "internal = LAN-only (internal DNS + Gateway on LAN); public = additionally edge port-forwarded 80/443 (docs/09; R-DNS/R-NET).",