Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
d44490736c |
No files matched your search
@@ -143,8 +143,6 @@ spec:
|
||||
- name: media
|
||||
mountPath: /media
|
||||
readOnly: true
|
||||
- name: dri
|
||||
mountPath: /dev/dri
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -152,10 +150,6 @@ spec:
|
||||
- name: media
|
||||
persistentVolumeClaim:
|
||||
claimName: plex-media
|
||||
- name: dri
|
||||
hostPath:
|
||||
path: /dev/dri
|
||||
type: DirectoryOrCreate
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
|
||||
@@ -80,58 +80,6 @@ env:
|
||||
# remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
|
||||
# clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
|
||||
# reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
|
||||
# HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
|
||||
# INTENT ONLY. This says "this workload needs an Intel Quick Sync render device"; it does NOT name a
|
||||
# device path. The renderer currently satisfies it with exactly two things — a `/dev/dri` hostPath
|
||||
# volume (`type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container —
|
||||
# and nothing else: no nodeSelector, no privileged, no resources (schemas/renderer-contract.md
|
||||
# `### hardware`). If that mechanism is later replaced by the Intel GPU device plugin
|
||||
# (`gpu.intel.com/i915`), it is a `catalog/render.py` change and THIS BLOCK DOES NOT MOVE. Do not
|
||||
# "simplify" it into a raw `/dev/...` path; that is the whole point of the enum.
|
||||
#
|
||||
# THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
|
||||
# anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
|
||||
# GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
|
||||
# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Plex happens to run on k3s-prod-2 today, but
|
||||
# that is scheduler history rather than a requirement — this field deliberately renders no
|
||||
# nodeSelector, because placement is a separate concern with its own justification.
|
||||
#
|
||||
# NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
|
||||
# The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
|
||||
# the host `render` GID in as a supplemental group. It cannot work here. MEASURED IN THE RUNNING
|
||||
# CONTAINER, 2026-09-30: the image's own
|
||||
# `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device under
|
||||
# `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
|
||||
# `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
|
||||
# GID renumbering self-healing (no GID is pinned anywhere in this repo). The service then execs via
|
||||
# `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from `/etc/group` rather
|
||||
# than appending to it: `Plex Media Server` runs with `Groups: 100 1000`, and root's gid `0` — which
|
||||
# PID 1 (`s6-svscan`) does hold — is GONE. Anything Kubernetes supplies therefore reaches s6-svscan
|
||||
# and is discarded before Plex ever starts. Mounting the device is sufficient; the group is the
|
||||
# image's job and it does it correctly. `render.py --selftest` asserts the ABSENCE of a pod
|
||||
# securityContext for this field precisely so nobody adds an inert one back later.
|
||||
#
|
||||
# `DirectoryOrCreate` IS THE SAFE FAILURE MODE, not laziness. A node that comes back without the
|
||||
# device (rebuild, BIOS reset, driver regression, hardware swap) gets an empty directory and Plex
|
||||
# degrades to SOFTWARE transcoding. `Directory` would fail the mount instead — and because
|
||||
# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that would stall the WHOLE app
|
||||
# tier for a feature that is not load-bearing. The price, stated plainly: a missing device is
|
||||
# SILENT. This SD renders no `metricsEndpoint`, so there is no ServiceMonitor, no transcode-mode
|
||||
# metric, and no alert that could ever fire. The only proof is an operator reading the Plex
|
||||
# dashboard.
|
||||
#
|
||||
# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Mounting the device does
|
||||
# not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
|
||||
# is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
|
||||
# set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
|
||||
# `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin`) found ZERO `PLEX_PREFERENCE*` support,
|
||||
# and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
|
||||
# `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
|
||||
# unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
|
||||
# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took and the
|
||||
# device-cgroup failure mode to expect first.
|
||||
hardware:
|
||||
gpu: intel-quicksync
|
||||
exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
|
||||
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
||||
storage:
|
||||
|
||||
@@ -163,15 +163,6 @@ OAUTH2_COOKIE_REFRESH = "2m30s"
|
||||
# are added. Do not read this flag as "metrics are working".
|
||||
OAUTH2_PROXY_METRICS_PORT = 44180
|
||||
|
||||
# --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
|
||||
# The ServiceDefinition declares the CAPABILITY (schemas/servicedefinition.schema.json
|
||||
# `hardware.gpu`); everything below is the MECHANISM that currently satisfies it and is
|
||||
# deliberately confined to this file. Swapping to the Intel GPU device plugin
|
||||
# (`gpu.intel.com/i915` resource requests instead of a hostPath) is a change here and
|
||||
# nowhere else -- no catalog entry moves.
|
||||
DRI_VOLUME_NAME = "dri"
|
||||
DRI_DEVICE_DIR = "/dev/dri"
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping.
|
||||
@@ -522,46 +513,6 @@ def r_deployment(s: dict) -> dict:
|
||||
"persistentVolumeClaim": {"claimName": f"{s['name']}-{vol['name']}"},
|
||||
}
|
||||
)
|
||||
# hardware capability (schema-optional): `hardware.gpu: intel-quicksync` means this
|
||||
# workload needs the host iGPU's render node for hardware video transcode. The SD says
|
||||
# only that; the wiring below is the platform's current mechanism for satisfying it.
|
||||
#
|
||||
# `type: DirectoryOrCreate` is DELIBERATE and load-bearing -- do NOT "tighten" it to
|
||||
# `Directory`. A node that comes back without the render device (rebuild, BIOS change,
|
||||
# driver regression, hardware swap) must degrade the workload to SOFTWARE transcoding,
|
||||
# not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` dependsOn it, so a
|
||||
# failed hostPath mount stalls the entire app tier for the sake of a feature that is not
|
||||
# load-bearing. `DirectoryOrCreate` turns that failure mode into "transcodes in
|
||||
# software" instead of "the app tier does not reconcile". /dev is tmpfs, so the empty
|
||||
# directory the kubelet creates on such a node leaves nothing durable behind.
|
||||
#
|
||||
# The mount is deliberately NOT readOnly: VA-API opens the render node read-write.
|
||||
#
|
||||
# `podSecurityContext.supplementalGroups` is DELIBERATELY NOT SET here, and adding it
|
||||
# would be a regression, not a hardening. MEASURED in the running container on prod-2,
|
||||
# 2026-09-30: the image's own cont-init hook
|
||||
# `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` `find`s every char device
|
||||
# under /dev/dri, `stat -c '%g'` each one's GID, `groupadd`s a matching group INSIDE the
|
||||
# container when none exists, and `usermod -a -G` the app user into it -- every start,
|
||||
# nothing hardcoded. cont-init.d completes before services.d, so the account database
|
||||
# s6 reads at exec time already carries the group. The service is then started with
|
||||
# `exec s6-setuidgid plex ...`, and `s6-setuidgid` REPLACES the process group list from
|
||||
# /etc/group rather than appending to it: `Plex Media Server` runs with
|
||||
# `Uid: 1000 Gid: 1000 Groups: 100 1000` -- root's gid 0, which PID 1 (s6-svscan) does
|
||||
# have, is GONE. Anything Kubernetes puts in supplementalGroups reaches s6-svscan and is
|
||||
# then discarded before the app ever starts. MOUNTING THE DEVICE IS SUFFICIENT; the
|
||||
# group handling is the image's job and it does it correctly. A supplementalGroups entry
|
||||
# here would be inert AND would mislead whoever debugs this next.
|
||||
hw_gpu = (s.get("hardware") or {}).get("gpu")
|
||||
if hw_gpu == "intel-quicksync":
|
||||
volume_mounts.append({"name": DRI_VOLUME_NAME, "mountPath": DRI_DEVICE_DIR})
|
||||
volumes.append(
|
||||
{
|
||||
"name": DRI_VOLUME_NAME,
|
||||
"hostPath": {"path": DRI_DEVICE_DIR, "type": "DirectoryOrCreate"},
|
||||
}
|
||||
)
|
||||
|
||||
if volume_mounts:
|
||||
container["volumeMounts"] = volume_mounts
|
||||
|
||||
@@ -2579,83 +2530,6 @@ def _selftest() -> int:
|
||||
"an SD without podSecurityContext must not gain a pod securityContext",
|
||||
)
|
||||
|
||||
# 6b) hardware.gpu: intel-quicksync -> a /dev/dri hostPath volume and a read-write mount
|
||||
# on the app container, and NOTHING ELSE. The SD declares the CAPABILITY; these
|
||||
# assertions pin the MECHANISM that currently satisfies it -- including what it
|
||||
# deliberately does NOT emit -- so a change to it is a deliberate, visible edit.
|
||||
hw_docs = render(
|
||||
{
|
||||
"name": "hw",
|
||||
"source": {"image": "a:1"},
|
||||
"hardware": {"gpu": "intel-quicksync"},
|
||||
}
|
||||
)
|
||||
hw_pod = next(
|
||||
d[1] for d in hw_docs if d[1]["kind"] == "Deployment"
|
||||
)["spec"]["template"]["spec"]
|
||||
check(
|
||||
{
|
||||
"name": "dri",
|
||||
"hostPath": {"path": "/dev/dri", "type": "DirectoryOrCreate"},
|
||||
}
|
||||
in hw_pod.get("volumes", []),
|
||||
"hardware.gpu=intel-quicksync must render a /dev/dri hostPath volume "
|
||||
"with type DirectoryOrCreate (a GPU-less node must degrade, not fail the mount)",
|
||||
)
|
||||
hw_mounts = hw_pod["containers"][0].get("volumeMounts", [])
|
||||
check(
|
||||
{"name": "dri", "mountPath": "/dev/dri"} in hw_mounts,
|
||||
"hardware.gpu=intel-quicksync must mount /dev/dri READ-WRITE on the app container",
|
||||
)
|
||||
# The device mount is the WHOLE feature: no pod securityContext is invented. The image's
|
||||
# own cont-init hook adds the app user to the device's group inside the container, and
|
||||
# s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway (see the
|
||||
# emission site). Asserting the absence is what stops someone adding an inert one back.
|
||||
check(
|
||||
"securityContext" not in hw_pod,
|
||||
"hardware.gpu must NOT invent a pod securityContext (supplementalGroups is inert "
|
||||
"under s6-setuidgid; the image's cont-init hook handles the device group)",
|
||||
)
|
||||
# Nothing privileged, and no node pinning: Kyverno restrict-host-namespaces (Enforce)
|
||||
# rejects `privileged: true` in an app namespace, and placement is a separate concern
|
||||
# with its own justification -- this field is about the DEVICE only.
|
||||
check(
|
||||
"privileged" not in hw_pod.get("securityContext", {})
|
||||
and "securityContext" not in hw_pod["containers"][0]
|
||||
and "nodeSelector" not in hw_pod,
|
||||
"hardware.gpu must emit no privileged securityContext and no nodeSelector",
|
||||
)
|
||||
# The regression that matters: absence renders NOTHING. Mirrors the podSecurityContext
|
||||
# negative above -- an SD that does not opt in must be byte-identical to before.
|
||||
nohw_pod = next(
|
||||
d[1] for d in render({"name": "nohw", "source": {"image": "a:1"}})
|
||||
if d[1]["kind"] == "Deployment"
|
||||
)["spec"]["template"]["spec"]
|
||||
check(
|
||||
"volumes" not in nohw_pod
|
||||
and "volumeMounts" not in nohw_pod["containers"][0]
|
||||
and "securityContext" not in nohw_pod,
|
||||
"an SD without `hardware` must gain no dri volume, no mount and no securityContext",
|
||||
)
|
||||
# An SD that declares BOTH keeps its own podSecurityContext exactly as written -- the
|
||||
# hardware field neither adds to it nor rewrites it.
|
||||
hwm_pod = next(
|
||||
d[1]
|
||||
for d in render(
|
||||
{
|
||||
"name": "hwm",
|
||||
"source": {"image": "a:1"},
|
||||
"hardware": {"gpu": "intel-quicksync"},
|
||||
"podSecurityContext": {"fsGroup": 1001, "supplementalGroups": [44]},
|
||||
}
|
||||
)
|
||||
if d[1]["kind"] == "Deployment"
|
||||
)["spec"]["template"]["spec"]
|
||||
check(
|
||||
hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]},
|
||||
"hardware.gpu must leave the SD's own podSecurityContext untouched",
|
||||
)
|
||||
|
||||
# 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that
|
||||
# fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged.
|
||||
fa_docs = render(
|
||||
|
||||
+54
-1
@@ -1,5 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/render-golden.sh — Stage: render==golden reproducibility gate.
|
||||
# ci/render-golden.sh — Stage: the renderer is trustworthy.
|
||||
# (1) catalog/render.py passes its OWN output-contract self-test, and
|
||||
# (2) every committed apps/ golden reproduces byte-for-byte from the canonical render.
|
||||
# Two independent assertions about one subject, catalog/render.py. (1) was added
|
||||
# 2026-10-01 because nothing in ci/ or .gitea/ had ever invoked `--selftest`; see the
|
||||
# block above the invocation for why it lives here and not in a conformance check.
|
||||
#
|
||||
# WHY: the committed apps/ tree is a set of GOLDENS produced by ONE canonical
|
||||
# invocation of catalog/render.py. If a golden is hand-edited, or a catalog
|
||||
@@ -54,6 +59,54 @@ if [ ! -f "$RENDER_PY" ]; then
|
||||
summary "- render-golden: **SKIPPED** (renderer missing)"
|
||||
exit 3
|
||||
fi
|
||||
# --- The renderer's OWN self-test, before anything is compared ---------------
|
||||
# `catalog/render.py --selftest` is ~106 executed assertions on the renderer's output
|
||||
# contract (default-deny covers both directions, the Kyverno namespace hook label, a valid
|
||||
# auth-mode, the oauth2-proxy sidecar's cookie-name/-domain/-refresh flags, the
|
||||
# placeholder-substitution guard, the observability refusals...). Until now NOTHING in ci/
|
||||
# or .gitea/ invoked it — measured by grep on 2026-09-30 and recorded as the top follow-up
|
||||
# of decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md — so every one of those
|
||||
# assertions was dev-time only and a merge gate could not tell a sound renderer from a
|
||||
# gutted one.
|
||||
#
|
||||
# WHY HERE, AND NOT IN A CONFORMANCE CHECK. The lead was
|
||||
# conformance/tests/l0-static/auditability-proofs-producible/check.sh, which drives
|
||||
# `conformance/lib/{record_run,status}.py --selftest` "because nothing else drives them".
|
||||
# That loop is NOT the right hook: that check serves R-SEC-03 and its subject is the
|
||||
# EVIDENCE CHAIN (can a forged or stale run become a proof?). The renderer is not part of
|
||||
# that chain, and hanging it there would make the check's name and its requirement mapping
|
||||
# a lie. This stage's subject already IS catalog/render.py — it exists to assert that the
|
||||
# renderer's output is trustworthy — so the renderer's own contract test belongs in it.
|
||||
#
|
||||
# It runs BEFORE the catalog/empty-catalog exit-3 below on purpose: the self-test is
|
||||
# deterministic and catalog-independent (it renders synthetic ServiceDefinitions in
|
||||
# memory), so an empty catalog must not be able to skip it.
|
||||
#
|
||||
# render-golden and the self-test are INDEPENDENT assertions, which is why both are needed:
|
||||
# render-golden catches golden-vs-renderer drift but is satisfied by ANY renderer whose
|
||||
# output was re-rendered, including one that just lost a contract invariant and had all ten
|
||||
# goldens regenerated in the same commit. The self-test catches exactly that.
|
||||
#
|
||||
# FAILURE IS A HARD FAIL, NOT A SKIP: the renderer is source in this repo, so "could not
|
||||
# run its self-test" has no legitimate environmental cause once python3 and render.py are
|
||||
# both present (both asserted above). ci/gate.sh maps this stage's rc 1 to BLOCKED.
|
||||
log "Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)"
|
||||
selftest_out="$(mktemp)"
|
||||
if "$PYTHON" "$RENDER_PY" --selftest >"$selftest_out" 2>&1; then
|
||||
ok "render.py --selftest passed"
|
||||
summary "- render-selftest: **passed** (\`catalog/render.py --selftest\`)"
|
||||
else
|
||||
err "catalog/render.py --selftest FAILED — the renderer violates its own output contract."
|
||||
err " Every golden under apps/ is produced by this renderer, so the goldens below cannot"
|
||||
err " be trusted even if they reproduce byte-for-byte. Reproduce with:"
|
||||
err " ${PYTHON} catalog/render.py --selftest"
|
||||
sed 's/^/ /' "$selftest_out" >&2 || true
|
||||
summary "- render-selftest: **FAILED** (\`catalog/render.py --selftest\` — renderer contract violated)"
|
||||
rm -f "$selftest_out"
|
||||
exit 1
|
||||
fi
|
||||
rm -f "$selftest_out"
|
||||
|
||||
if [ ! -d "$CATALOG_DIR" ] || ! compgen -G "${CATALOG_DIR}/*.yaml" >/dev/null 2>&1; then
|
||||
skip "no catalog ServiceDefinitions under '${CATALOG_DIR}' — nothing to render."
|
||||
summary "- render-golden: **SKIPPED** (empty catalog)"
|
||||
|
||||
@@ -1,174 +0,0 @@
|
||||
---
|
||||
date: 2026-09-30
|
||||
branch: feat/plex-quicksync
|
||||
title: Plex opts into `hardware.gpu: intel-quicksync` — the device mount is GitOps, the Plex-side toggle is a documented manual step, and hardware transcoding has NOT been observed working
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Three things, and the third is the point of the branch.
|
||||
|
||||
1. **`catalog/plex.yaml`** — added the four lines the whole feature reduces to:
|
||||
|
||||
```yaml
|
||||
hardware:
|
||||
gpu: intel-quicksync
|
||||
```
|
||||
|
||||
placed between `env` and `exposure` to match the schema's own property order, with ~48 lines of
|
||||
comment in the file's established style. The comment records: the field is **intent, not
|
||||
mechanism** (a later move to the Intel device plugin is a `render.py` change and this block does
|
||||
not move); that `/dev/dri/renderD128` is present on **all three** prod nodes with `i915` bound;
|
||||
**why a `supplementalGroups` entry would be inert** (the image's cont-init hook does the group
|
||||
work inside the container and `s6-setuidgid` replaces the process group list from `/etc/group`);
|
||||
why `DirectoryOrCreate` is the load-bearing safe failure mode (`apps-wave-0` is `wait: true` with
|
||||
`apps-wave-1` depending on it, so a failed mount would stall the whole app tier for a feature
|
||||
that is not load-bearing); and that the Plex-side toggle is a **separate manual operator step**,
|
||||
pointing at the new runbook.
|
||||
|
||||
2. **`apps/wave-0/plex.yaml`** — re-rendered. Six added lines, nothing else, in the whole `apps/`
|
||||
tree:
|
||||
|
||||
```
|
||||
- name: dri
|
||||
mountPath: /dev/dri
|
||||
- name: dri
|
||||
hostPath:
|
||||
path: /dev/dri
|
||||
type: DirectoryOrCreate
|
||||
```
|
||||
|
||||
3. **`runbooks/plex-quicksync.md`** (new), registered in `runbooks/kustomization.yaml`'s
|
||||
`configMapGenerator` as `- plex-quicksync=plex-quicksync.md` and indexed in `runbooks/README.md`
|
||||
under "Operator procedures". **The file alone would have been a 404** — the pod serves a
|
||||
ConfigMap, not the git tree, and `conformance/tests/l0-static/alert-runbooks-resolvable` exists
|
||||
precisely because "the file is in git" and "the URL resolves" are different claims. The page
|
||||
covers: the GitOps/manual boundary as a table; the toggle (Settings → Transcoder → "Use hardware
|
||||
acceleration when available", preference key `HardwareAcceleratedCodecs`); the real neighbouring
|
||||
keys and the widely-cited one that **does not exist**; that the only proof is `Transcode (hw)` on
|
||||
the Plex dashboard **and why no alert is possible**; the device-cgroup failure to expect first
|
||||
with a concrete open() diagnostic and the Intel-device-plugin fallback; and the floating-`:latest`
|
||||
caveat with a digest check.
|
||||
|
||||
**Stacked on PR #27 (`feat/sd-hardware-capability`), which is open and unmerged.** This branch was
|
||||
cut from it rather than from `main`, so its one commit (`d9baa2a`, the schema + renderer + contract)
|
||||
is in this PR's range. That is deliberate: merging #27 is not an agent's action (CLAUDE.md §1), and
|
||||
duplicating its changes here would have been worse than showing them. **This must merge after #27.**
|
||||
|
||||
Not done, deliberately: no `--emit-waves` (it writes into `clusters/base/`, a sensitive path that
|
||||
would force a human reviewer, and it is a no-op unless the *set* of files under `apps/` changes — it
|
||||
does not here). No `podSecurityContext`. No `nodeSelector`. No alert rule. No `kubectl apply`.
|
||||
|
||||
### Two follow-ups left unclaimed, with the reasoning
|
||||
|
||||
- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
|
||||
`TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
|
||||
overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
|
||||
transcode scratch performance changes silently on reschedule, with nothing reporting it. Both
|
||||
candidate fixes need a **schema** change and so neither belongs in this PR: pin Plex to prod-2
|
||||
(needs a `nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory`
|
||||
emptyDir and transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not
|
||||
have). **The RAM option is the better one** — it removes the node asymmetry outright instead of
|
||||
working around it — which is exactly why it deserves its own PR and its own review rather than
|
||||
being smuggled in behind a four-line catalog change. Documented in the runbook's "Known gaps".
|
||||
- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
|
||||
quality only; decode and this feature are unaffected. Changing it needs a kernel parameter plus a
|
||||
**serialized rolling reboot** — an operator action under CLAUDE.md §5, on a cluster with a
|
||||
documented history of that going badly (a prod node hung on a warm reboot on 2026-09-03). Kept
|
||||
well clear of this PR.
|
||||
|
||||
### Rebase note (2026-10-01, Ruth Okonkwo) — resolution only, no redesign
|
||||
|
||||
Rebased onto `main` @ `08deb20` (was cut from `0fbf544`), restacked on the rebased
|
||||
`feat/sd-hardware-capability` (PR #27) so the stack stays coherent — this branch still carries
|
||||
exactly PR #27's commit plus its own, with no duplicate of anything already in `main`. The branch
|
||||
ref was force-updated with `--force-with-lease`, so PR #32 is the same PR with a new head.
|
||||
|
||||
**The `catalog/render.py` conflict disappeared** once PR #27 was rebased; it was never this
|
||||
branch's own conflict. **One conflict remained, in `runbooks/README.md`, purely textual.** PR #30
|
||||
(`fix/jackett-oauth2-proxy-session`, merged as `08deb20`) appended an `oauth2-proxy-session` index
|
||||
entry under *Operator procedures* at the same point this branch appended its `plex-quicksync` entry.
|
||||
**Both entries were kept, in full.** Ordering follows this file's existing convention, which is
|
||||
grouped/narrative rather than alphabetical (the *Nodes* section runs down/io/memory/disk/cpu):
|
||||
`plex-quicksync` sits next to `plex-claim` as the other Plex page, leaving `oauth2-proxy-session`
|
||||
adjacent to `crafty-admin-credential`, which cross-references the same sidecar gate. The change to
|
||||
`runbooks/README.md` is an 8-line pure insertion, so PR #30's entry is untouched.
|
||||
`runbooks/kustomization.yaml` auto-merged and **both** `configMapGenerator` entries are present.
|
||||
|
||||
Re-verified after the resolution, on the rebased stack: `--selftest` PASS (rc 0); a full re-render
|
||||
reproduces every committed golden byte-identically; and the **only** `apps/` diff against the new
|
||||
`main` is `apps/wave-0/plex.yaml`, confined to the expected six lines (the `dri` mount and its
|
||||
`DirectoryOrCreate` hostPath volume). No other service's golden moved — PR #28 added ~176 lines of
|
||||
oauth2-proxy sidecar rendering, and this confirms by measurement rather than assumption that Plex
|
||||
(`auth: native`, no sidecar) is unaffected by it.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **The render is exactly the intended change, and nothing else moved.** After
|
||||
`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` (exit 0,
|
||||
silent): `git diff --stat apps/` = `apps/wave-0/plex.yaml | 6 ++++++`, and
|
||||
`git status --porcelain apps/` = a single ` M apps/wave-0/plex.yaml`. Read line by line, the diff
|
||||
is the `dri` volumeMount and the `dri` hostPath volume, and nothing more. **Specifically checked:**
|
||||
the same invocation also rewrites `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, which
|
||||
are git-tracked but excluded from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in
|
||||
`ci/render-golden.sh` — a spurious diff there would have landed **ungated**. Neither file shows a
|
||||
diff; whole-repo `git status --porcelain` lists only `apps/wave-0/plex.yaml` and `catalog/plex.yaml`
|
||||
plus my new files.
|
||||
- **The rendered output matches what the contract promises** — `schemas/renderer-contract.md`
|
||||
`### hardware` says exactly two things (a `/dev/dri` hostPath with `type: DirectoryOrCreate` and a
|
||||
read-write mount on the app container) and exactly two things appeared. No `securityContext`, no
|
||||
`nodeSelector`, no resources in the diff.
|
||||
- **The runbook is projected into the server, not merely committed.** `runbooks/kustomization.yaml`
|
||||
carries `- plex-quicksync=plex-quicksync.md`, whose shape matches the generator-entry regex the
|
||||
L0 check parses; the extensionless key is what makes `/runbooks/plex-quicksync` a plain static
|
||||
lookup. Also indexed in `runbooks/README.md`.
|
||||
- **The runbook clears the L0 quality floor** that `alert-runbooks-resolvable` applies to
|
||||
alert-linked pages, even though nothing links to this one (it is advisory for an unreferenced
|
||||
page): all four required sections (`## What this means here`, `## First commands`,
|
||||
`## Likely causes, most likely first`, `## When to escalate or stop`), well over the 1500-byte
|
||||
floor, `dev.sh` used throughout, and `--context k3s-prod` on every command.
|
||||
- **The gate.** See the run recorded in the PR description for the per-stage table and the real rc.
|
||||
- **No secret was printed.** Every `Preferences.xml` read in the runbook greps one named key, and the
|
||||
page states explicitly why: the file holds `PlexOnlineToken`. Nothing was `sops -d`'d.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
**Hardware transcoding has NOT been observed working.** Nothing in this branch is evidence that it
|
||||
does. A green gate here proves the manifest renders and the golden matches — it proves nothing about
|
||||
runtime behaviour, and `vars.CLUSTER_AVAILABLE` is unset so L2/L3 conformance did not run at all.
|
||||
|
||||
- **The device-cgroup question is OPEN, and it is the largest remaining technical risk.** Whether
|
||||
containerd on this k3s build permits a **non-privileged** pod to `open()` a bind-mounted character
|
||||
device is **UNVERIFIED**. It can only be settled by deploying and trying. The container inits as
|
||||
uid 0, which settles *file* permissions, but uid 0 does **not** bypass the cgroup device gate — and
|
||||
the two failures look identical from the outside. If it is refused, the fallback is the Intel
|
||||
Device Plugin (`gpu.intel.com/i915`), which is a `catalog/render.py` change plus a re-render with
|
||||
**no ServiceDefinition edit**; `privileged: true` is not available (Kyverno
|
||||
`restrict-host-namespaces` is Enforce). The runbook has the diagnostic sequence.
|
||||
- **The Plex-side toggle was still unset at the time of writing.** `HardwareAcceleratedCodecs` was
|
||||
**absent** from `Preferences.xml` — the toggle has never been enabled on this server. Until an
|
||||
operator ticks it in the UI, this branch changes the pod spec and **nothing about transcoding**.
|
||||
That is the whole reason the runbook exists.
|
||||
- **No metric, so no alert, so no automated detection — permanently.** `catalog/plex.yaml` declares
|
||||
no `metricsEndpoint`, so Plex renders no ServiceMonitor; there is no transcode-mode signal for a
|
||||
rule to evaluate. A silent degradation to software transcoding (a node losing its iGPU, the
|
||||
`DirectoryOrCreate` path) will **not** be reported by anything. I deliberately did not add an
|
||||
alert: with no metric behind it, it could never fire, and this repo has been burned at scale by
|
||||
exactly that. Closing the gap honestly means an exporter first.
|
||||
- **I did not re-derive this session's node or in-container measurements.** The three-node
|
||||
`renderD128`/`i915`/GID-992 facts, the cont-init hook behaviour, the `s6-setuidgid` group
|
||||
replacement, the `PLEX_PREFERENCE*` absence, the six `setPref` keys and the
|
||||
`HardwareAcceleratedCodecs` string were established earlier in this session and are cited, not
|
||||
re-measured, by me. Per CLAUDE.md §2b, treat this file as a record of belief and re-measure against
|
||||
the cluster before relying on any of it.
|
||||
- **The image is a floating tag.** Everything known about the hook came from digest
|
||||
`sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), while the Deployment runs
|
||||
`plexinc/pms-docker:latest` with `imagePullPolicy: Always`. A silent pull can invalidate the
|
||||
"no `supplementalGroups` needed" conclusion **with no commit in this repo**. Not guarded against —
|
||||
the runbook tells the reader to re-check the hook after an image bump, which is documentation, not
|
||||
enforcement.
|
||||
- **I did not verify the live Gitea gate state** (branch protection, required contexts, runner
|
||||
online) for this PR. Per §2b that must be queried, not read out of `CLAUDE.md`'s table; I did not
|
||||
query it.
|
||||
- **`/runbooks/plex-quicksync` was not fetched.** The L0 check asserts every static precondition for
|
||||
an HTTP 200; it does not make the request, and I did not either.
|
||||
@@ -1,204 +0,0 @@
|
||||
---
|
||||
date: 2026-09-30
|
||||
branch: feat/sd-hardware-capability
|
||||
title: ServiceDefinition gains a `hardware` capability field — `gpu: intel-quicksync` mounts /dev/dri and nothing else; the enum is what keeps the mechanism swappable
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
A new optional top-level ServiceDefinition property, `hardware`, with one optional member,
|
||||
`gpu`, whose enum today has exactly one value: `intel-quicksync`. **No catalog entry uses it.**
|
||||
This PR delivers the capability; consuming it (Plex) is separate work on top.
|
||||
|
||||
- `schemas/servicedefinition.schema.json` — adds `hardware` (object, `additionalProperties: false`,
|
||||
`minProperties: 1`) with `gpu` (enum `["intel-quicksync"]`). Schema version bumped **1.7.0 →
|
||||
1.8.0**, changelog entry prepended to the schema `description` per that file's own convention.
|
||||
- `catalog/render.py` — two module-level constants (`DRI_VOLUME_NAME`, `DRI_DEVICE_DIR`) and the
|
||||
emission in `r_deployment`.
|
||||
- `schemas/renderer-contract.md` — new `### hardware` field-mapping section, plus cross-field
|
||||
invariant **8** in that file's numbered list.
|
||||
- `schemas/README.md` — its `**Schema version:**` line said `1.0.0`; it had been stale since 1.1.0.
|
||||
Synced to 1.8.0 and annotated that the schema `description` is the changelog of record. Added one
|
||||
bullet to "Cross-field rules" for the new enum. That was the whole of the staleness; nothing
|
||||
ballooned.
|
||||
|
||||
**What the renderer emits** for `hardware.gpu: intel-quicksync`, on the app container only (never a
|
||||
sidecar), and it is exactly two things:
|
||||
|
||||
1. a hostPath volume `dri` → `/dev/dri`, `type: DirectoryOrCreate`;
|
||||
2. a read-write `volumeMount` of `dri` at `/dev/dri` on the app container (deliberately not
|
||||
`readOnly` — VA-API opens the render node read-write).
|
||||
|
||||
**Mounting the device is sufficient.** It emits **no** `privileged: true` (Kyverno
|
||||
`restrict-host-namespaces` is Enforce on app namespaces and it is not needed), **no** `nodeSelector`
|
||||
(placement is a separate concern with its own justification), **no** resource requests (there is no
|
||||
app-container resource render path), and — the one that is worth spelling out — **no
|
||||
`podSecurityContext.supplementalGroups`**.
|
||||
|
||||
**Why no `supplementalGroups`, when `/dev/dri/renderD128` is `root:render 0660`.** An earlier
|
||||
revision of this branch merged the host `render` GID into `supplementalGroups`. **It was measured to
|
||||
be a no-op and has been removed.** In the running container on prod-2, 2026-09-30:
|
||||
|
||||
PID 1 s6-svscan Uid: 0 Gid: 0 Groups: 0
|
||||
pid 246 Plex Media Server Uid: 1000 Gid: 1000 Groups: 100 1000
|
||||
|
||||
`/etc/services.d/plex/run` ends in `exec s6-setuidgid plex …`, and the child's group list is exactly
|
||||
the `/etc/group` account-database list — root's gid `0`, which PID 1 holds, has **vanished**. That is
|
||||
a `setgroups()` that *replaces* rather than appends, so anything Kubernetes puts in
|
||||
`supplementalGroups` reaches `s6-svscan` and is discarded before the app ever starts. What actually
|
||||
works is the image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner`, which `find`s
|
||||
every char device under `/dev/dri`, `stat -c '%g'`s its GID, `groupadd`s a matching group inside the
|
||||
container when none exists, and `usermod -a -G`s the app user into it — every start, nothing
|
||||
hardcoded; `cont-init.d` completes before `services.d`, so the database is correct at exec time.
|
||||
|
||||
Shipping an inert `supplementalGroups` would have been worse than shipping nothing: it would read as
|
||||
the mechanism to whoever debugs this next. The reasoning lives in a comment **at the emission site**
|
||||
and in `renderer-contract.md`, and `render.py --selftest` now asserts the *absence* of a
|
||||
`securityContext`, so it cannot be "helpfully" added back without a test failing. A pleasant
|
||||
corollary: because the hook re-`stat`s the device every start, host GID drift is **self-healing**,
|
||||
and no GID is pinned anywhere in this repo.
|
||||
|
||||
**The design decision worth defending: the field is an enum of intent, not a device path.** The SD
|
||||
says *what class of host device the workload needs*; the hostPath and the mount exist only in
|
||||
`catalog/render.py`. Migrating to the Intel GPU device plugin (`gpu.intel.com/i915` resource
|
||||
requests, no hostPath at all) is then a renderer change with zero catalog edits. Both the schema
|
||||
`description` and the contract section say this explicitly, because the obvious "simplification" —
|
||||
letting an SD name `/dev/dri` — would push a platform implementation detail into the operator-facing
|
||||
contract and make that migration a catalog-wide edit.
|
||||
|
||||
**`DirectoryOrCreate` is load-bearing.** A node that comes back without the render device (rebuild,
|
||||
BIOS change, driver regression, hardware swap) must degrade the workload to software transcoding,
|
||||
not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so a failed
|
||||
hostPath mount stalls the entire app tier for the sake of a feature that is not load-bearing.
|
||||
`/dev` is tmpfs, so the empty directory the kubelet creates on such a node leaves nothing durable
|
||||
behind. Do not "tighten" it to `Directory`. The reasoning is recorded at the emission site.
|
||||
|
||||
### Rebase note (2026-10-01, Ruth Okonkwo) — resolution only, no redesign
|
||||
|
||||
Rebased onto `main` @ `08deb20` (was cut from `0fbf544`). Re-landed as a single cherry-pick onto
|
||||
current `main`; the branch ref was force-updated with `--force-with-lease`, so PR #27 is the same
|
||||
PR with a new head.
|
||||
|
||||
**One conflict, in `catalog/render.py`, purely textual.** PR #28
|
||||
(`fix/oauth2-proxy-cookie-collision-fleet`, merged as `6caa5f2`) inserted its own constants blocks
|
||||
(`OAUTH2_ACCESS_TOKEN_LIFETIME_S` / `OAUTH2_COOKIE_REFRESH`, and `OAUTH2_PROXY_METRICS_PORT`) at the
|
||||
same insertion point immediately after `KEYCLOAK_ISSUER` that this branch used for
|
||||
`DRI_VOLUME_NAME` / `DRI_DEVICE_DIR`. The two blocks are unrelated and do not interact, so **both
|
||||
sides were kept in full**, oauth2-proxy's first and the hardware block after it. Nothing was
|
||||
dropped and nothing was rewritten: the rebased commit adds the **same 126 `render.py` lines** as the
|
||||
pre-rebase commit (verified by comparing the added-line sets) and deletes none. In particular the
|
||||
`supplementalGroups`-is-deliberately-not-emitted and `DirectoryOrCreate`-is-load-bearing comments
|
||||
survive verbatim.
|
||||
|
||||
Re-verified after the resolution: `--selftest` PASS (rc 0), and a full re-render still produces a
|
||||
**zero** diff in `apps/` — this PR remains inert, as it must be while no ServiceDefinition opts in.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured, in this worktree, on this branch:
|
||||
|
||||
- **`python3 catalog/render.py --selftest` → `render.py selftest: PASS`, rc 0.** New cases in the
|
||||
existing `check(...)` style: the positive (volume + mount, and *no* `securityContext`, *no*
|
||||
privileged, *no* nodeSelector), the negative that matters (an SD without `hardware` gains no
|
||||
volume, no volumeMount and no `securityContext` — mirroring the existing `podSecurityContext`
|
||||
negative), and a pass-through case (an SD declaring both `hardware` and its own
|
||||
`podSecurityContext: {fsGroup: 1001, supplementalGroups: [44]}` keeps it **exactly** as written).
|
||||
- **The self-tests actually bite.** I mutated the guard (`if hw_gpu == "intel-quicksync":` → `if
|
||||
True:`) in a scratch copy and re-ran: **rc 1**, failing on "an SD without `hardware` must gain no
|
||||
dri volume, no mount and no securityContext". A test that cannot fail is not a test; this one can.
|
||||
- **Re-render is a zero diff.**
|
||||
`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` then
|
||||
`git diff --stat apps/` → **empty**, and `git status --porcelain apps/` → **empty** (so the two
|
||||
not-onboarded by-products, `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, were
|
||||
rewritten byte-identically and needed no revert). The change is inert until a service opts in.
|
||||
- **Schema validity and catalog conformance.** `Draft202012Validator.check_schema` passes; all
|
||||
**12** `catalog/*.yaml` validate with **0** errors. On the new field specifically: `gpu:
|
||||
intel-quicksync` accepted; `gpu: /dev/dri` rejected (enum); `hardware: {gpu: …, path: …}` rejected
|
||||
(`additionalProperties: false`); `hardware: {}` rejected (`minProperties: 1`).
|
||||
- **Hand-checked the emitted YAML** by rendering a throwaway SD carrying `hardware`, a Longhorn
|
||||
volume and its own `podSecurityContext: {runAsUser: 1000, supplementalGroups: [44]}`. Result:
|
||||
`volumeMounts` = config then `{name: dri, mountPath: /dev/dri}`; `volumes` = the PVC then
|
||||
`{name: dri, hostPath: {path: /dev/dri, type: DirectoryOrCreate}}`; `securityContext` =
|
||||
`{runAsUser: 1000, supplementalGroups: [44]}` — i.e. **unchanged**, nothing appended. That toy SD
|
||||
was never written into `catalog/`.
|
||||
- **GATE: `rc 3` — UNDETERMINED, NOT green.** Run in the pinned toolbox from this worktree with
|
||||
both CLAUDE.md §3 workarounds (`DEV_KUBECONFIG=/dev/null`, and an `ENGINE` shim bind-mounting the
|
||||
main checkout's `.git` at its own absolute path). Diff base `origin/main @ 0fbf544d5e2f`; touched
|
||||
requirements `R-DNS-03 R-GITOPS-02 R-IDP-02`. Run **four** times on this branch with an identical
|
||||
result each time, most recently against tree `c132893045b1` — the final commit differs from that
|
||||
run only by this paragraph. Per stage:
|
||||
|
||||
| stage | result |
|
||||
|---|---|
|
||||
| spec-immutability | **SKIPPED** — sibling spec repo not at `/work/../home-network-k3s`. Expected from a worktree; tolerated by §3. |
|
||||
| decisions-log | pass |
|
||||
| lint | pass |
|
||||
| l0 | pass — 19 offline cases, `0 failure(s), 0 skip(s), 0 unwired` |
|
||||
| render-golden | pass — *"all 10 onboarded service golden(s) reproduce from the canonical render"* |
|
||||
| l1 | pass — 3 offline checks, `0 failure(s), 0 skip(s), 0 unwired`; coverage 51/51 requirements scaffolded |
|
||||
| approval-policy | pass (`event='none'`, not a pull_request event — N/A locally; it will evaluate for real on the PR) |
|
||||
| gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset. Expected; branch protection and runner registration are therefore UNVERIFIED by this run. |
|
||||
| conformance (L2/L3) | **SKIPPED** — `CLUSTER_AVAILABLE!=true`. Phase-0 standing skip. |
|
||||
|
||||
The gate's own verdict: *"Stages that did not run: spec-immutability gitea-settings"*. **No stage
|
||||
failed, and no stage skipped for a reason caused by this change.** rc 3 is reported as rc 3; I am
|
||||
not calling this green.
|
||||
- **Blast radius of the schema change:** `hardware` is absent from every catalog entry, so
|
||||
`apply_defaults` never materializes it and `s.get("hardware")` is `None` on all 12 — which is what
|
||||
the zero diff above demonstrates rather than asserts.
|
||||
- **Touched no sensitive path.** Changes are confined to `schemas/`, `catalog/`, `decisions/`; per
|
||||
`infra/gitea/approval-policy.json` none of those are sensitive. `conformance/**` and `clusters/**`
|
||||
are untouched. I did not run `--emit-waves` (it writes `clusters/base/`, and the set of files
|
||||
under `apps/` did not change).
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Whether containerd's device cgroup permits a non-privileged pod to `open()` a bind-mounted char
|
||||
device. Still unverified, and it is the single largest technical risk in this workstream.**
|
||||
containerd installs a per-container **device cgroup** allow-list, and whether
|
||||
`/dev/dri/renderD128` arriving as a bind-mounted device node is permitted by it on this
|
||||
k3s/containerd version is **unmeasured**. Settling it requires applying a pod, which is an
|
||||
operator action, not an agent one. The container's PID 1 running as uid 0 helps on the DAC side
|
||||
but does **not** bypass the cgroup gate. If it blocks, the mount will succeed, the image's group
|
||||
hook will do its job, and the `open()` will still fail — and the fallback is the Intel GPU device
|
||||
plugin, which *sets* the cgroup entry. Note that that fallback is a **renderer-only change**: no
|
||||
ServiceDefinition moves, because the authoring surface is an intent enum. That is the design
|
||||
justifying itself, and it is the reason to keep the enum even if the hostPath mechanism loses.
|
||||
- **That a missing device degrades silently — and nothing will tell you.** This is the accepted cost
|
||||
of `DirectoryOrCreate`. The workload will software-transcode and **no signal reports it**: media
|
||||
apps in this catalog declare no `metricsEndpoint`, so no ServiceMonitor exists, and there is no
|
||||
metric to alert on. I deliberately did **not** propose an alert, because an alert that cannot fire
|
||||
is worse than none — see the 2026-09-13 correction in `renderer-contract.md` §`observability`,
|
||||
where twelve monitors discovered zero targets. The only proof of hardware transcode is an operator
|
||||
seeing it in the application's own dashboard.
|
||||
- **That "mounting the device is sufficient" generalizes beyond this one image.** It does not, and
|
||||
I have not verified that it does. The claim was measured for an image that resolves the device
|
||||
group itself in `cont-init.d`. A future consumer of `hardware.gpu` that runs non-root with no
|
||||
equivalent hook would still need the group solved — and the fix for that case is a **renderer**
|
||||
change (per-image handling, or the device plugin), never a new SD field. I scoped the claim in
|
||||
`renderer-contract.md` rather than letting it read as general; the first such consumer must
|
||||
re-measure.
|
||||
- **The feature now depends on a hook inside an image that floats.** The `cont-init.d` behaviour
|
||||
above was read from digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), but the
|
||||
deployment runs `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. **A silent pull can
|
||||
change the behaviour this feature depends on, with no commit anywhere in this repo to show for
|
||||
it** — and the failure would be the silent software-transcode described above. I have not fixed
|
||||
that here (it is the Plex wiring's tag/digest decision, not the capability's), but it should not
|
||||
be forgotten.
|
||||
- **GID drift is NOT on this list any more, deliberately.** An earlier revision pinned
|
||||
`INTEL_RENDER_GID = 992` and listed "a node rebuild could move it" as an open risk. The hook
|
||||
re-`stat`s the device on every start, so drift is self-healing and no GID is pinned anywhere. The
|
||||
risk is retired, not overlooked.
|
||||
- **Anything on the live cluster.** I ran no `kubectl` and I opened no shell in any container. The
|
||||
node facts (device present, `root:render` 0660, `i915` bound, no device plugin, no
|
||||
`gpu.intel.com/*` advertised), the in-container measurements quoted above, and the Kyverno
|
||||
posture (`restrict-host-namespaces` Enforce; no hostPath or `supplementalGroups`
|
||||
restriction; no PSA labels on app namespaces) are **prior recon handed to me, not re-measured by
|
||||
me**. Per CLAUDE.md §2b, treat them as second-hand until re-queried.
|
||||
- **That CI exercises the self-tests. It does not.** `git grep -- --selftest ci/` returns nothing —
|
||||
no gate stage invokes the renderer's self-test. They are the executable spec and house style, and
|
||||
I ran them by hand, but a future regression in this code path **will not be caught by the gate**.
|
||||
Wiring `--selftest` into a gate stage is a real gap and is not part of this PR (`ci/` is a
|
||||
sensitive path and would drag in a human reviewer for a change unrelated to this one).
|
||||
- **Runtime behaviour of any kind.** The gate proves the L0/L1 offline subset only;
|
||||
`vars.CLUSTER_AVAILABLE` is unset, so L2/L3 conformance does not run in CI.
|
||||
@@ -0,0 +1,205 @@
|
||||
---
|
||||
date: 2026-10-01
|
||||
branch: fix/render-selftest-runs-in-gate
|
||||
title: catalog/render.py's 106 assertions were dev-time only; the gate now runs them and blocks on them
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Single subject, one file: **`ci/render-golden.sh` now invokes `catalog/render.py --selftest`**
|
||||
and exits 1 when it fails. Nothing else. No renderer change, no new conformance test, no new
|
||||
gate stage.
|
||||
|
||||
This closes the top follow-up of `decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md`:
|
||||
|
||||
> Nothing in `ci/` or `.gitea/` runs `catalog/render.py --selftest` — I grepped for it. So the
|
||||
> twenty-odd assertions in that selftest, including the six I just added, are dev-time only: a
|
||||
> future edit that drops either flag *and* re-renders the goldens would pass the entire merge
|
||||
> gate.
|
||||
|
||||
Re-measured here and confirmed: before this commit, `grep -rl -- --selftest` over `ci/` and
|
||||
`.gitea/` returned **nothing** for the renderer. The selftest executes **106** assertions
|
||||
(`check()` invocations counted at runtime via `sys.settrace`, without editing `render.py`; 109
|
||||
static call sites, three of them on failure-only branches). All 106 were unexercised by the
|
||||
merge gate.
|
||||
|
||||
### The lead was WRONG, and here is why
|
||||
|
||||
The brief named `conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` as the
|
||||
one-line fix. That line is real and it does drive selftests:
|
||||
|
||||
for tool in ("record_run.py", "status.py"):
|
||||
r = run([sys.executable, str(LIB / tool), "--selftest"])
|
||||
|
||||
But it is **the wrong hook**. That loop is section (d) of a check that serves **R-SEC-03**, and
|
||||
its subject is the *evidence chain* — can a self-skipped run, or a stale ledger row, become an
|
||||
audit proof? `record_run.py` and `status.py` are **in** that chain; `catalog/render.py` is not.
|
||||
Hanging the renderer there would have made the check's name, its `intent` annotation and its
|
||||
requirement mapping (`R-SEC-03`) all untrue, and a future decision about the renderer could then
|
||||
only be expressed by weakening a *security-auditability* check. That is the precise coupling the
|
||||
cookie-collision entry refused for its own check, and it applies here.
|
||||
|
||||
The second option that entry offered — a new scope-declaring L0 test — is the supported mechanism
|
||||
for a **whole-repo invariant** (`ci/l0.sh` Stage 3b's `--include-scope whole-repo` union, ratcheted
|
||||
by `tests/l0-static/whole-repo-invariants-wired`). It is the right shape for an invariant
|
||||
quantified over the tracked tree. It is the wrong shape for *"a script in this repo must pass its
|
||||
own unit test"*: it would need to declare requirements it does not serve, and it would be two new
|
||||
files of several hundred lines by this tree's conventions for what is a six-line invocation.
|
||||
|
||||
**The honest home is `ci/render-golden.sh`** — the gate stage whose subject *already is*
|
||||
`catalog/render.py`. It already resolves `$PYTHON` and `$RENDER_PY`, already reports rc 3 when
|
||||
either is missing, already exits 1 on renderer error, and `ci/gate.sh` already maps it through
|
||||
`for s in lint l0 render-golden l1` where a FAIL is BLOCKED and a SKIP is UNDETERMINED. The
|
||||
stage's two assertions are independent and both are needed:
|
||||
|
||||
- `render-golden` catches golden-vs-renderer drift, but is satisfied by **any** renderer whose
|
||||
output was re-rendered — including one that just lost a contract invariant and had all ten
|
||||
goldens regenerated in the same commit. That is exactly the hole named in the quote above.
|
||||
- `--selftest` catches that, and is blind to golden drift.
|
||||
|
||||
The file header and the stage name were updated so neither is a lie: the stage is now "the
|
||||
renderer is trustworthy", with two numbered assertions, and the step summary emits its own
|
||||
`- render-selftest:` line so the gate output still NAMES what ran.
|
||||
|
||||
The selftest runs **before** the empty-catalog `exit 3` on purpose: it renders synthetic
|
||||
ServiceDefinitions in memory and is catalog-independent, so an empty catalog must not be able to
|
||||
skip it.
|
||||
|
||||
`catalog/render.py` itself was **not touched** (owned by a concurrent branch this session), and
|
||||
neither was any file under `conformance/`.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **The selftest runs and passes: rc 0, `render.py selftest: PASS`, 106 assertions executed.**
|
||||
Counted at runtime, not from the source: a `sys.settrace` hook counting `check()` calls, in a
|
||||
throwaway script outside the repo, with no edit to `render.py`.
|
||||
- **The stage passes with it wired in**, host run of the single stage:
|
||||
|
||||
[gate] Stage: render==golden reproducibility (canonical render vs committed apps/)
|
||||
[gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
|
||||
[ ok ] render.py --selftest passed
|
||||
[ ok ] render==golden: all 10 onboarded service golden(s) reproduce from the canonical render
|
||||
|
||||
rc **0**. The golden comparison still reports all 10 goldens, so the new invocation did not
|
||||
displace what the stage already asserted.
|
||||
- **IT CAN FAIL — provoked twice, against a mutated COPY of the renderer, never against the
|
||||
tree.** `ci/render-golden.sh` already honours `RENDER_PY`, so each mutant lived in
|
||||
`/tmp/claude-1000/desmond-selftest/` and the repo's `catalog/render.py` was never modified —
|
||||
which matters, because that file belongs to a concurrent branch this session.
|
||||
- **Mutation 1 — the regression this selftest was written for.** Deleted the
|
||||
`f"--cookie-name=_oauth2_proxy_{s['name']}"` emission from `_oauth2_sidecar_container()`,
|
||||
i.e. reverted the fleet-wide cookie-collision fix of PR #28. Stage rc **1**. Note the shape
|
||||
of the failure, because it is a real (minor) weakness and is listed as a follow-up: the
|
||||
selftest died with an uncaught `StopIteration` out of `next(a for a in container["args"] if
|
||||
a.startswith("--cookie-name="))` at `render.py:2641`, rather than printing its collected
|
||||
`SELFTEST FAIL:` lines. It is correctly non-zero and the stage correctly blocks, but the
|
||||
operator gets a traceback instead of the assertion message.
|
||||
- **Mutation 2 — a clean, named assertion failure.** `OAUTH2_COOKIE_REFRESH = "2m30s"` ->
|
||||
`"10m"`, widening the refresh past the measured 300s access-token lifetime. Stage rc **1**:
|
||||
|
||||
[fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
|
||||
[fail] Every golden under apps/ is produced by this renderer, so the goldens below cannot
|
||||
[fail] be trusted even if they reproduce byte-for-byte. Reproduce with:
|
||||
[fail] python3 catalog/render.py --selftest
|
||||
SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). A
|
||||
refresh at or past the access-token lifetime never renews in time, which is the same
|
||||
logout it was added to stop
|
||||
|
||||
This is the mutation that proves the wiring's value most directly: mutation 2 would **not**
|
||||
be caught by `render-golden`'s diff if the goldens were regenerated in the same commit, and
|
||||
it is one of the two flags the cookie-collision entry explicitly recorded as **ungated**.
|
||||
- **THE NON-ZERO PROPAGATES ALL THE WAY TO `gate.sh`'s rc — provoked, in the pinned toolbox, not
|
||||
reasoned about.** `RENDER_PY=/desmond-selftest/render-mutant2.py ./ci/gate.sh`:
|
||||
|
||||
[ ok ] stage spec-immutability: pass
|
||||
[ ok ] stage decisions-log: pass
|
||||
[ ok ] stage lint: pass
|
||||
[ ok ] stage l0: pass
|
||||
[gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
|
||||
[fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
|
||||
SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). ...
|
||||
[fail] stage render-golden: FAIL
|
||||
[ ok ] stage l1: pass
|
||||
...
|
||||
[fail] render-golden: FAILED -> blocks merge
|
||||
[fail] MERGE GATE: BLOCKED
|
||||
|
||||
`gate.sh` rc **1**. Three links in the chain, each measured: selftest rc 1 -> stage rc 1 ->
|
||||
gate rc 1 / `MERGE GATE: BLOCKED`. This mattered enough to provoke rather than assume: this
|
||||
repo has a documented habit of exit codes getting flattened, and
|
||||
`.gitea/workflows/gate.yml:138` runs `./ci/gate.sh || [ "$?" -eq 3 ]`, which tolerates rc 3 —
|
||||
so a stage that degraded a failure to a skip would be swallowed by CI. rc 1 is not tolerated
|
||||
there, and rc 1 is what this produces.
|
||||
- **`spec-immutability` genuinely RAN and passed, it did not skip.** The two-mount worktree shim
|
||||
plus `-e SPEC_REPO=/home-network-k3s` (the fix recorded in the 2026-09-30 cookie entry) works:
|
||||
`stage spec-immutability: pass`, not SKIPPED. R-TEST-06 is therefore actually covered by these
|
||||
runs.
|
||||
- **No collision with PR #29.** The brief warned that `feat/oauth2-proxy-auth-failure-visible`
|
||||
had added checks under `conformance/tests/l0-static/`. Measured —
|
||||
`git diff origin/main...origin/feat/oauth2-proxy-auth-failure-visible --name-only` — it touches
|
||||
**no `conformance/` path at all**: `decisions/`, `platform/oauth2-proxy/` (a promtool test, an
|
||||
alert rule, its kustomization), `runbooks/`. Disjoint from this branch either way, since this
|
||||
branch touches only `ci/render-golden.sh` and this file.
|
||||
- **Priya's files were not touched.** `catalog/render.py`, `schemas/servicedefinition.schema.json`,
|
||||
`apps/wave-*` and `conformance/tests/l1-rendering/` are all untouched by this branch — confirmed
|
||||
by `git diff --name-only`, which lists exactly two paths.
|
||||
- **`catalog/render.py --selftest` was the LAST unwired selftest in the tree.** Measured:
|
||||
`grep -rl -- --selftest` across the repo finds selftest-capable artifacts in exactly four
|
||||
places, and after this commit every one is driven by the gate — `conformance/lib/record_run.py`
|
||||
and `conformance/lib/status.py` by `auditability-proofs-producible/check.sh:291`, the Keycloak
|
||||
client-secret reconciler (embedded in `platform/keycloak/client-secret-reconciler.yaml`) by
|
||||
`keycloak-client-secret-reconciler/check.sh:633`, and `catalog/render.py` by this change.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Local gate: rc 3 (UNDETERMINED), NOT 0. This is not green and must not be reported as
|
||||
green.** Measured in the pinned toolbox on `eeb6211`:
|
||||
|
||||
[ ok ] spec-immutability: green
|
||||
[ ok ] approval-policy: green
|
||||
[ ok ] decisions-log: green
|
||||
[ ok ] lint: green
|
||||
[ ok ] l0: green
|
||||
[ ok ] render-golden: green
|
||||
[ ok ] l1: green
|
||||
[fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
|
||||
[fail] Stages that did not run: gitea-settings
|
||||
|
||||
Nothing FAILED. The single cause is `gitea-settings` **SKIPPED** — `SETTINGS_API_TOKEN` is not
|
||||
set in my environment, so branch protection and runner registration were not read; CI has that
|
||||
token and the stage is expected to run there. **L2/L3 `conformance`** also skipped
|
||||
(`CLUSTER_AVAILABLE` unset) but is a *declared* gap that `gate.sh` treats as not-applicable
|
||||
rather than undetermined, so it is not what produced the 3. Touched requirements
|
||||
**R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06** are therefore **UNVERIFIED at L2/L3**, all four.
|
||||
The stage this branch changes, `render-golden`, is green **and** was independently proven able
|
||||
to turn the whole gate red (above).
|
||||
- **`approval-policy` passed LOCALLY only because there is no pull_request event**
|
||||
(`not a pull_request event (event='none') — approval policy N/A`). That is not evidence that it
|
||||
passes. This diff touches `ci/**`, a sensitive path, so in CI the stage is expected to **FAIL**
|
||||
until a non-author approving review exists. That is §1 working, fail-closed, not a defect.
|
||||
- **The selftest's own quality is unchanged and ungated.** I wired its *invocation*; I did not
|
||||
review its 106 assertions, and nothing ratchets them. Deleting the body of `_selftest()` would
|
||||
still pass this stage, because an empty selftest returns 0. The repo already has the right
|
||||
answer for that — `conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py`
|
||||
injects regressions one at a time and requires `--selftest` to catch every one — and applying
|
||||
that pattern to `catalog/render.py` is the follow-up, deliberately not done here: it is a
|
||||
second subject, it touches `conformance/`, and it would need to live beside the renderer whose
|
||||
file another branch owns this session. **Until it exists, "the selftest runs" is proven and
|
||||
"the selftest is falsifiable" rests on my two hand-run mutations, which are not in the gate.**
|
||||
- **Mutation 1's traceback is not fixed.** `render.py:2641`'s `next(...)` has no default, so the
|
||||
absence of `--cookie-name` raises `StopIteration` and the selftest's collected `SELFTEST FAIL:`
|
||||
lines are never printed. Correct verdict, poor diagnostics, and it suppresses the other
|
||||
assertions that mutation also broke. The one-line fix (`next(..., None)` plus a `check()`) is in
|
||||
`catalog/render.py`, which is another branch's file this session, so it is a follow-up and not
|
||||
a thing I touched.
|
||||
- **Nothing about the live cluster.** No `kubectl` of any kind was run on this branch — not even
|
||||
read-only; this change is offline CI wiring and needed none. No `apply`/`edit`/`patch`, no
|
||||
`flux reconcile`. Merging this deploys nothing: it changes only what the gate refuses.
|
||||
- **CI's own run of this branch.** Quoted in the PR body from the commit's `/statuses`, with
|
||||
descriptions read rather than colours. A *skipped* Gitea job reports `success` with
|
||||
"Has been skipped" and certifies nothing.
|
||||
- **`vars.CLUSTER_AVAILABLE` is still unset in CI**, so even a fully green PR proves the L0/L1
|
||||
offline subset and nothing about runtime behaviour.
|
||||
- **I did not re-measure branch protection, required approvals, or the runner.** Those are
|
||||
CLAUDE.md §2 claims and §2b says re-measure before relying on them; this branch relies on none
|
||||
of them beyond the PR flow itself, and I read the gate's own output rather than the API.
|
||||
@@ -115,14 +115,6 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
|
||||
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
|
||||
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
|
||||
- **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod
|
||||
(`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware
|
||||
acceleration when available"**, and no environment variable can do it (the image supports no
|
||||
`PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`,
|
||||
read out of the binary), says which widely-cited key **does not exist**, and is explicit that the
|
||||
**only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no
|
||||
alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin
|
||||
fallback, and why a `:latest` pull can silently break it.
|
||||
- **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
|
||||
re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
|
||||
(the four lines that matter, including the upstream `-` that means the proxy answered a request the
|
||||
|
||||
@@ -95,12 +95,6 @@ configMapGenerator:
|
||||
# because the plex.tv claim window is short and is not a thing to go looking
|
||||
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
|
||||
- plex-claim=plex-claim.md
|
||||
# Operator procedure, not an alert page, and NO alert is possible for it: plex renders no
|
||||
# metricsEndpoint, so there is no ServiceMonitor, no transcode-mode metric and nothing a rule
|
||||
# could evaluate. Served because the feature it completes is INERT until a human ticks a box in
|
||||
# the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped
|
||||
# feature stays switched off for months.
|
||||
- plex-quicksync=plex-quicksync.md
|
||||
- pvc-capacity=pvc-capacity.md
|
||||
- targets-down=targets-down.md
|
||||
- update-rollback=update-rollback.md
|
||||
|
||||
@@ -1,283 +0,0 @@
|
||||
# plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did
|
||||
|
||||
**Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**,
|
||||
not an incident page. It is indexed here because the feature it completes is **inert until a human
|
||||
clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
|
||||
to prevent.
|
||||
**Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the
|
||||
rendered golden — the `dri` hostPath volume and the `/dev/dri` mount), `catalog/render.py` (the
|
||||
mechanism), `schemas/renderer-contract.md` `### hardware` (the contract).
|
||||
**Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile.
|
||||
|
||||
## What this means here
|
||||
|
||||
Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync
|
||||
moves that work off the CPU — which matters on a three-node cluster where the CPU is also running
|
||||
everything else. Two independent things have to be true before that happens, and **only one of them
|
||||
is in git**:
|
||||
|
||||
| Half | Who does it | State |
|
||||
|---|---|---|
|
||||
| The render device is inside the container at `/dev/dri` | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders a `dri` hostPath volume (`/dev/dri`, `type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container. Flux applies it. | Done by merging that change. |
|
||||
| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
|
||||
|
||||
**There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
|
||||
grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`,
|
||||
`/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin` — found **zero** `PLEX_PREFERENCE*` support.
|
||||
Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentifier`,
|
||||
`ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`,
|
||||
`TranscoderTempDirectory`), none of them the transcode toggle. No environment variable, no ConfigMap
|
||||
and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this;
|
||||
there isn't one, and the search has already been done.
|
||||
|
||||
What the device mount does **not** do, so nobody reads more into a green gate than is there: it does
|
||||
not enable hardware transcoding, it does not prove the device can be *opened* from inside a
|
||||
non-privileged pod (see "Likely causes" #2), and it renders no `nodeSelector` — Plex can land on any
|
||||
of the three nodes and all three carry the device.
|
||||
|
||||
## Running the commands in this runbook
|
||||
|
||||
`kubectl` does not work on the operator workstation; the toolchain lives only in the pinned dev
|
||||
container. Every command below runs as:
|
||||
|
||||
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||
|
||||
Always pass `--context k3s-prod` explicitly. `dev.sh` merges both clusters into one kubeconfig and
|
||||
picks a current-context **per invocation**, and that default has already changed once. Do not run two
|
||||
`dev.sh` invocations at once.
|
||||
|
||||
**Never `cat` `Preferences.xml`.** It contains `PlexOnlineToken`, a plex.tv account credential. Every
|
||||
read of that file below greps a single named key — do not widen one.
|
||||
|
||||
## First commands
|
||||
|
||||
**1. Is the device mounted into the pod at all?** (Proves Flux applied the change, not that it works.)
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
|
||||
-o jsonpath="{.items[0].spec.containers[0].volumeMounts[*].mountPath}"; echo'
|
||||
```
|
||||
|
||||
`/dev/dri` in that list ⇒ the GitOps half landed. Absent ⇒ the running pod predates the merge; compare
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 \
|
||||
-o jsonpath="{.status.lastAppliedRevision}"; echo'
|
||||
```
|
||||
|
||||
against the merge commit before blaming anything else.
|
||||
|
||||
**2. Is the device actually visible inside the container?**
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'
|
||||
```
|
||||
|
||||
Expect `renderD128` (usually `card0` too), mode `crw-rw----`, group `render`. An **empty directory**
|
||||
here is the `DirectoryOrCreate` degradation: the kubelet created an empty `/dev/dri` because the node
|
||||
had none. That is the designed-safe outcome — the app tier still reconciles — but it means software
|
||||
transcoding, silently. Find which node the pod is on and ask why that node lost its iGPU:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
|
||||
```
|
||||
|
||||
**3. Can Plex *open* it?** This is the question the manifest cannot answer — see "Likely causes" #2.
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- \
|
||||
bash -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
|
||||
```
|
||||
|
||||
`RW-OPEN-DENIED` (or `Operation not permitted` / `Permission denied`) with the device plainly visible
|
||||
in step 2 is the **device-cgroup** failure, not a file-permission failure. The two look identical from
|
||||
the outside and are not the same problem.
|
||||
|
||||
**4. Is the toggle on?** Named key only — never the whole file:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- \
|
||||
grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
|
||||
"/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'
|
||||
```
|
||||
|
||||
`ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on,
|
||||
and the remaining question is whether it *works* — go to "How to verify".
|
||||
|
||||
## The one manual step
|
||||
|
||||
Plex web UI → **Settings → Transcoder → "Use hardware acceleration when available"** → tick → Save.
|
||||
(It sits under *Show Advanced*.) A transcode already in flight is unaffected; the next one picks it up.
|
||||
|
||||
The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by
|
||||
extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from
|
||||
recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise
|
||||
them and not invent them:
|
||||
|
||||
| Key | What it is |
|
||||
|---|---|
|
||||
| `HardwareAcceleratedCodecs` | the toggle this page is about |
|
||||
| `HardwareAcceleratedEncoders` | hardware **encode**, as distinct from decode |
|
||||
| `HardwareDevicePath` | how Plex is pointed at a *specific* render node — relevant only if a host ever exposes more than one |
|
||||
| `TranscoderToneMapping`, `TranscoderToneMappingAgorithm` | HDR→SDR tone mapping. **That misspelling of "Algorithm" is upstream's, in the binary. It is not a typo in this page — do not "fix" it** here or anywhere else; the corrected spelling is not a key Plex reads. |
|
||||
| `TranscoderHEVCEncoding` | HEVC output |
|
||||
|
||||
**`HardwareAcceleratedCodecsEnableHEVC` does not exist.** It is widely cited and it is not in the
|
||||
binary. Recorded explicitly so nobody spends an evening hunting for it or, worse, writes it into
|
||||
`Preferences.xml` and concludes hardware transcoding is broken.
|
||||
|
||||
Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn
|
||||
tier, it is in no git commit, and the UI writes the key correctly. Use the UI.
|
||||
|
||||
## How to verify — and why there is no alert
|
||||
|
||||
Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a
|
||||
lower quality explicitly), then read Plex's own **Dashboard → Activity**:
|
||||
|
||||
- **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.**
|
||||
- plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says.
|
||||
|
||||
**That dashboard is the *only* proof available, and nothing else will ever tell you.**
|
||||
`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
|
||||
there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work
|
||||
to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently
|
||||
inert, and this repo has already been burned at scale by rules that evaluated against nothing while
|
||||
looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that
|
||||
gap is ever worth closing, the honest order is a real exporter first and a rule second.
|
||||
|
||||
Corroborating, not a substitute — the server log on the config PVC:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- \
|
||||
sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
|
||||
\"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'
|
||||
```
|
||||
|
||||
Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions.
|
||||
Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence
|
||||
of hardware. The dashboard remains the verdict.
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
When the dashboard still says plain `Transcode`:
|
||||
|
||||
1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists.
|
||||
First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
|
||||
investigating anything technical.
|
||||
|
||||
2. **containerd's device cgroup is refusing the `open()` — UNVERIFIED, and the largest remaining
|
||||
technical risk in this feature.** Whether a non-privileged pod may `open()` a bind-mounted
|
||||
character device on this k3s/containerd build **has not been established**; it can only be settled
|
||||
by deploying and trying. The container inits as uid 0, which settles *file* permissions, but
|
||||
**uid 0 does not bypass the cgroup device gate.** Symptom: step 2 shows `renderD128` present and
|
||||
correctly owned, step 3 returns `RW-OPEN-DENIED`.
|
||||
**The documented fallback is the Intel Device Plugin for Kubernetes** (`gpu.intel.com/i915` as a
|
||||
resource request, no hostPath). Because the ServiceDefinition field expresses *intent* rather than
|
||||
a device path, that migration is a **`catalog/render.py` change plus a re-render — with no
|
||||
ServiceDefinition edit anywhere**; `catalog/plex.yaml` keeps saying exactly what it says today.
|
||||
**`privileged: true` is not an available escape hatch:** Kyverno `restrict-host-namespaces` is in
|
||||
**Enforce** mode and rejects it in app namespaces. Do not reach for it.
|
||||
|
||||
3. **The pod landed on a node without the device.** Step 2 shows an empty `/dev/dri`. All three prod
|
||||
nodes carried `renderD128` with `i915` bound when this was measured (2026-09-30), so this means a
|
||||
node changed — a rebuild, a BIOS reset, or a driver regression. `DirectoryOrCreate` is why the app
|
||||
tier still came up; the cost is that the loss is silent. **Fix the node, do not tighten the mount:**
|
||||
`Directory` would stall `apps-wave-0` (`wait: true`, with `apps-wave-1` depending on it) for a
|
||||
feature that is not load-bearing.
|
||||
|
||||
4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
|
||||
mechanism this feature leans on with no commit in this repo.
|
||||
|
||||
5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so
|
||||
`Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as
|
||||
a hardware failure.
|
||||
|
||||
6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which
|
||||
Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
|
||||
sessions show `(hw)` and others do not. Not a fault.
|
||||
|
||||
## The floating-image caveat
|
||||
|
||||
Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
|
||||
user to the device's group, the absence of `PLEX_PREFERENCE*`, the six `setPref` keys, the
|
||||
`HardwareAcceleratedCodecs` string — was read from digest **`sha256:e0ab27395614…`**
|
||||
(`plexmediaserver 1.43.4.10903`).
|
||||
|
||||
The deployment runs **`plexinc/pms-docker:latest` with `imagePullPolicy: Always`**. A silent pull can
|
||||
therefore change the behaviour this feature depends on **with no commit in this repo**, while the
|
||||
manifest still looks identical. Symptom to watch for: hardware transcoding that worked and quietly
|
||||
stopped, with no git change to point at.
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
|
||||
-o jsonpath="{.items[0].status.containerStatuses[0].imageID}"; echo'
|
||||
```
|
||||
|
||||
If that digest is not the one above, **re-verify the hook before trusting this page**:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- ls /etc/cont-init.d/'
|
||||
```
|
||||
|
||||
`45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one
|
||||
described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups`
|
||||
needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that
|
||||
hook exists.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
- **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be
|
||||
**inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's
|
||||
`cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its
|
||||
group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces**
|
||||
the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with
|
||||
`Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
|
||||
discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
|
||||
- **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
|
||||
reject it. If the cgroup gate is the blocker, the answer is the Intel device plugin (cause #2).
|
||||
- **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles
|
||||
`main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating
|
||||
`kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change
|
||||
through a branch, a PR and a green gate.
|
||||
- **Do not edit `Preferences.xml` on the PVC**, never `cat` it, and never `sops -d` anything into a
|
||||
terminal. It holds `PlexOnlineToken`.
|
||||
- **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and
|
||||
watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
|
||||
lose the DB.
|
||||
- **Stop and get the operator** if the device cannot be opened from the pod (cause #2 confirmed). The
|
||||
next step is a platform change — the Intel device plugin — with its own PR, not more poking at this
|
||||
workload.
|
||||
|
||||
## Known gaps, deliberately not fixed here
|
||||
|
||||
Recorded so they are findable, and so nobody assumes they were overlooked:
|
||||
|
||||
- **Transcode scratch is on the wrong medium, and it is invisible.** The image sets
|
||||
`TranscoderTempDirectory=/transcode`, which is **not a volume** — it is the container's writable
|
||||
overlay, i.e. the node's `/data`. That is NVMe on prod-2 but **SATA on prod-1 and prod-3**, so
|
||||
transcode scratch performance changes silently whenever the pod is rescheduled. Both candidate fixes
|
||||
need a **schema** change and are therefore out of scope here: pin Plex to prod-2 (needs a
|
||||
`nodeSelector` field the schema does not have), or give `/transcode` a `medium: Memory` emptyDir and
|
||||
transcode in RAM (~31 GiB per node; needs an emptyDir field the schema does not have). The RAM option
|
||||
is the better one, because it removes the node asymmetry outright instead of working around it.
|
||||
- **GuC/HuC firmware is not loaded** (`i915.enable_guc=0`). The cost is low-power H.264 **encode**
|
||||
quality only — decode, and the feature on this page, are unaffected. Changing it needs a kernel
|
||||
parameter and a **serialized rolling reboot**, which is an operator action (CLAUDE.md §5), and this
|
||||
cluster has a documented history of that going badly (a prod node hung on a warm reboot on
|
||||
2026-09-03). Not worth coupling to this change.
|
||||
|
||||
---
|
||||
Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
|
||||
The node measurements (all three prod nodes) and the in-container image measurements cited here were
|
||||
taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
|
||||
(`plexmediaserver 1.43.4.10903`). Hardware transcoding had **not** been observed working at the time
|
||||
of writing, and the Plex-side toggle was still unset.
|
||||
+1
-4
@@ -15,7 +15,7 @@ docs/13 exactly. Do not diverge from it here — escalate spec problems, don't f
|
||||
| `servicedefinition.schema.json` | Formal JSON Schema (draft 2020-12) for a ServiceDefinition. Enforces enums, required fields, patterns, and cross-field invariants. |
|
||||
| `renderer-contract.md` | Authoritative field → rendered-resource mapping, with requirement IDs. The contract other workstreams consume. |
|
||||
|
||||
**Schema version:** 1.8.0 (in the schema `description`, which carries the full per-version changelog — this line had said `1.0.0` since 1.1.0 and is now synced). Treat any change as an interface change:
|
||||
**Schema version:** 1.0.0 (in the schema `description`). Treat any change as an interface change:
|
||||
propose on the task board, update dependent renderers/tests in the same change, never silently break a
|
||||
consumer (coordination-protocol §2).
|
||||
|
||||
@@ -35,9 +35,6 @@ consumer (coordination-protocol §2).
|
||||
- `ingressRoutes[].host` must match `<label>.home.keithjackson.dev`.
|
||||
- `storage[].class ∈ {longhorn, bulk}`; `exposure ∈ {internal, public}`; `egress ∈ {internet, vpn}`;
|
||||
`auth.mode ∈ {oidc, forward-auth, native, none}`.
|
||||
- `hardware.gpu ∈ {intel-quicksync}` — a host **capability class**, never a device path. The enum
|
||||
is the contract: the mechanism that satisfies it lives only in `catalog/render.py` (see
|
||||
`renderer-contract.md` §`hardware`).
|
||||
- `additionalProperties: false` everywhere — unknown keys are rejected (catches typos in review/CI).
|
||||
|
||||
## Validate examples against the schema — L1, no cluster (R-TEST-02)
|
||||
|
||||
@@ -160,64 +160,6 @@ platform must render, and cites the requirement each mapping satisfies. Changing
|
||||
- **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication),
|
||||
**R-STATE-04** (bulk tier). (docs/04)
|
||||
|
||||
### `hardware` (`gpu`) — host device capability
|
||||
- **The field is INTENT, never MECHANISM.** `hardware.gpu` is an **enum of capability classes**
|
||||
(today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
|
||||
ServiceDefinition says *what class of host device this workload needs*; how the platform hands it
|
||||
over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw
|
||||
`/dev/...` path.** The entire payoff is that migrating from the hostPath mechanism below to the
|
||||
Intel GPU device plugin (`resources.limits: gpu.intel.com/i915`) is a renderer change with **zero**
|
||||
catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely
|
||||
different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one.
|
||||
- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), exactly two things:
|
||||
1. a **hostPath volume** `dri` → `/dev/dri`, **`type: DirectoryOrCreate`**;
|
||||
2. a **read-write volumeMount** of `dri` at `/dev/dri` on the app container (VA-API opens the
|
||||
render node read-write, so it is deliberately not `readOnly`).
|
||||
|
||||
**Mounting the device is sufficient. That is the whole mapping.**
|
||||
- **`podSecurityContext.supplementalGroups` is deliberately NOT rendered, and adding it would be a
|
||||
regression rather than a hardening.** `/dev/dri/renderD128` is `root:render` `0660`, so the obvious
|
||||
move is to push the host `render` GID in — and it does not work. **Measured in the running
|
||||
container on prod-2, 2026-09-30:** the image's own
|
||||
`/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` hook `find`s every char device under
|
||||
`/dev/dri`, `stat -c '%g'` each one's GID, `groupadd`s a matching group *inside the container* when
|
||||
none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded.
|
||||
`cont-init.d` completes before `services.d`, so the account database is already correct when the
|
||||
service execs. The service then starts via `exec s6-setuidgid plex …`, and `s6-setuidgid`
|
||||
**replaces** the process group list from `/etc/group` rather than appending to it:
|
||||
`Plex Media Server` runs as `Uid: 1000 Gid: 1000 Groups: 100 1000`, with root's gid `0` — which
|
||||
PID 1 (`s6-svscan`) does hold — **gone**. A Kubernetes `supplementalGroups` therefore reaches
|
||||
`s6-svscan` and is discarded before the app ever starts. It would be inert *and* it would mislead
|
||||
whoever debugs this next. The renderer asserts its own absence in `render.py --selftest`.
|
||||
- A corollary worth having: because the hook re-`stat`s the device on every start, the host GID
|
||||
**drifting** (a node rebuild renumbering the *systemd dynamic* `render` group) is self-healing.
|
||||
No GID is pinned anywhere in this repo.
|
||||
- **Scope of that measurement, stated so nobody over-reads it.** "Mounting the device is
|
||||
sufficient" was established for an image that resolves the device group itself. A FUTURE
|
||||
consumer that runs non-root and has no equivalent init hook would still need the group solved
|
||||
somehow — and the answer for that case is a **renderer** change (per-image handling, or the
|
||||
device plugin), not a new ServiceDefinition field. The enum is what keeps that option open. The
|
||||
first such consumer must re-measure rather than assume this bullet covers it.
|
||||
- **`DirectoryOrCreate` is load-bearing — do not tighten it to `Directory`.** A node that comes back
|
||||
without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the
|
||||
workload to **software transcoding**, not fail the mount. `apps-wave-0` is `wait: true` and
|
||||
`apps-wave-1` `dependsOn` it, so a failed hostPath mount stalls the **whole app tier** for the sake
|
||||
of a feature that is not load-bearing. `/dev` is tmpfs, so the empty directory the kubelet creates
|
||||
on such a node leaves nothing durable behind.
|
||||
- **The cost of that choice, stated plainly:** a missing device is **silent**. The workload
|
||||
software-transcodes and nothing reports it. There is no metric to alert on — media apps here
|
||||
render no `metricsEndpoint`, so no ServiceMonitor exists — and an alert that cannot fire is worse
|
||||
than none (see §`observability`'s 2026-09-13 correction). The only proof of hardware transcode is
|
||||
an operator observing it in the application's own dashboard.
|
||||
- **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno
|
||||
`restrict-host-namespaces` is Enforce on app namespaces and would reject it, and the render node
|
||||
does not need it); no `nodeSelector` (placement is a separate concern with its own justification —
|
||||
this field is about the device, not about where the pod lands); no resource requests/limits (there
|
||||
is no app-container resource render path, and inventing one here would be scope creep).
|
||||
- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical.
|
||||
Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`.
|
||||
- **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need).
|
||||
|
||||
### `ingressRoutes[]` (`host` / `port`) — HTTPRoute + Certificate + DNS
|
||||
- Each entry renders an **HTTPRoute** attaching `host` → the service `port` on the shared Gateway.
|
||||
- TLS: the Gateway terminates with the cert-manager **wildcard `*.home.keithjackson.dev`** certificate
|
||||
@@ -344,11 +286,6 @@ platform must render, and cites the requirement each mapping satisfies. Changing
|
||||
`external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it.
|
||||
Derived from `exposure` alone; enforced on every gate run by
|
||||
`conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03).
|
||||
8. **`hardware` is inert until opted into, and its mechanism is renderer-local.** An SD with no
|
||||
`hardware` key renders no volume and no volumeMount; an SD *with* it gains no `securityContext`
|
||||
either. Both directions are proven by `render.py --selftest`. The `/dev/dri` hostPath and the
|
||||
`DirectoryOrCreate` type appear in `catalog/render.py` and nowhere else, so the capability's
|
||||
implementation can be replaced without touching a single ServiceDefinition.
|
||||
|
||||
## Requirement traceability (contract-level)
|
||||
Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
|
||||
"title": "ServiceDefinition",
|
||||
"description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 Line truncated
|
||||
"description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 / R-IDP-02).",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["name", "source"],
|
||||
@@ -139,19 +139,6 @@
|
||||
"supplementalGroups": { "type": "array", "items": { "type": "integer" }, "description": "Additional GIDs added to the first process in each container." }
|
||||
}
|
||||
},
|
||||
"hardware": {
|
||||
"type": "object",
|
||||
"description": "Optional host HARDWARE CAPABILITY the workload needs, for the container kind (docs/13). This field declares INTENT (`what device class this service needs`), never MECHANISM (`which host path to bind-mount`, `which extended resource to request`). That separation is the whole point of the enum: the platform currently satisfies `intel-quicksync` with a /dev/dri hostPath volume and a matching mount on the app container, but a later migration to the Intel GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) must be a catalog/render.py change and NOTHING else - no ServiceDefinition may need editing, and no reviewer may have to re-derive per-service device wiring. Do NOT 'simplify' this into a raw device path, a hostPath, or a resource request; that would push a platform implementation detail into the operator-facing contract and make the migration a catalog-wide edit. Absent => no device wiring at all (existing apps render byte-identical). Ignored for kind=vm (a VM gets devices through KubeVirt's own domain.devices).",
|
||||
"additionalProperties": false,
|
||||
"minProperties": 1,
|
||||
"properties": {
|
||||
"gpu": {
|
||||
"type": "string",
|
||||
"description": "GPU capability class this service needs. `intel-quicksync` = the host's Intel Quick Sync Video render device (the iGPU's /dev/dri/renderD* node) for hardware video transcode/decode. Enum, not a free-form device path, so the satisfying mechanism stays swappable (see the parent description). Extend the enum when a genuinely different capability class appears (e.g. nvidia-nvenc), never to name a different path to the same one.",
|
||||
"enum": ["intel-quicksync"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"exposure": {
|
||||
"type": "string",
|
||||
"description": "internal = LAN-only (internal DNS + Gateway on LAN); public = additionally edge port-forwarded 80/443 (docs/09; R-DNS/R-NET).",
|
||||
|
||||
Reference in new issue
Block a user