Compare commits

...
Author SHA1 Message Date
jakman85andClaude Opus 5 1040d253be feat(schema): declare a host hardware capability, hardware.gpu: intel-quicksync
gate / gate (pull_request) Successful in 1m40s
gate / conformance (pull_request) Has been skipped
A ServiceDefinition could not say it needs the host's Intel QuickSync render
device. The schema is `additionalProperties: false`, so there was no authorable
form for it at all, and `apps/` is generated — a hand-edit would be reverted by
the next render.

Adds optional top-level `hardware`, with one optional member `gpu`, an enum
whose only current value is `intel-quicksync` (schema 1.7.0 -> 1.8.0).

The enum is the point. The SD declares WHAT CLASS of device it needs; the
hostPath and the mount live only in catalog/render.py. Migrating to the Intel
GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) is then
a renderer change with zero catalog edits. Letting an SD name `/dev/dri`
directly would look simpler and make that migration a catalog-wide edit, so
both the schema description and the renderer contract say so explicitly.

When `hardware.gpu == intel-quicksync`, r_deployment emits exactly two things,
on the app container only:

  - a hostPath volume `dri` -> /dev/dri, type DirectoryOrCreate;
  - a read-write volumeMount of it at /dev/dri (VA-API opens the render node
    read-write, so deliberately not readOnly).

Mounting the device is sufficient. In particular NO
podSecurityContext.supplementalGroups is emitted, and adding one would be a
regression rather than a hardening. Measured in the running container on
prod-2, 2026-09-30:

  PID 1   s6-svscan            Uid: 0     Gid: 0     Groups: 0
  pid 246 Plex Media Server    Uid: 1000  Gid: 1000  Groups: 100 1000

The service is started with `exec s6-setuidgid plex ...`, and the child's group
list is exactly the /etc/group account-database list — root's gid 0, which
PID 1 holds, is gone. That is a setgroups() that REPLACES rather than appends,
so anything Kubernetes supplies is discarded before the app starts. What works
is the image's own /etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner,
which stats each /dev/dri char device, groupadds a matching group inside the
container and usermods the app user into it, every start, nothing hardcoded.
An inert supplementalGroups would have read as the mechanism to whoever
debugged this next, so the absence is documented at the emission site and
asserted by --selftest. Corollary: the hook re-stats on every start, so host
GID drift is self-healing and no GID is pinned anywhere in this repo.

`DirectoryOrCreate` is load-bearing. A node that comes back without the render
device (rebuild, BIOS change, driver regression, hardware swap) must degrade
the workload to software transcoding, not fail the mount: apps-wave-0 is
`wait: true` and apps-wave-1 dependsOn it, so a failed hostPath mount stalls
the whole app tier for a feature that is not load-bearing. /dev is tmpfs, so
the empty directory the kubelet creates leaves nothing durable behind. Its cost
— a missing device is silent, and no metric exists to alert on — is recorded in
the contract rather than papered over with an alert that could not fire.

Also deliberately NOT emitted: no `privileged: true` (Kyverno restrict-host-
namespaces is Enforce and it is not needed), no nodeSelector (placement is a
separate concern with its own justification), no resource requests (there is no
app-container resource render path).

No catalog entry uses the field. Re-rendering `--all catalog -o apps` is a zero
diff, which is the proof the change is inert until opted into.

Self-tests in the existing check(...) style: the positive (volume + mount, and
no securityContext / privileged / nodeSelector), the negative (an SD without
`hardware` gains no volume, no mount, no securityContext — mirroring the
podSecurityContext negative) and a pass-through case (an SD's own
podSecurityContext survives byte-for-byte). Note that NO gate stage runs
`render.py --selftest`; they were run by hand and the gap is recorded in the
handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-30 12:48:47 -05:00
jakman85 0fbf544d5e Merge pull request 'fix(obs): the deploy notification was inhibited before it could send' (#26) from fix/flux-deploy-notification-inhibited into main
gate / gate (push) Successful in 1m39s
gate / conformance (push) Has been skipped
Reviewed-on: #26
Reviewed-by: Keith Jackson <[email protected]>
2026-09-30 15:18:54 +00:00
jakman85andClaude Opus 5 8427b28b0e fix(obs): the deploy notification was inhibited before it could send
gate / gate (pull_request) Successful in 1m39s
gate / conformance (pull_request) Has been skipped
FluxRolloutInProgress fired correctly on the first real deploy and its mail
was suppressed before Alertmanager dispatched it. One label was wrong.

kube-prometheus-stack ships general.rules -> InfoInhibitor:

  group by (namespace) (ALERTS{severity="info"} == 1)
    unless on (namespace) group by (namespace)
      (ALERTS{alertname!="InfoInhibitor", alertstate="firing",
              severity=~"warning|critical"} == 1)

which fires for any namespace holding a firing info alert AND NOTHING ELSE
wrong in it, and the chart's third inhibit_rule (source alertname=InfoInhibitor
-> target severity="info", equal [namespace]) then drops exactly that alert.

For a deploy notification that is the worst possible interaction, because the
suppressed case is the ONLY case that matters: on a healthy rollout of a healthy
cluster flux-system holds no warning or critical alert, so the notification is
inhibited precisely when it is wanted, and would arrive only when something else
in flux-system was already broken. Measured on prod: InfoInhibitor went active
2026-09-30T04:58:48Z, the same moment the alert began firing for 12
Kustomizations, and amtool showed no FluxRolloutInProgress in Alertmanager
either active or --inhibited, with zero send failures.

Upstream's intent is explicit and fair: severity "info" means "only worth
showing next to a real problem". A deploy notice is not that, so it must not
claim that severity. FluxRolloutInProgress now carries `notification: deploy`
and no severity at all, and the route matches on that label — no inhibit_rule
in the chart targets anything but severity values, and omitting severity also
stops InfoInhibitor firing for flux-system rather than merely working around it.
FluxRolloutStuck keeps severity: warning and is unaffected.

Also verified on prod in the process, all previously UNVERIFIED: the 33 metric
series exist exactly as designed, the `revision` label IS populated (so the
lastAppliedRevision/artifact.revision paths are right), the podAnnotations roll
forced a new KSM pod, and the rule both fired for 12 lagging Kustomizations and
then CLEARED on convergence — the completion signal's whole mechanism, on real
data. Routing proven by `amtool config routes test notification=deploy` ->
email-deploy.

Still unverified: no mail observed in the inbox. The 226->227 email counter
increment during the window cannot be attributed (no receiver label, other
alerts active), and I am not claiming it either way. See the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-30 00:05:10 -05:00
jakman85 64c72fd394 Merge pull request 'feat(obs): notify on Flux deploy start and completion' (#25) from feat/flux-deploy-notifications into main
gate / gate (push) Successful in 1m39s
gate / conformance (push) Has been skipped
Reviewed-on: #25
Reviewed-by: Keith Jackson <[email protected]>
2026-09-30 04:55:50 +00:00
9 changed files with 520 additions and 11 deletions

No files matched your search

+125
View File
@@ -133,6 +133,15 @@ OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.8.1"
OAUTH2_PROXY_PORT = 4180
KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab"
# --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
# The ServiceDefinition declares the CAPABILITY (schemas/servicedefinition.schema.json
# `hardware.gpu`); everything below is the MECHANISM that currently satisfies it and is
# deliberately confined to this file. Swapping to the Intel GPU device plugin
# (`gpu.intel.com/i915` resource requests instead of a hostPath) is a change here and
# nowhere else -- no catalog entry moves.
DRI_VOLUME_NAME = "dri"
DRI_DEVICE_DIR = "/dev/dri"
# =============================================================================
# Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping.
@@ -483,6 +492,46 @@ def r_deployment(s: dict) -> dict:
"persistentVolumeClaim": {"claimName": f"{s['name']}-{vol['name']}"},
}
)
# hardware capability (schema-optional): `hardware.gpu: intel-quicksync` means this
# workload needs the host iGPU's render node for hardware video transcode. The SD says
# only that; the wiring below is the platform's current mechanism for satisfying it.
#
# `type: DirectoryOrCreate` is DELIBERATE and load-bearing -- do NOT "tighten" it to
# `Directory`. A node that comes back without the render device (rebuild, BIOS change,
# driver regression, hardware swap) must degrade the workload to SOFTWARE transcoding,
# not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` dependsOn it, so a
# failed hostPath mount stalls the entire app tier for the sake of a feature that is not
# load-bearing. `DirectoryOrCreate` turns that failure mode into "transcodes in
# software" instead of "the app tier does not reconcile". /dev is tmpfs, so the empty
# directory the kubelet creates on such a node leaves nothing durable behind.
#
# The mount is deliberately NOT readOnly: VA-API opens the render node read-write.
#
# `podSecurityContext.supplementalGroups` is DELIBERATELY NOT SET here, and adding it
# would be a regression, not a hardening. MEASURED in the running container on prod-2,
# 2026-09-30: the image's own cont-init hook
# `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` `find`s every char device
# under /dev/dri, `stat -c '%g'` each one's GID, `groupadd`s a matching group INSIDE the
# container when none exists, and `usermod -a -G` the app user into it -- every start,
# nothing hardcoded. cont-init.d completes before services.d, so the account database
# s6 reads at exec time already carries the group. The service is then started with
# `exec s6-setuidgid plex ...`, and `s6-setuidgid` REPLACES the process group list from
# /etc/group rather than appending to it: `Plex Media Server` runs with
# `Uid: 1000 Gid: 1000 Groups: 100 1000` -- root's gid 0, which PID 1 (s6-svscan) does
# have, is GONE. Anything Kubernetes puts in supplementalGroups reaches s6-svscan and is
# then discarded before the app ever starts. MOUNTING THE DEVICE IS SUFFICIENT; the
# group handling is the image's job and it does it correctly. A supplementalGroups entry
# here would be inert AND would mislead whoever debugs this next.
hw_gpu = (s.get("hardware") or {}).get("gpu")
if hw_gpu == "intel-quicksync":
volume_mounts.append({"name": DRI_VOLUME_NAME, "mountPath": DRI_DEVICE_DIR})
volumes.append(
{
"name": DRI_VOLUME_NAME,
"hostPath": {"path": DRI_DEVICE_DIR, "type": "DirectoryOrCreate"},
}
)
if volume_mounts:
container["volumeMounts"] = volume_mounts
@@ -2440,6 +2489,82 @@ def _selftest() -> int:
"an SD without podSecurityContext must not gain a pod securityContext",
)
# 6b) hardware.gpu: intel-quicksync -> /dev/dri hostPath volume + RW mount + the host
# `render` GID. The SD declares the CAPABILITY; these assertions pin the MECHANISM
# that currently satisfies it, so a change to it is a deliberate, visible edit.
hw_docs = render(
{
"name": "hw",
"source": {"image": "a:1"},
"hardware": {"gpu": "intel-quicksync"},
}
)
hw_pod = next(
d[1] for d in hw_docs if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]
check(
{
"name": "dri",
"hostPath": {"path": "/dev/dri", "type": "DirectoryOrCreate"},
}
in hw_pod.get("volumes", []),
"hardware.gpu=intel-quicksync must render a /dev/dri hostPath volume "
"with type DirectoryOrCreate (a GPU-less node must degrade, not fail the mount)",
)
hw_mounts = hw_pod["containers"][0].get("volumeMounts", [])
check(
{"name": "dri", "mountPath": "/dev/dri"} in hw_mounts,
"hardware.gpu=intel-quicksync must mount /dev/dri READ-WRITE on the app container",
)
# The device mount is the WHOLE feature: no pod securityContext is invented. The image's
# own cont-init hook adds the app user to the device's group inside the container, and
# s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway (see the
# emission site). Asserting the absence is what stops someone adding an inert one back.
check(
"securityContext" not in hw_pod,
"hardware.gpu must NOT invent a pod securityContext (supplementalGroups is inert "
"under s6-setuidgid; the image's cont-init hook handles the device group)",
)
# Nothing privileged, and no node pinning: Kyverno restrict-host-namespaces (Enforce)
# rejects `privileged: true` in an app namespace, and placement is a separate concern
# with its own justification -- this field is about the DEVICE only.
check(
"privileged" not in hw_pod.get("securityContext", {})
and "securityContext" not in hw_pod["containers"][0]
and "nodeSelector" not in hw_pod,
"hardware.gpu must emit no privileged securityContext and no nodeSelector",
)
# The regression that matters: absence renders NOTHING. Mirrors the podSecurityContext
# negative above -- an SD that does not opt in must be byte-identical to before.
nohw_pod = next(
d[1] for d in render({"name": "nohw", "source": {"image": "a:1"}})
if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]
check(
"volumes" not in nohw_pod
and "volumeMounts" not in nohw_pod["containers"][0]
and "securityContext" not in nohw_pod,
"an SD without `hardware` must gain no dri volume, no mount and no securityContext",
)
# An SD that declares BOTH keeps its own podSecurityContext exactly as written -- the
# hardware field neither adds to it nor rewrites it.
hwm_pod = next(
d[1]
for d in render(
{
"name": "hwm",
"source": {"image": "a:1"},
"hardware": {"gpu": "intel-quicksync"},
"podSecurityContext": {"fsGroup": 1001, "supplementalGroups": [44]},
}
)
if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]
check(
hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]},
"hardware.gpu must leave the SD's own podSecurityContext untouched",
)
# 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that
# fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged.
fa_docs = render(
@@ -0,0 +1,95 @@
---
date: 2026-09-30
branch: fix/flux-deploy-notification-inhibited
title: The deploy notification routed correctly and was then inhibited before sending — `severity: info` is a suppressed class in kube-prometheus-stack
---
## What changed
`FluxRolloutInProgress` (shipped hours earlier in PR #25) fired correctly on the very first real
deploy and **its mail was suppressed before Alertmanager sent it**. One label changed:
- `platform/monitoring/alerts.yaml` — `FluxRolloutInProgress` now carries `notification: deploy` and
**no `severity` label at all**, where it previously carried `severity: info`.
- `platform/monitoring/kube-prometheus-stack.yaml` — the deploy route matches
`notification = "deploy"` instead of `severity = "info"`. Receiver, subject and grouping unchanged.
- `platform/monitoring/tests/flux-rollout.test.yaml` — the three `FluxRolloutInProgress` `exp_labels`
blocks updated. `FluxRolloutStuck` keeps `severity: warning` and is unaffected.
**The mechanism, measured rather than reasoned about.** kube-prometheus-stack's `general.rules`
ships:
InfoInhibitor:
group by (namespace) (ALERTS{severity="info"} == 1)
unless on (namespace) group by (namespace)
(ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
which fires for any namespace holding a firing `severity="info"` alert **and nothing else wrong in
it**. The chart's third `inhibit_rule` then drops exactly that alert:
source_matchers: [alertname="InfoInhibitor"]
target_matchers: [severity="info"]
equal: [namespace]
For a deploy notification this is the worst possible interaction, because the suppressed case is the
*only* case that matters: on a healthy rollout of a healthy cluster, `flux-system` holds no warning
or critical alert, so `InfoInhibitor` fires and the deploy mail is inhibited. It would have been
delivered only when something else in `flux-system` was already broken.
Upstream's intent is explicit and reasonable — `severity: info` means "only worth showing next to a
real problem". A deploy notice is not that, so it must not claim that severity. `notification` is a
label **no** `inhibit_rule` in the chart targets (all three match on severity values), and dropping
`severity` also stops `InfoInhibitor` firing for `flux-system` at all rather than merely working
around it. Both files carry the full reasoning inline, and `alerts.yaml`'s design header now names
the severity label as a third thing that must not regress — "do not tidy this by adding a severity"
— because the next reader's instinct will be to add one back.
## What I verified
Against live prod, after PR #25 merged as `64c72fd` and Flux converged:
- **The metrics work.** `kube_flux_kustomization_info` 16, `kube_flux_kustomization_suspended` 16,
`kube_flux_gitrepository_info` 1 = **33 series, exactly as designed**. The `revision` label IS
populated (`main@sha1:296d6d3…` / `…64c72fd…`), so the `status.lastAppliedRevision` /
`status.artifact.revision` paths are correct — previously UNVERIFIED.
- **The `podAnnotations` roll mechanism works.** The KSM pod carries
`homelab.keithjackson.dev/crs-config-rollout: 1` and is a new pod
(`…-7b56d98f46-56n8j`, replacing the 2026-09-19 one), so the config-only-change trap was
genuinely closed rather than merely documented.
- **The rule works, in both directions.** It fired for **12** lagging Kustomizations while 4 had
converged (`count by (revision)` → 12 old / 4 new), each alert carrying its own `name` and
`revision`; then the rollout converged (`count(count by (revision) (…))` → **1**) and the alert
**cleared**. Firing and self-clearing are both confirmed on real data, which is the completion
notification's whole mechanism.
- **The routing works.** `amtool config routes` shows
`{notification="deploy"} → receiver: email-deploy` under the default route, and
`amtool config routes test notification=deploy` returns `email-deploy`.
- **The inhibition was real.** `InfoInhibitor` went active `2026-09-30T04:58:48Z` — the same moment
the alert began firing — and `amtool alert query` showed **no** `FluxRolloutInProgress` in
Alertmanager, active or `--inhibited`. `alertmanager_notifications_failed_total{integration=email}`
is 0 for every reason, so nothing failed to send; it was never dispatched.
- promtool: **SUCCESS**, 53 rules, 6 scenarios, promtool 3.14.0 in the pinned toolbox.
## What I did NOT verify
- **No deploy mail has been observed in the inbox — this fix is UNVERIFIED end to end.** The
suppression was proven (nothing in Alertmanager, zero send failures) and the routing was proven
(`amtool config routes test` → `email-deploy`), but the delivered message is not. It can only be
proven by merging this and watching the next commit produce `[FLUX] applying` and then
`[FLUX] applied`. Same structural gap as PR #25, one layer further along.
- **Whether the single email during the window was ours.**
`alertmanager_notifications_total{integration="email"}` went 226 → 227 while other alerts
(`Watchdog`, `CNPGPrimariesCoLocated`) were also active, and that counter carries no receiver
label, so the increment cannot be attributed. I am NOT claiming our mail was or was not sent — the
attribution is genuinely unavailable from this metric.
- **Whether `InfoInhibitor` interacts with `FluxRolloutStuck`.** Stuck is `severity: warning`;
inhibit_rule 1 (`critical` → `warning|info`) requires an equal `alertname`, so it should not
apply — reasoned, not measured. Worth confirming the first time a rollout genuinely hangs.
- **The `[FLUX] applying|applied` subject template** is still untested; `.Status` is assumed to be
`firing`/`resolved` in the notification template data.
- **`required_approvals` disagreement, unresolved and not mine to settle.** Live branch protection
reads `required_approvals: 1`, while the `ci/gate.sh` comment introduced by PR #17 describes the
model as `required_approvals: 0` ("the gate is the approver"). `ci/gitea-settings.sh` may report
this as drift against `infra/gitea/expected-branch-protection.json`. Flagged only.
- **The drift loop is still unexplained.** Five Kustomizations re-applying the same object every
10 minutes (event counts ~2500) — untouched, cause unknown, still deserves its own branch.
@@ -0,0 +1,176 @@
---
date: 2026-09-30
branch: feat/sd-hardware-capability
title: ServiceDefinition gains a `hardware` capability field — `gpu: intel-quicksync` mounts /dev/dri and nothing else; the enum is what keeps the mechanism swappable
---
## What changed
A new optional top-level ServiceDefinition property, `hardware`, with one optional member,
`gpu`, whose enum today has exactly one value: `intel-quicksync`. **No catalog entry uses it.**
This PR delivers the capability; consuming it (Plex) is separate work on top.
- `schemas/servicedefinition.schema.json` — adds `hardware` (object, `additionalProperties: false`,
`minProperties: 1`) with `gpu` (enum `["intel-quicksync"]`). Schema version bumped **1.7.0 →
1.8.0**, changelog entry prepended to the schema `description` per that file's own convention.
- `catalog/render.py` — two module-level constants (`DRI_VOLUME_NAME`, `DRI_DEVICE_DIR`) and the
emission in `r_deployment`.
- `schemas/renderer-contract.md` — new `### hardware` field-mapping section, plus cross-field
invariant **8** in that file's numbered list.
- `schemas/README.md` — its `**Schema version:**` line said `1.0.0`; it had been stale since 1.1.0.
Synced to 1.8.0 and annotated that the schema `description` is the changelog of record. Added one
bullet to "Cross-field rules" for the new enum. That was the whole of the staleness; nothing
ballooned.
**What the renderer emits** for `hardware.gpu: intel-quicksync`, on the app container only (never a
sidecar), and it is exactly two things:
1. a hostPath volume `dri` → `/dev/dri`, `type: DirectoryOrCreate`;
2. a read-write `volumeMount` of `dri` at `/dev/dri` on the app container (deliberately not
`readOnly` — VA-API opens the render node read-write).
**Mounting the device is sufficient.** It emits **no** `privileged: true` (Kyverno
`restrict-host-namespaces` is Enforce on app namespaces and it is not needed), **no** `nodeSelector`
(placement is a separate concern with its own justification), **no** resource requests (there is no
app-container resource render path), and — the one that is worth spelling out — **no
`podSecurityContext.supplementalGroups`**.
**Why no `supplementalGroups`, when `/dev/dri/renderD128` is `root:render 0660`.** An earlier
revision of this branch merged the host `render` GID into `supplementalGroups`. **It was measured to
be a no-op and has been removed.** In the running container on prod-2, 2026-09-30:
PID 1 s6-svscan Uid: 0 Gid: 0 Groups: 0
pid 246 Plex Media Server Uid: 1000 Gid: 1000 Groups: 100 1000
`/etc/services.d/plex/run` ends in `exec s6-setuidgid plex …`, and the child's group list is exactly
the `/etc/group` account-database list — root's gid `0`, which PID 1 holds, has **vanished**. That is
a `setgroups()` that *replaces* rather than appends, so anything Kubernetes puts in
`supplementalGroups` reaches `s6-svscan` and is discarded before the app ever starts. What actually
works is the image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner`, which `find`s
every char device under `/dev/dri`, `stat -c '%g'`s its GID, `groupadd`s a matching group inside the
container when none exists, and `usermod -a -G`s the app user into it — every start, nothing
hardcoded; `cont-init.d` completes before `services.d`, so the database is correct at exec time.
Shipping an inert `supplementalGroups` would have been worse than shipping nothing: it would read as
the mechanism to whoever debugs this next. The reasoning lives in a comment **at the emission site**
and in `renderer-contract.md`, and `render.py --selftest` now asserts the *absence* of a
`securityContext`, so it cannot be "helpfully" added back without a test failing. A pleasant
corollary: because the hook re-`stat`s the device every start, host GID drift is **self-healing**,
and no GID is pinned anywhere in this repo.
**The design decision worth defending: the field is an enum of intent, not a device path.** The SD
says *what class of host device the workload needs*; the hostPath and the mount exist only in
`catalog/render.py`. Migrating to the Intel GPU device plugin (`gpu.intel.com/i915` resource
requests, no hostPath at all) is then a renderer change with zero catalog edits. Both the schema
`description` and the contract section say this explicitly, because the obvious "simplification" —
letting an SD name `/dev/dri` — would push a platform implementation detail into the operator-facing
contract and make that migration a catalog-wide edit.
**`DirectoryOrCreate` is load-bearing.** A node that comes back without the render device (rebuild,
BIOS change, driver regression, hardware swap) must degrade the workload to software transcoding,
not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so a failed
hostPath mount stalls the entire app tier for the sake of a feature that is not load-bearing.
`/dev` is tmpfs, so the empty directory the kubelet creates on such a node leaves nothing durable
behind. Do not "tighten" it to `Directory`. The reasoning is recorded at the emission site.
## What I verified
Measured, in this worktree, on this branch:
- **`python3 catalog/render.py --selftest` → `render.py selftest: PASS`, rc 0.** New cases in the
existing `check(...)` style: the positive (volume + mount, and *no* `securityContext`, *no*
privileged, *no* nodeSelector), the negative that matters (an SD without `hardware` gains no
volume, no volumeMount and no `securityContext` — mirroring the existing `podSecurityContext`
negative), and a pass-through case (an SD declaring both `hardware` and its own
`podSecurityContext: {fsGroup: 1001, supplementalGroups: [44]}` keeps it **exactly** as written).
- **The self-tests actually bite.** I mutated the guard (`if hw_gpu == "intel-quicksync":` → `if
True:`) in a scratch copy and re-ran: **rc 1**, failing on "an SD without `hardware` must gain no
dri volume, no mount and no securityContext". A test that cannot fail is not a test; this one can.
- **Re-render is a zero diff.**
`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` then
`git diff --stat apps/` → **empty**, and `git status --porcelain apps/` → **empty** (so the two
not-onboarded by-products, `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, were
rewritten byte-identically and needed no revert). The change is inert until a service opts in.
- **Schema validity and catalog conformance.** `Draft202012Validator.check_schema` passes; all
**12** `catalog/*.yaml` validate with **0** errors. On the new field specifically: `gpu:
intel-quicksync` accepted; `gpu: /dev/dri` rejected (enum); `hardware: {gpu: …, path: …}` rejected
(`additionalProperties: false`); `hardware: {}` rejected (`minProperties: 1`).
- **Hand-checked the emitted YAML** by rendering a throwaway SD carrying `hardware`, a Longhorn
volume and its own `podSecurityContext: {runAsUser: 1000, supplementalGroups: [44]}`. Result:
`volumeMounts` = config then `{name: dri, mountPath: /dev/dri}`; `volumes` = the PVC then
`{name: dri, hostPath: {path: /dev/dri, type: DirectoryOrCreate}}`; `securityContext` =
`{runAsUser: 1000, supplementalGroups: [44]}` — i.e. **unchanged**, nothing appended. That toy SD
was never written into `catalog/`.
- **GATE: `rc 3` — UNDETERMINED, NOT green.** Run in the pinned toolbox from this worktree with
both CLAUDE.md §3 workarounds (`DEV_KUBECONFIG=/dev/null`, and an `ENGINE` shim bind-mounting the
main checkout's `.git` at its own absolute path). Diff base `origin/main @ 0fbf544d5e2f`; the run's
`HEAD` was `d7df88c9c264`, which differs from the final commit only by this table being written
into this file. Touched requirements `R-DNS-03 R-GITOPS-02 R-IDP-02`. Per stage:
| stage | result |
|---|---|
| spec-immutability | **SKIPPED** — sibling spec repo not at `/work/../home-network-k3s`. Expected from a worktree; tolerated by §3. |
| decisions-log | pass |
| lint | pass |
| l0 | pass — 19 offline cases, `0 failure(s), 0 skip(s), 0 unwired` |
| render-golden | pass — *"all 10 onboarded service golden(s) reproduce from the canonical render"* |
| l1 | pass — 3 offline checks, `0 failure(s), 0 skip(s), 0 unwired`; coverage 51/51 requirements scaffolded |
| approval-policy | pass (`event='none'`, not a pull_request event — N/A locally; it will evaluate for real on the PR) |
| gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset. Expected; branch protection and runner registration are therefore UNVERIFIED by this run. |
| conformance (L2/L3) | **SKIPPED** — `CLUSTER_AVAILABLE!=true`. Phase-0 standing skip. |
The gate's own verdict: *"Stages that did not run: spec-immutability gitea-settings"*. **No stage
failed, and no stage skipped for a reason caused by this change.** rc 3 is reported as rc 3; I am
not calling this green.
- **Blast radius of the schema change:** `hardware` is absent from every catalog entry, so
`apply_defaults` never materializes it and `s.get("hardware")` is `None` on all 12 — which is what
the zero diff above demonstrates rather than asserts.
- **Touched no sensitive path.** Changes are confined to `schemas/`, `catalog/`, `decisions/`; per
`infra/gitea/approval-policy.json` none of those are sensitive. `conformance/**` and `clusters/**`
are untouched. I did not run `--emit-waves` (it writes `clusters/base/`, and the set of files
under `apps/` did not change).
## What I did NOT verify
- **Whether containerd's device cgroup permits a non-privileged pod to `open()` a bind-mounted char
device. Still unverified, and it is the single largest technical risk in this workstream.**
containerd installs a per-container **device cgroup** allow-list, and whether
`/dev/dri/renderD128` arriving as a bind-mounted device node is permitted by it on this
k3s/containerd version is **unmeasured**. Settling it requires applying a pod, which is an
operator action, not an agent one. The container's PID 1 running as uid 0 helps on the DAC side
but does **not** bypass the cgroup gate. If it blocks, the mount will succeed, the image's group
hook will do its job, and the `open()` will still fail — and the fallback is the Intel GPU device
plugin, which *sets* the cgroup entry. Note that that fallback is a **renderer-only change**: no
ServiceDefinition moves, because the authoring surface is an intent enum. That is the design
justifying itself, and it is the reason to keep the enum even if the hostPath mechanism loses.
- **That a missing device degrades silently — and nothing will tell you.** This is the accepted cost
of `DirectoryOrCreate`. The workload will software-transcode and **no signal reports it**: media
apps in this catalog declare no `metricsEndpoint`, so no ServiceMonitor exists, and there is no
metric to alert on. I deliberately did **not** propose an alert, because an alert that cannot fire
is worse than none — see the 2026-09-13 correction in `renderer-contract.md` §`observability`,
where twelve monitors discovered zero targets. The only proof of hardware transcode is an operator
seeing it in the application's own dashboard.
- **The feature now depends on a hook inside an image that floats.** The `cont-init.d` behaviour
above was read from digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`), but the
deployment runs `plexinc/pms-docker:latest` with `imagePullPolicy: Always`. **A silent pull can
change the behaviour this feature depends on, with no commit anywhere in this repo to show for
it** — and the failure would be the silent software-transcode described above. I have not fixed
that here (it is the Plex wiring's tag/digest decision, not the capability's), but it should not
be forgotten.
- **GID drift is NOT on this list any more, deliberately.** An earlier revision pinned
`INTEL_RENDER_GID = 992` and listed "a node rebuild could move it" as an open risk. The hook
re-`stat`s the device on every start, so drift is self-healing and no GID is pinned anywhere. The
risk is retired, not overlooked.
- **Anything on the live cluster.** I ran no `kubectl` and I opened no shell in any container. The
node facts (device present, `root:render` 0660, `i915` bound, no device plugin, no
`gpu.intel.com/*` advertised), the in-container measurements quoted above, and the Kyverno
posture (`restrict-host-namespaces` Enforce; no hostPath or `supplementalGroups`
restriction; no PSA labels on app namespaces) are **prior recon handed to me, not re-measured by
me**. Per CLAUDE.md §2b, treat them as second-hand until re-queried.
- **That CI exercises the self-tests. It does not.** `git grep -- --selftest ci/` returns nothing —
no gate stage invokes the renderer's self-test. They are the executable spec and house style, and
I ran them by hand, but a future regression in this code path **will not be caught by the gate**.
Wiring `--selftest` into a gate stage is a real gap and is not part of this PR (`ci/` is a
sensitive path and would drag in a human reviewer for a change unrelated to this one).
- **Runtime behaviour of any kind.** The gate proves the L0/L1 offline subset only;
`vars.CLUSTER_AVAILABLE` is unset, so L2/L3 conformance does not run in CI.
+33 -1
View File
@@ -839,6 +839,14 @@ spec:
# that has never applied anything — or a GitRepository with no artifact yet — publishes the
# series with no `revision` label. Without the matcher those group together under the empty
# string and compare EQUAL, i.e. a never-applied Kustomization would read as converged.
#
# A THIRD THING THAT MUST NOT REGRESS — THE SEVERITY LABEL. These alerts are delivered
# because of the labels they do NOT carry as much as the ones they do. Do not "tidy" this
# pair by giving FluxRolloutInProgress a `severity`, and never `severity: info`: the chart's
# InfoInhibitor rule plus its `severity="info"` inhibit_rule suppress an info alert whenever
# it is the only thing firing in its namespace, which is every healthy deploy. Measured on
# prod 2026-09-30 — the alert fired correctly for 12 Kustomizations, resolved correctly, and
# was inhibited in between. The full reasoning is on the `notification: deploy` label below.
# ===================================================================================
- alert: FluxRolloutInProgress
expr: |
@@ -857,7 +865,31 @@ spec:
# `flux reconcile`, a change touching one Kustomization), and a deploy notification that
# only sometimes arrives is worse than no notification at all.
labels:
severity: info # -> the email-deploy Alertmanager route; subject "[FLUX] applying/applied"
# ROUTED BY `notification`, AND DELIBERATELY CARRYING NO `severity` (fixed 2026-09-30).
# This alert originally used `severity: info`. That was WRONG, and wrong in the silent
# direction — measured on prod within minutes of the first real deploy.
# kube-prometheus-stack ships `general.rules` -> InfoInhibitor:
# group by (namespace) (ALERTS{severity="info"} == 1)
# unless on (namespace) group by (namespace)
# (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
# which fires for any namespace that has a firing info alert AND NOTHING ELSE WRONG in
# it; paired with the chart's inhibit_rule
# source_matchers: [alertname="InfoInhibitor"]
# target_matchers: [severity="info"]
# equal: [namespace]
# the effect is that an info alert is suppressed EXACTLY WHEN it is the only thing
# happening in its namespace. For a deploy notification that is the only case that
# matters: on a healthy rollout of a healthy cluster, flux-system has no warning or
# critical alert, so InfoInhibitor fired and this alert never reached the receiver.
# Verified live: InfoInhibitor went active 2026-09-30T04:58:48Z, the same moment this
# alert began firing for 12 Kustomizations.
# Upstream's intent is explicit — `severity: info` means "only worth showing next to a
# real problem". A deploy notice is not that, so it must not claim that severity.
# `notification` is a label no inhibit_rule in the chart targets (all three match on
# severity values), and omitting `severity` also stops InfoInhibitor firing for
# flux-system at all. Route: platform/monitoring/kube-prometheus-stack.yaml matches
# `notification = "deploy"` -> the email-deploy receiver, subject "[FLUX] applying|applied".
notification: deploy
annotations:
summary: "Flux is applying a new revision — {{ $labels.name }} is behind main"
description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+13 -5
View File
@@ -139,11 +139,19 @@ spec:
#
# A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a
# routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a
# real platform fault, which is how alert mail stops being read at all. `severity: info`
# is used by NO other rule in this repo (grepped 2026-09-29: zero matches), so this
# route claims exactly the deploy notices and nothing else. Keep it that way — giving a
# genuine fault severity: info would silently route it here and re-label it a deploy.
- matchers: ['severity = "info"']
# real platform fault, which is how alert mail stops being read at all.
#
# MATCHES `notification`, NOT `severity` (corrected 2026-09-30). This route first read
# `severity = "info"`, which routed correctly and then had the mail SUPPRESSED before it
# was sent: the chart's own InfoInhibitor rule fires for any namespace holding a firing
# info alert with no warning/critical beside it, and its inhibit_rule
# (source alertname="InfoInhibitor" -> target severity="info", equal [namespace]) drops
# exactly that alert. A healthy deploy in a healthy flux-system is precisely that case,
# so the notification was inhibited on every occasion it was wanted. Measured live on
# prod 2026-09-30: InfoInhibitor went active at 04:58:48Z as the alert began firing.
# Routing on a non-severity label sidesteps all three of the chart's inhibit_rules,
# which match on severity values only. See the label's own comment in alerts.yaml.
- matchers: ['notification = "deploy"']
receiver: email-deploy
# One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging
# Kustomization (16 possible), and grouping on alertname alone collapses them into a
@@ -89,7 +89,7 @@ tests:
alertname: FluxRolloutInProgress
exp_alerts:
- exp_labels:
severity: info
notification: deploy
job: kube-state-metrics
namespace: flux-system
name: platform
@@ -148,7 +148,7 @@ tests:
alertname: FluxRolloutInProgress
exp_alerts:
- exp_labels:
severity: info
notification: deploy
job: kube-state-metrics
namespace: flux-system
name: platform
@@ -229,7 +229,7 @@ tests:
alertname: FluxRolloutInProgress
exp_alerts:
- exp_labels:
severity: info
notification: deploy
job: kube-state-metrics
namespace: flux-system
name: conformance
+4 -1
View File
@@ -15,7 +15,7 @@ docs/13 exactly. Do not diverge from it here — escalate spec problems, don't f
| `servicedefinition.schema.json` | Formal JSON Schema (draft 2020-12) for a ServiceDefinition. Enforces enums, required fields, patterns, and cross-field invariants. |
| `renderer-contract.md` | Authoritative field → rendered-resource mapping, with requirement IDs. The contract other workstreams consume. |
**Schema version:** 1.0.0 (in the schema `description`). Treat any change as an interface change:
**Schema version:** 1.8.0 (in the schema `description`, which carries the full per-version changelog — this line had said `1.0.0` since 1.1.0 and is now synced). Treat any change as an interface change:
propose on the task board, update dependent renderers/tests in the same change, never silently break a
consumer (coordination-protocol §2).
@@ -35,6 +35,9 @@ consumer (coordination-protocol §2).
- `ingressRoutes[].host` must match `<label>.home.keithjackson.dev`.
- `storage[].class ∈ {longhorn, bulk}`; `exposure ∈ {internal, public}`; `egress ∈ {internet, vpn}`;
`auth.mode ∈ {oidc, forward-auth, native, none}`.
- `hardware.gpu ∈ {intel-quicksync}` — a host **capability class**, never a device path. The enum
is the contract: the mechanism that satisfies it lives only in `catalog/render.py` (see
`renderer-contract.md` §`hardware`).
- `additionalProperties: false` everywhere — unknown keys are rejected (catches typos in review/CI).
## Validate examples against the schema — L1, no cluster (R-TEST-02)
+57
View File
@@ -160,6 +160,58 @@ platform must render, and cites the requirement each mapping satisfies. Changing
- **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication),
**R-STATE-04** (bulk tier). (docs/04)
### `hardware` (`gpu`) — host device capability
- **The field is INTENT, never MECHANISM.** `hardware.gpu` is an **enum of capability classes**
(today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
ServiceDefinition says *what class of host device this workload needs*; how the platform hands it
over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw
`/dev/...` path.** The entire payoff is that migrating from the hostPath mechanism below to the
Intel GPU device plugin (`resources.limits: gpu.intel.com/i915`) is a renderer change with **zero**
catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely
different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one.
- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), exactly two things:
1. a **hostPath volume** `dri` → `/dev/dri`, **`type: DirectoryOrCreate`**;
2. a **read-write volumeMount** of `dri` at `/dev/dri` on the app container (VA-API opens the
render node read-write, so it is deliberately not `readOnly`).
**Mounting the device is sufficient. That is the whole mapping.**
- **`podSecurityContext.supplementalGroups` is deliberately NOT rendered, and adding it would be a
regression rather than a hardening.** `/dev/dri/renderD128` is `root:render` `0660`, so the obvious
move is to push the host `render` GID in — and it does not work. **Measured in the running
container on prod-2, 2026-09-30:** the image's own
`/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` hook `find`s every char device under
`/dev/dri`, `stat -c '%g'` each one's GID, `groupadd`s a matching group *inside the container* when
none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded.
`cont-init.d` completes before `services.d`, so the account database is already correct when the
service execs. The service then starts via `exec s6-setuidgid plex …`, and `s6-setuidgid`
**replaces** the process group list from `/etc/group` rather than appending to it:
`Plex Media Server` runs as `Uid: 1000 Gid: 1000 Groups: 100 1000`, with root's gid `0` — which
PID 1 (`s6-svscan`) does hold — **gone**. A Kubernetes `supplementalGroups` therefore reaches
`s6-svscan` and is discarded before the app ever starts. It would be inert *and* it would mislead
whoever debugs this next. The renderer asserts its own absence in `render.py --selftest`.
- A corollary worth having: because the hook re-`stat`s the device on every start, the host GID
**drifting** (a node rebuild renumbering the *systemd dynamic* `render` group) is self-healing.
No GID is pinned anywhere in this repo.
- **`DirectoryOrCreate` is load-bearing — do not tighten it to `Directory`.** A node that comes back
without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the
workload to **software transcoding**, not fail the mount. `apps-wave-0` is `wait: true` and
`apps-wave-1` `dependsOn` it, so a failed hostPath mount stalls the **whole app tier** for the sake
of a feature that is not load-bearing. `/dev` is tmpfs, so the empty directory the kubelet creates
on such a node leaves nothing durable behind.
- **The cost of that choice, stated plainly:** a missing device is **silent**. The workload
software-transcodes and nothing reports it. There is no metric to alert on — media apps here
render no `metricsEndpoint`, so no ServiceMonitor exists — and an alert that cannot fire is worse
than none (see §`observability`'s 2026-09-13 correction). The only proof of hardware transcode is
an operator observing it in the application's own dashboard.
- **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno
`restrict-host-namespaces` is Enforce on app namespaces and would reject it, and the render node
does not need it); no `nodeSelector` (placement is a separate concern with its own justification —
this field is about the device, not about where the pod lands); no resource requests/limits (there
is no app-container resource render path, and inventing one here would be scope creep).
- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical.
Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`.
- **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need).
### `ingressRoutes[]` (`host` / `port`) — HTTPRoute + Certificate + DNS
- Each entry renders an **HTTPRoute** attaching `host` → the service `port` on the shared Gateway.
- TLS: the Gateway terminates with the cert-manager **wildcard `*.home.keithjackson.dev`** certificate
@@ -286,6 +338,11 @@ platform must render, and cites the requirement each mapping satisfies. Changing
`external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it.
Derived from `exposure` alone; enforced on every gate run by
`conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03).
8. **`hardware` is inert until opted into, and its mechanism is renderer-local.** An SD with no
`hardware` key renders no volume and no volumeMount; an SD *with* it gains no `securityContext`
either. Both directions are proven by `render.py --selftest`. The `/dev/dri` hostPath and the
`DirectoryOrCreate` type appear in `catalog/render.py` and nowhere else, so the capability's
implementation can be replaced without touching a single ServiceDefinition.
## Requirement traceability (contract-level)
Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02
+14 -1
View File
@@ -2,7 +2,7 @@
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
"title": "ServiceDefinition",
"description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 / R-IDP-02).",
"description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 Line truncated
"type": "object",
"additionalProperties": false,
"required": ["name", "source"],
@@ -139,6 +139,19 @@
"supplementalGroups": { "type": "array", "items": { "type": "integer" }, "description": "Additional GIDs added to the first process in each container." }
}
},
"hardware": {
"type": "object",
"description": "Optional host HARDWARE CAPABILITY the workload needs, for the container kind (docs/13). This field declares INTENT (`what device class this service needs`), never MECHANISM (`which host path to bind-mount`, `which extended resource to request`). That separation is the whole point of the enum: the platform currently satisfies `intel-quicksync` with a /dev/dri hostPath volume and a matching mount on the app container, but a later migration to the Intel GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) must be a catalog/render.py change and NOTHING else - no ServiceDefinition may need editing, and no reviewer may have to re-derive per-service device wiring. Do NOT 'simplify' this into a raw device path, a hostPath, or a resource request; that would push a platform implementation detail into the operator-facing contract and make the migration a catalog-wide edit. Absent => no device wiring at all (existing apps render byte-identical). Ignored for kind=vm (a VM gets devices through KubeVirt's own domain.devices).",
"additionalProperties": false,
"minProperties": 1,
"properties": {
"gpu": {
"type": "string",
"description": "GPU capability class this service needs. `intel-quicksync` = the host's Intel Quick Sync Video render device (the iGPU's /dev/dri/renderD* node) for hardware video transcode/decode. Enum, not a free-form device path, so the satisfying mechanism stays swappable (see the parent description). Extend the enum when a genuinely different capability class appears (e.g. nvidia-nvenc), never to name a different path to the same one.",
"enum": ["intel-quicksync"]
}
}
},
"exposure": {
"type": "string",
"description": "internal = LAN-only (internal DNS + Gateway on LAN); public = additionally edge port-forwarded 80/443 (docs/09; R-DNS/R-NET).",