Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5 2279d14ec5 feat(schema): declare a host hardware capability, hardware.gpu: intel-quicksync
gate / gate (pull_request) Successful in 1m41s
gate / conformance (pull_request) Has been skipped
A ServiceDefinition could not say it needs the host's Intel QuickSync render
device. The schema is `additionalProperties: false`, so there was no authorable
form for it at all, and `apps/` is generated — a hand-edit would be reverted by
the next render.

Adds optional top-level `hardware`, with one optional member `gpu`, an enum
whose only current value is `intel-quicksync` (schema 1.7.0 -> 1.8.0).

The enum is the point. The SD declares WHAT CLASS of device it needs; the
hostPath, the mount and the GID live only in catalog/render.py. Migrating to
the Intel GPU device plugin (`gpu.intel.com/i915` resource requests, no
hostPath) is then a renderer change with zero catalog edits. Letting an SD name
`/dev/dri` directly would look simpler and make that migration a catalog-wide
edit, so both the schema description and the renderer contract say so
explicitly.

When `hardware.gpu == intel-quicksync`, r_deployment emits, on the app
container / pod spec only:

  - a hostPath volume `dri` -> /dev/dri, type DirectoryOrCreate;
  - a read-write volumeMount of it at /dev/dri (VA-API opens the render node
    read-write, so deliberately not readOnly);
  - INTEL_RENDER_GID MERGED into podSecurityContext.supplementalGroups, never
    clobbering what the SD already declares.

`DirectoryOrCreate` is load-bearing. A node that comes back without the render
device (rebuild, BIOS change, driver regression, hardware swap) must degrade
the workload to software transcoding, not fail the mount: apps-wave-0 is
`wait: true` and apps-wave-1 dependsOn it, so a failed hostPath mount stalls
the whole app tier for a feature that is not load-bearing. /dev is tmpfs, so
the empty directory the kubelet creates leaves nothing durable behind. Its cost
— a missing device is silent, and no metric exists to alert on — is recorded in
the contract rather than papered over with an alert that could not fire.

Deliberately NOT emitted: no `privileged: true` (Kyverno restrict-host-
namespaces is Enforce and it is not needed), no nodeSelector (placement is a
separate concern with its own justification), no resource requests (there is no
app-container resource render path).

INTEL_RENDER_GID = 992 is a single named constant carrying its provenance:
measured on all three prod nodes on 2026-09-30, and a *systemd dynamic*
allocation rather than a static GID, so a node rebuild can move it.

No catalog entry uses the field. Re-rendering `--all catalog -o apps` is a zero
diff, which is the proof the change is inert until opted into.

Three self-test cases in the existing check(...) style: the positive, the
negative (an SD without `hardware` gains no volume, no mount, no
securityContext — mirroring the podSecurityContext negative) and the merge case
([44] + 992 -> [44, 992], fsGroup preserved). Note that NO gate stage runs
`render.py --selftest`; they were run by hand and the gap is recorded in the
handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-30 10:49:49 -05:00
jakman85 0fbf544d5e Merge pull request 'fix(obs): the deploy notification was inhibited before it could send' (#26) from fix/flux-deploy-notification-inhibited into main
gate / gate (push) Successful in 1m39s
gate / conformance (push) Has been skipped
Reviewed-on: #26
Reviewed-by: Keith Jackson <[email protected]>
2026-09-30 15:18:54 +00:00
jakman85andClaude Opus 5 8427b28b0e fix(obs): the deploy notification was inhibited before it could send
gate / gate (pull_request) Successful in 1m39s
gate / conformance (pull_request) Has been skipped
FluxRolloutInProgress fired correctly on the first real deploy and its mail
was suppressed before Alertmanager dispatched it. One label was wrong.

kube-prometheus-stack ships general.rules -> InfoInhibitor:

  group by (namespace) (ALERTS{severity="info"} == 1)
    unless on (namespace) group by (namespace)
      (ALERTS{alertname!="InfoInhibitor", alertstate="firing",
              severity=~"warning|critical"} == 1)

which fires for any namespace holding a firing info alert AND NOTHING ELSE
wrong in it, and the chart's third inhibit_rule (source alertname=InfoInhibitor
-> target severity="info", equal [namespace]) then drops exactly that alert.

For a deploy notification that is the worst possible interaction, because the
suppressed case is the ONLY case that matters: on a healthy rollout of a healthy
cluster flux-system holds no warning or critical alert, so the notification is
inhibited precisely when it is wanted, and would arrive only when something else
in flux-system was already broken. Measured on prod: InfoInhibitor went active
2026-09-30T04:58:48Z, the same moment the alert began firing for 12
Kustomizations, and amtool showed no FluxRolloutInProgress in Alertmanager
either active or --inhibited, with zero send failures.

Upstream's intent is explicit and fair: severity "info" means "only worth
showing next to a real problem". A deploy notice is not that, so it must not
claim that severity. FluxRolloutInProgress now carries `notification: deploy`
and no severity at all, and the route matches on that label — no inhibit_rule
in the chart targets anything but severity values, and omitting severity also
stops InfoInhibitor firing for flux-system rather than merely working around it.
FluxRolloutStuck keeps severity: warning and is unaffected.

Also verified on prod in the process, all previously UNVERIFIED: the 33 metric
series exist exactly as designed, the `revision` label IS populated (so the
lastAppliedRevision/artifact.revision paths are right), the podAnnotations roll
forced a new KSM pod, and the rule both fired for 12 lagging Kustomizations and
then CLEARED on convergence — the completion signal's whole mechanism, on real
data. Routing proven by `amtool config routes test notification=deploy` ->
email-deploy.

Still unverified: no mail observed in the inbox. The 226->227 email counter
increment during the window cannot be attributed (no receiver label, other
alerts active), and I am not claiming it either way. See the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-30 00:05:10 -05:00
jakman85 64c72fd394 Merge pull request 'feat(obs): notify on Flux deploy start and completion' (#25) from feat/flux-deploy-notifications into main
gate / gate (push) Successful in 1m39s
gate / conformance (push) Has been skipped
Reviewed-on: #25
Reviewed-by: Keith Jackson <[email protected]>
2026-09-30 04:55:50 +00:00
9 changed files with 487 additions and 13 deletions

No files matched your search

+134 -2
View File
@@ -133,6 +133,24 @@ OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.8.1"
OAUTH2_PROXY_PORT = 4180 OAUTH2_PROXY_PORT = 4180
KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab" KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab"
# --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
# The ServiceDefinition declares the CAPABILITY (schemas/servicedefinition.schema.json
# `hardware.gpu`); everything below is the MECHANISM that currently satisfies it and is
# deliberately confined to this file. Swapping to the Intel GPU device plugin
# (`gpu.intel.com/i915` resource requests instead of a hostPath) is a change here and
# nowhere else -- no catalog entry moves.
DRI_VOLUME_NAME = "dri"
DRI_DEVICE_DIR = "/dev/dri"
# GID of the host `render` group, which owns /dev/dri/renderD128 (mode 0660, root:render).
# MEASURED 2026-09-30 on ALL THREE prod nodes (HP EliteDesk 800 G5 Mini, i7-9700T, UHD
# Graphics 630, i915 bound -- not xe): GID 992, group name `render`, identical on each.
# This is a *systemd dynamic* allocation (systemd-udev creates `render` out of the dynamic
# system range at first boot), NOT a Debian/LSB static GID, so it is NOT guaranteed stable:
# a node rebuild, or a reinstall that creates system groups in a different order, can move
# it. If hardware transcode stops working after a node rebuild, re-measure
# `getent group render` on every node before assuming anything else.
INTEL_RENDER_GID = 992
# ============================================================================= # =============================================================================
# Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping. # Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping.
@@ -483,6 +501,30 @@ def r_deployment(s: dict) -> dict:
"persistentVolumeClaim": {"claimName": f"{s['name']}-{vol['name']}"}, "persistentVolumeClaim": {"claimName": f"{s['name']}-{vol['name']}"},
} }
) )
# hardware capability (schema-optional): `hardware.gpu: intel-quicksync` means this
# workload needs the host iGPU's render node for hardware video transcode. The SD says
# only that; the wiring below is the platform's current mechanism for satisfying it.
#
# `type: DirectoryOrCreate` is DELIBERATE and load-bearing -- do NOT "tighten" it to
# `Directory`. A node that comes back without the render device (rebuild, BIOS change,
# driver regression, hardware swap) must degrade the workload to SOFTWARE transcoding,
# not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` dependsOn it, so a
# failed hostPath mount stalls the entire app tier for the sake of a feature that is not
# load-bearing. `DirectoryOrCreate` turns that failure mode into "transcodes in
# software" instead of "the app tier does not reconcile". /dev is tmpfs, so the empty
# directory the kubelet creates on such a node leaves nothing durable behind.
#
# The mount is deliberately NOT readOnly: VA-API opens the render node read-write.
hw_gpu = (s.get("hardware") or {}).get("gpu")
if hw_gpu == "intel-quicksync":
volume_mounts.append({"name": DRI_VOLUME_NAME, "mountPath": DRI_DEVICE_DIR})
volumes.append(
{
"name": DRI_VOLUME_NAME,
"hostPath": {"path": DRI_DEVICE_DIR, "type": "DirectoryOrCreate"},
}
)
if volume_mounts: if volume_mounts:
container["volumeMounts"] = volume_mounts container["volumeMounts"] = volume_mounts
@@ -505,8 +547,19 @@ def r_deployment(s: dict) -> dict:
# pod spec. Its primary use is `fsGroup`, which makes a mounted Longhorn PVC group-writable by a # pod spec. Its primary use is `fsGroup`, which makes a mounted Longhorn PVC group-writable by a
# non-root image (e.g. semaphore runs as UID 1001 and cannot otherwise write its config PVC). # non-root image (e.g. semaphore runs as UID 1001 and cannot otherwise write its config PVC).
# Absent => the key is never set, so apps that declare none render byte-identical. # Absent => the key is never set, so apps that declare none render byte-identical.
if s.get("podSecurityContext"): #
pod_spec["securityContext"] = s["podSecurityContext"] # `hardware.gpu: intel-quicksync` additionally needs the host `render` GID in
# supplementalGroups, because /dev/dri/renderD128 is root:render 0660 and the container
# does not run as root. It is MERGED into whatever the SD already declares rather than
# replacing it -- an SD that sets its own supplementalGroups keeps every one of them.
psc = dict(s.get("podSecurityContext") or {})
if hw_gpu == "intel-quicksync":
groups = list(psc.get("supplementalGroups") or [])
if INTEL_RENDER_GID not in groups:
groups.append(INTEL_RENDER_GID)
psc["supplementalGroups"] = groups
if psc:
pod_spec["securityContext"] = psc
if volumes: if volumes:
pod_spec["volumes"] = volumes pod_spec["volumes"] = volumes
@@ -2440,6 +2493,85 @@ def _selftest() -> int:
"an SD without podSecurityContext must not gain a pod securityContext", "an SD without podSecurityContext must not gain a pod securityContext",
) )
# 6b) hardware.gpu: intel-quicksync -> /dev/dri hostPath volume + RW mount + the host
# `render` GID. The SD declares the CAPABILITY; these assertions pin the MECHANISM
# that currently satisfies it, so a change to it is a deliberate, visible edit.
hw_docs = render(
{
"name": "hw",
"source": {"image": "a:1"},
"hardware": {"gpu": "intel-quicksync"},
}
)
hw_pod = next(
d[1] for d in hw_docs if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]
check(
{
"name": "dri",
"hostPath": {"path": "/dev/dri", "type": "DirectoryOrCreate"},
}
in hw_pod.get("volumes", []),
"hardware.gpu=intel-quicksync must render a /dev/dri hostPath volume "
"with type DirectoryOrCreate (a GPU-less node must degrade, not fail the mount)",
)
hw_mounts = hw_pod["containers"][0].get("volumeMounts", [])
check(
{"name": "dri", "mountPath": "/dev/dri"} in hw_mounts,
"hardware.gpu=intel-quicksync must mount /dev/dri READ-WRITE on the app container",
)
check(
hw_pod.get("securityContext", {}).get("supplementalGroups")
== [INTEL_RENDER_GID],
"hardware.gpu=intel-quicksync must add the host render GID to supplementalGroups",
)
# Nothing privileged, and no node pinning: Kyverno restrict-host-namespaces (Enforce)
# rejects `privileged: true` in an app namespace, and placement is a separate concern
# with its own justification -- this field is about the DEVICE only.
check(
"privileged" not in hw_pod.get("securityContext", {})
and "securityContext" not in hw_pod["containers"][0]
and "nodeSelector" not in hw_pod,
"hardware.gpu must emit no privileged securityContext and no nodeSelector",
)
# The regression that matters: absence renders NOTHING. Mirrors the podSecurityContext
# negative above -- an SD that does not opt in must be byte-identical to before.
nohw_pod = next(
d[1] for d in render({"name": "nohw", "source": {"image": "a:1"}})
if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]
check(
"volumes" not in nohw_pod
and "volumeMounts" not in nohw_pod["containers"][0]
and "securityContext" not in nohw_pod,
"an SD without `hardware` must gain no dri volume, no mount and no securityContext",
)
# Merge, not clobber: an SD that declares its own supplementalGroups keeps them AND
# gains the render GID, and its other podSecurityContext keys survive untouched.
hwm_pod = next(
d[1]
for d in render(
{
"name": "hwm",
"source": {"image": "a:1"},
"hardware": {"gpu": "intel-quicksync"},
"podSecurityContext": {
"fsGroup": 1001,
"supplementalGroups": [44],
},
}
)
if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]
check(
hwm_pod["securityContext"] == {
"fsGroup": 1001,
"supplementalGroups": [44, INTEL_RENDER_GID],
},
"hardware.gpu must MERGE the render GID into the SD's own supplementalGroups "
"(both GIDs, fsGroup preserved), never clobber them",
)
# 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that # 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that
# fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged. # fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged.
fa_docs = render( fa_docs = render(
@@ -0,0 +1,95 @@
---
date: 2026-09-30
branch: fix/flux-deploy-notification-inhibited
title: The deploy notification routed correctly and was then inhibited before sending — `severity: info` is a suppressed class in kube-prometheus-stack
---
## What changed
`FluxRolloutInProgress` (shipped hours earlier in PR #25) fired correctly on the very first real
deploy and **its mail was suppressed before Alertmanager sent it**. One label changed:
- `platform/monitoring/alerts.yaml` — `FluxRolloutInProgress` now carries `notification: deploy` and
**no `severity` label at all**, where it previously carried `severity: info`.
- `platform/monitoring/kube-prometheus-stack.yaml` — the deploy route matches
`notification = "deploy"` instead of `severity = "info"`. Receiver, subject and grouping unchanged.
- `platform/monitoring/tests/flux-rollout.test.yaml` — the three `FluxRolloutInProgress` `exp_labels`
blocks updated. `FluxRolloutStuck` keeps `severity: warning` and is unaffected.
**The mechanism, measured rather than reasoned about.** kube-prometheus-stack's `general.rules`
ships:
InfoInhibitor:
group by (namespace) (ALERTS{severity="info"} == 1)
unless on (namespace) group by (namespace)
(ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
which fires for any namespace holding a firing `severity="info"` alert **and nothing else wrong in
it**. The chart's third `inhibit_rule` then drops exactly that alert:
source_matchers: [alertname="InfoInhibitor"]
target_matchers: [severity="info"]
equal: [namespace]
For a deploy notification this is the worst possible interaction, because the suppressed case is the
*only* case that matters: on a healthy rollout of a healthy cluster, `flux-system` holds no warning
or critical alert, so `InfoInhibitor` fires and the deploy mail is inhibited. It would have been
delivered only when something else in `flux-system` was already broken.
Upstream's intent is explicit and reasonable — `severity: info` means "only worth showing next to a
real problem". A deploy notice is not that, so it must not claim that severity. `notification` is a
label **no** `inhibit_rule` in the chart targets (all three match on severity values), and dropping
`severity` also stops `InfoInhibitor` firing for `flux-system` at all rather than merely working
around it. Both files carry the full reasoning inline, and `alerts.yaml`'s design header now names
the severity label as a third thing that must not regress — "do not tidy this by adding a severity"
— because the next reader's instinct will be to add one back.
## What I verified
Against live prod, after PR #25 merged as `64c72fd` and Flux converged:
- **The metrics work.** `kube_flux_kustomization_info` 16, `kube_flux_kustomization_suspended` 16,
`kube_flux_gitrepository_info` 1 = **33 series, exactly as designed**. The `revision` label IS
populated (`main@sha1:296d6d3…` / `…64c72fd…`), so the `status.lastAppliedRevision` /
`status.artifact.revision` paths are correct — previously UNVERIFIED.
- **The `podAnnotations` roll mechanism works.** The KSM pod carries
`homelab.keithjackson.dev/crs-config-rollout: 1` and is a new pod
(`…-7b56d98f46-56n8j`, replacing the 2026-09-19 one), so the config-only-change trap was
genuinely closed rather than merely documented.
- **The rule works, in both directions.** It fired for **12** lagging Kustomizations while 4 had
converged (`count by (revision)` → 12 old / 4 new), each alert carrying its own `name` and
`revision`; then the rollout converged (`count(count by (revision) (…))` → **1**) and the alert
**cleared**. Firing and self-clearing are both confirmed on real data, which is the completion
notification's whole mechanism.
- **The routing works.** `amtool config routes` shows
`{notification="deploy"} → receiver: email-deploy` under the default route, and
`amtool config routes test notification=deploy` returns `email-deploy`.
- **The inhibition was real.** `InfoInhibitor` went active `2026-09-30T04:58:48Z` — the same moment
the alert began firing — and `amtool alert query` showed **no** `FluxRolloutInProgress` in
Alertmanager, active or `--inhibited`. `alertmanager_notifications_failed_total{integration=email}`
is 0 for every reason, so nothing failed to send; it was never dispatched.
- promtool: **SUCCESS**, 53 rules, 6 scenarios, promtool 3.14.0 in the pinned toolbox.
## What I did NOT verify
- **No deploy mail has been observed in the inbox — this fix is UNVERIFIED end to end.** The
suppression was proven (nothing in Alertmanager, zero send failures) and the routing was proven
(`amtool config routes test` → `email-deploy`), but the delivered message is not. It can only be
proven by merging this and watching the next commit produce `[FLUX] applying` and then
`[FLUX] applied`. Same structural gap as PR #25, one layer further along.
- **Whether the single email during the window was ours.**
`alertmanager_notifications_total{integration="email"}` went 226 → 227 while other alerts
(`Watchdog`, `CNPGPrimariesCoLocated`) were also active, and that counter carries no receiver
label, so the increment cannot be attributed. I am NOT claiming our mail was or was not sent — the
attribution is genuinely unavailable from this metric.
- **Whether `InfoInhibitor` interacts with `FluxRolloutStuck`.** Stuck is `severity: warning`;
inhibit_rule 1 (`critical` → `warning|info`) requires an equal `alertname`, so it should not
apply — reasoned, not measured. Worth confirming the first time a rollout genuinely hangs.
- **The `[FLUX] applying|applied` subject template** is still untested; `.Status` is assumed to be
`firing`/`resolved` in the notification template data.
- **`required_approvals` disagreement, unresolved and not mine to settle.** Live branch protection
reads `required_approvals: 1`, while the `ci/gate.sh` comment introduced by PR #17 describes the
model as `required_approvals: 0` ("the gate is the approver"). `ci/gitea-settings.sh` may report
this as drift against `infra/gitea/expected-branch-protection.json`. Flagged only.
- **The drift loop is still unexplained.** Five Kustomizations re-applying the same object every
10 minutes (event counts ~2500) — untouched, cause unknown, still deserves its own branch.
@@ -0,0 +1,147 @@
---
date: 2026-09-30
branch: feat/sd-hardware-capability
title: ServiceDefinition gains a `hardware` capability field — `gpu: intel-quicksync` renders the /dev/dri wiring, and the enum is what keeps the mechanism swappable
---
## What changed
A new optional top-level ServiceDefinition property, `hardware`, with one optional member,
`gpu`, whose enum today has exactly one value: `intel-quicksync`. **No catalog entry uses it.**
This PR delivers the capability; consuming it (Plex) is separate work on top.
- `schemas/servicedefinition.schema.json` — adds `hardware` (object, `additionalProperties: false`,
`minProperties: 1`) with `gpu` (enum `["intel-quicksync"]`). Schema version bumped **1.7.0 →
1.8.0**, changelog entry prepended to the schema `description` per that file's own convention.
- `catalog/render.py` — three module-level constants (`DRI_VOLUME_NAME`, `DRI_DEVICE_DIR`,
`INTEL_RENDER_GID = 992`) and the emission in `r_deployment`.
- `schemas/renderer-contract.md` — new `### hardware` field-mapping section, plus cross-field
invariant **8** in that file's numbered list.
- `schemas/README.md` — its `**Schema version:**` line said `1.0.0`; it had been stale since 1.1.0.
Synced to 1.8.0 and annotated that the schema `description` is the changelog of record. Added one
bullet to "Cross-field rules" for the new enum. That was the whole of the staleness; nothing
ballooned.
**What the renderer emits** for `hardware.gpu: intel-quicksync`, on the app container / pod spec
only (never a sidecar):
1. a hostPath volume `dri` → `/dev/dri`, `type: DirectoryOrCreate`;
2. a read-write `volumeMount` of `dri` at `/dev/dri` on the app container (deliberately not
`readOnly` — VA-API opens the render node read-write);
3. `INTEL_RENDER_GID` **merged into** `podSecurityContext.supplementalGroups`, never clobbering
what the SD already declares.
It emits **no** `privileged: true` (Kyverno `restrict-host-namespaces` is Enforce on app namespaces
and it is not needed), **no** `nodeSelector` (placement is a separate concern with its own
justification) and **no** resource requests (there is no app-container resource render path).
**The design decision worth defending: the field is an enum of intent, not a device path.** The SD
says *what class of host device the workload needs*; the hostPath, the mount and the GID exist only
in `catalog/render.py`. Migrating to the Intel GPU device plugin (`gpu.intel.com/i915` resource
requests, no hostPath at all) is then a renderer change with zero catalog edits. Both the schema
`description` and the contract section say this explicitly, because the obvious "simplification" —
letting an SD name `/dev/dri` — would push a platform implementation detail into the operator-facing
contract and make that migration a catalog-wide edit.
**`DirectoryOrCreate` is load-bearing.** A node that comes back without the render device (rebuild,
BIOS change, driver regression, hardware swap) must degrade the workload to software transcoding,
not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so a failed
hostPath mount stalls the entire app tier for the sake of a feature that is not load-bearing.
`/dev` is tmpfs, so the empty directory the kubelet creates on such a node leaves nothing durable
behind. Do not "tighten" it to `Directory`. The reasoning is recorded at the emission site.
## What I verified
Measured, in this worktree, on this branch:
- **`python3 catalog/render.py --selftest` → `render.py selftest: PASS`, rc 0.** Three new cases
were added in the existing `check(...)` style: the positive (volume + mount + GID, and *no*
privileged/nodeSelector), the negative (an SD without `hardware` gains no volume, no volumeMount
and no `securityContext`), and the merge case (`supplementalGroups: [44]` + the field →
`[44, 992]`, with `fsGroup` preserved).
- **The self-tests actually bite.** I mutated the guard (`if hw_gpu == "intel-quicksync":` → `if
True:`) in a scratch copy and re-ran: rc 1 with three named failures, including "an SD without
`hardware` must gain no dri volume, no mount and no securityContext". A test that cannot fail is
not a test; this one can.
- **Re-render is a zero diff.**
`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation` then
`git diff --stat apps/` → **empty**, and `git status --porcelain apps/` → **empty** (so the two
not-onboarded by-products, `apps/wave-1/keycloak.yaml` and `apps/wave-0/vm-demo.yaml`, were
rewritten byte-identically and needed no revert). The change is inert until a service opts in.
- **Schema validity and catalog conformance.** `Draft202012Validator.check_schema` passes; all
**12** `catalog/*.yaml` validate with **0** errors. On the new field specifically: `gpu:
intel-quicksync` accepted; `gpu: /dev/dri` rejected (enum); `hardware: {gpu: …, path: …}` rejected
(`additionalProperties: false`); `hardware: {}` rejected (`minProperties: 1`).
- **Hand-checked the emitted YAML** by rendering a throwaway SD carrying `hardware`, a Longhorn
volume and its own `podSecurityContext: {runAsUser: 1000, supplementalGroups: [44]}`. Result:
`volumeMounts` = config then `{name: dri, mountPath: /dev/dri}`; `securityContext` =
`{runAsUser: 1000, supplementalGroups: [44, 992]}`; `volumes` = the PVC then
`{name: dri, hostPath: {path: /dev/dri, type: DirectoryOrCreate}}`. That toy SD was never written
into `catalog/`.
- **GATE: `rc 3` — UNDETERMINED, NOT green.** Run in the pinned toolbox from this worktree with
both CLAUDE.md §3 workarounds (`DEV_KUBECONFIG=/dev/null`, and an `ENGINE` shim bind-mounting the
main checkout's `.git` at its own absolute path). Diff base `origin/main @ 0fbf544d5e2f` … `HEAD @
9f0db1495cfd`; touched requirements `R-DNS-03 R-GITOPS-02 R-IDP-02`. Per stage:
| stage | result |
|---|---|
| spec-immutability | **SKIPPED** — sibling spec repo not at `/work/../home-network-k3s`. Expected from a worktree; tolerated by §3. |
| decisions-log | pass |
| lint | pass |
| l0 | pass |
| render-golden | pass — *"all 10 onboarded service golden(s) reproduce from the canonical render"* |
| l1 | pass — 3 offline checks, `0 failure(s), 0 skip(s), 0 unwired`; coverage 51/51 requirements scaffolded |
| approval-policy | pass (`event='none'`, not a pull_request event — N/A locally; it will evaluate for real on the PR) |
| gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset. Expected; branch protection and runner registration are therefore UNVERIFIED by this run. |
| conformance (L2/L3) | **SKIPPED** — `CLUSTER_AVAILABLE!=true`. Phase-0 standing skip. |
The gate's own verdict: *"Stages that did not run: spec-immutability gitea-settings"*. **No stage
failed, and no stage skipped for a reason caused by this change.** rc 3 is reported as rc 3; I am
not calling this green.
- **Blast radius of the schema change:** `hardware` is absent from every catalog entry, so
`apply_defaults` never materializes it and `s.get("hardware")` is `None` on all 12 — which is what
the zero diff above demonstrates rather than asserts.
- **Touched no sensitive path.** Changes are confined to `schemas/`, `catalog/`, `decisions/`; per
`infra/gitea/approval-policy.json` none of those are sensitive. `conformance/**` and `clusters/**`
are untouched. I did not run `--emit-waves` (it writes `clusters/base/`, and the set of files
under `apps/` did not change).
## What I did NOT verify
- **Whether a non-privileged pod can actually `open()` this hostPath char device at all. This is
the largest technical risk in the workstream and nobody has tested it.** `supplementalGroups` gets
the DAC permission bits right, but containerd installs a **device cgroup** allow-list per
container, and whether `/dev/dri/renderD128` reaching the container as a bind-mounted device node
is also permitted by that cgroup on this k3s/containerd version is **unmeasured**. If it is not,
the mount will succeed, the group will be present, and the `open()` will still fail `EPERM` — and
the design would need the device plugin (which sets the cgroup entry) rather than a hostPath. The
first real consumer proves or disproves this; nothing in this PR does.
- **That a missing device degrades silently — and nothing will tell you.** This is the accepted cost
of `DirectoryOrCreate`. The workload will software-transcode and **no signal reports it**: media
apps in this catalog declare no `metricsEndpoint`, so no ServiceMonitor exists, and there is no
metric to alert on. I deliberately did **not** propose an alert, because an alert that cannot fire
is worse than none — see the 2026-09-13 correction in `renderer-contract.md` §`observability`,
where twelve monitors discovered zero targets. The only proof of hardware transcode is an operator
seeing it in the application's own dashboard.
- **Whether GID 992 survives a node rebuild.** It was measured on all three prod nodes on
2026-09-30 and was identical on each, but `render` is a *systemd dynamic* group allocation, not a
Debian/LSB static GID. A reinstall that creates system groups in a different order can move it,
and nothing in this repo would notice. Recorded in the constant's comment; re-measure `getent
group render` on every node after any rebuild.
- **Whether `supplementalGroups` is the load-bearing mechanism or belt-and-braces.** A concurrent
agent was checking whether the pms-docker image's `45-plex-hw-transcode-and-connected-tuner` init
hook resolves the render group dynamically at startup. **No finding reached me**, so the constant's
comment describes the GID and its provenance without claiming which mechanism does the work.
Update that comment if the answer arrives.
- **Anything on the live cluster.** I ran no `kubectl`. The node facts above (device present,
`root:render` 0660, GID 992, `i915` bound, no device plugin, no `gpu.intel.com/*` advertised) and
the Kyverno posture (`restrict-host-namespaces` Enforce; no hostPath or `supplementalGroups`
restriction; no PSA labels on app namespaces) are **prior recon handed to me, not re-measured by
me**. Per CLAUDE.md §2b, treat them as second-hand until re-queried.
- **That CI exercises the self-tests. It does not.** `git grep -- --selftest ci/` returns nothing —
no gate stage invokes the renderer's self-test. They are the executable spec and house style, and
I ran them by hand, but a future regression in this code path **will not be caught by the gate**.
Wiring `--selftest` into a gate stage is a real gap and is not part of this PR (`ci/` is a
sensitive path and would drag in a human reviewer for a change unrelated to this one).
- **Runtime behaviour of any kind.** The gate proves the L0/L1 offline subset only;
`vars.CLUSTER_AVAILABLE` is unset, so L2/L3 conformance does not run in CI.
+33 -1
View File
@@ -839,6 +839,14 @@ spec:
# that has never applied anything — or a GitRepository with no artifact yet — publishes the # that has never applied anything — or a GitRepository with no artifact yet — publishes the
# series with no `revision` label. Without the matcher those group together under the empty # series with no `revision` label. Without the matcher those group together under the empty
# string and compare EQUAL, i.e. a never-applied Kustomization would read as converged. # string and compare EQUAL, i.e. a never-applied Kustomization would read as converged.
#
# A THIRD THING THAT MUST NOT REGRESS — THE SEVERITY LABEL. These alerts are delivered
# because of the labels they do NOT carry as much as the ones they do. Do not "tidy" this
# pair by giving FluxRolloutInProgress a `severity`, and never `severity: info`: the chart's
# InfoInhibitor rule plus its `severity="info"` inhibit_rule suppress an info alert whenever
# it is the only thing firing in its namespace, which is every healthy deploy. Measured on
# prod 2026-09-30 — the alert fired correctly for 12 Kustomizations, resolved correctly, and
# was inhibited in between. The full reasoning is on the `notification: deploy` label below.
# =================================================================================== # ===================================================================================
- alert: FluxRolloutInProgress - alert: FluxRolloutInProgress
expr: | expr: |
@@ -857,7 +865,31 @@ spec:
# `flux reconcile`, a change touching one Kustomization), and a deploy notification that # `flux reconcile`, a change touching one Kustomization), and a deploy notification that
# only sometimes arrives is worse than no notification at all. # only sometimes arrives is worse than no notification at all.
labels: labels:
severity: info # -> the email-deploy Alertmanager route; subject "[FLUX] applying/applied" # ROUTED BY `notification`, AND DELIBERATELY CARRYING NO `severity` (fixed 2026-09-30).
# This alert originally used `severity: info`. That was WRONG, and wrong in the silent
# direction — measured on prod within minutes of the first real deploy.
# kube-prometheus-stack ships `general.rules` -> InfoInhibitor:
# group by (namespace) (ALERTS{severity="info"} == 1)
# unless on (namespace) group by (namespace)
# (ALERTS{alertname!="InfoInhibitor", alertstate="firing", severity=~"warning|critical"} == 1)
# which fires for any namespace that has a firing info alert AND NOTHING ELSE WRONG in
# it; paired with the chart's inhibit_rule
# source_matchers: [alertname="InfoInhibitor"]
# target_matchers: [severity="info"]
# equal: [namespace]
# the effect is that an info alert is suppressed EXACTLY WHEN it is the only thing
# happening in its namespace. For a deploy notification that is the only case that
# matters: on a healthy rollout of a healthy cluster, flux-system has no warning or
# critical alert, so InfoInhibitor fired and this alert never reached the receiver.
# Verified live: InfoInhibitor went active 2026-09-30T04:58:48Z, the same moment this
# alert began firing for 12 Kustomizations.
# Upstream's intent is explicit — `severity: info` means "only worth showing next to a
# real problem". A deploy notice is not that, so it must not claim that severity.
# `notification` is a label no inhibit_rule in the chart targets (all three match on
# severity values), and omitting `severity` also stops InfoInhibitor firing for
# flux-system at all. Route: platform/monitoring/kube-prometheus-stack.yaml matches
# `notification = "deploy"` -> the email-deploy receiver, subject "[FLUX] applying|applied".
notification: deploy
annotations: annotations:
summary: "Flux is applying a new revision — {{ $labels.name }} is behind main" summary: "Flux is applying a new revision — {{ $labels.name }} is behind main"
description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete." description: "Kustomization {{ $labels.namespace }}/{{ $labels.name }} has applied {{ $labels.revision }}, which is not the revision the flux-system GitRepository has fetched from main. THIS MAIL MEANS A GIT CHANGE STARTED ROLLING OUT; the RESOLVED mail for this same alert means every Kustomization reached the new revision and the rollout is complete."
+13 -5
View File
@@ -139,11 +139,19 @@ spec:
# #
# A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a # A DEDICATED ROUTE, because the default receiver is `email-warning`: without it a
# routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a # routine deploy arrives subject-lined "[WARNING]" and becomes indistinguishable from a
# real platform fault, which is how alert mail stops being read at all. `severity: info` # real platform fault, which is how alert mail stops being read at all.
# is used by NO other rule in this repo (grepped 2026-09-29: zero matches), so this #
# route claims exactly the deploy notices and nothing else. Keep it that way — giving a # MATCHES `notification`, NOT `severity` (corrected 2026-09-30). This route first read
# genuine fault severity: info would silently route it here and re-label it a deploy. # `severity = "info"`, which routed correctly and then had the mail SUPPRESSED before it
- matchers: ['severity = "info"'] # was sent: the chart's own InfoInhibitor rule fires for any namespace holding a firing
# info alert with no warning/critical beside it, and its inhibit_rule
# (source alertname="InfoInhibitor" -> target severity="info", equal [namespace]) drops
# exactly that alert. A healthy deploy in a healthy flux-system is precisely that case,
# so the notification was inhibited on every occasion it was wanted. Measured live on
# prod 2026-09-30: InfoInhibitor went active at 04:58:48Z as the alert began firing.
# Routing on a non-severity label sidesteps all three of the chart's inhibit_rules,
# which match on severity values only. See the label's own comment in alerts.yaml.
- matchers: ['notification = "deploy"']
receiver: email-deploy receiver: email-deploy
# One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging # One mail per ROLLOUT, not per Kustomization: the alert fires once per lagging
# Kustomization (16 possible), and grouping on alertname alone collapses them into a # Kustomization (16 possible), and grouping on alertname alone collapses them into a
@@ -89,7 +89,7 @@ tests:
alertname: FluxRolloutInProgress alertname: FluxRolloutInProgress
exp_alerts: exp_alerts:
- exp_labels: - exp_labels:
severity: info notification: deploy
job: kube-state-metrics job: kube-state-metrics
namespace: flux-system namespace: flux-system
name: platform name: platform
@@ -148,7 +148,7 @@ tests:
alertname: FluxRolloutInProgress alertname: FluxRolloutInProgress
exp_alerts: exp_alerts:
- exp_labels: - exp_labels:
severity: info notification: deploy
job: kube-state-metrics job: kube-state-metrics
namespace: flux-system namespace: flux-system
name: platform name: platform
@@ -229,7 +229,7 @@ tests:
alertname: FluxRolloutInProgress alertname: FluxRolloutInProgress
exp_alerts: exp_alerts:
- exp_labels: - exp_labels:
severity: info notification: deploy
job: kube-state-metrics job: kube-state-metrics
namespace: flux-system namespace: flux-system
name: conformance name: conformance
+4 -1
View File
@@ -15,7 +15,7 @@ docs/13 exactly. Do not diverge from it here — escalate spec problems, don't f
| `servicedefinition.schema.json` | Formal JSON Schema (draft 2020-12) for a ServiceDefinition. Enforces enums, required fields, patterns, and cross-field invariants. | | `servicedefinition.schema.json` | Formal JSON Schema (draft 2020-12) for a ServiceDefinition. Enforces enums, required fields, patterns, and cross-field invariants. |
| `renderer-contract.md` | Authoritative field → rendered-resource mapping, with requirement IDs. The contract other workstreams consume. | | `renderer-contract.md` | Authoritative field → rendered-resource mapping, with requirement IDs. The contract other workstreams consume. |
**Schema version:** 1.0.0 (in the schema `description`). Treat any change as an interface change: **Schema version:** 1.8.0 (in the schema `description`, which carries the full per-version changelog — this line had said `1.0.0` since 1.1.0 and is now synced). Treat any change as an interface change:
propose on the task board, update dependent renderers/tests in the same change, never silently break a propose on the task board, update dependent renderers/tests in the same change, never silently break a
consumer (coordination-protocol §2). consumer (coordination-protocol §2).
@@ -35,6 +35,9 @@ consumer (coordination-protocol §2).
- `ingressRoutes[].host` must match `<label>.home.keithjackson.dev`. - `ingressRoutes[].host` must match `<label>.home.keithjackson.dev`.
- `storage[].class ∈ {longhorn, bulk}`; `exposure ∈ {internal, public}`; `egress ∈ {internet, vpn}`; - `storage[].class ∈ {longhorn, bulk}`; `exposure ∈ {internal, public}`; `egress ∈ {internet, vpn}`;
`auth.mode ∈ {oidc, forward-auth, native, none}`. `auth.mode ∈ {oidc, forward-auth, native, none}`.
- `hardware.gpu ∈ {intel-quicksync}` — a host **capability class**, never a device path. The enum
is the contract: the mechanism that satisfies it lives only in `catalog/render.py` (see
`renderer-contract.md` §`hardware`).
- `additionalProperties: false` everywhere — unknown keys are rejected (catches typos in review/CI). - `additionalProperties: false` everywhere — unknown keys are rejected (catches typos in review/CI).
## Validate examples against the schema — L1, no cluster (R-TEST-02) ## Validate examples against the schema — L1, no cluster (R-TEST-02)
+44
View File
@@ -160,6 +160,45 @@ platform must render, and cites the requirement each mapping satisfies. Changing
- **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication), - **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication),
**R-STATE-04** (bulk tier). (docs/04) **R-STATE-04** (bulk tier). (docs/04)
### `hardware` (`gpu`) — host device capability
- **The field is INTENT, never MECHANISM.** `hardware.gpu` is an **enum of capability classes**
(today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
ServiceDefinition says *what class of host device this workload needs*; how the platform hands it
over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw
`/dev/...` path.** The entire payoff is that migrating from the hostPath mechanism below to the
Intel GPU device plugin (`resources.limits: gpu.intel.com/i915`) is a renderer change with **zero**
catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely
different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one.
- `gpu: intel-quicksync` → on the **app container / pod spec only** (never a sidecar):
1. a **hostPath volume** `dri` → `/dev/dri`, **`type: DirectoryOrCreate`**;
2. a **read-write volumeMount** of `dri` at `/dev/dri` on the app container (VA-API opens the
render node read-write, so it is deliberately not `readOnly`);
3. the host `render` **GID merged into `podSecurityContext.supplementalGroups`** — merged, never
clobbered, so an SD that declares its own supplemental groups keeps all of them and gains one.
`/dev/dri/renderD128` is `root:render` mode `0660`, so a non-root container needs the group.
The GID is the single constant `INTEL_RENDER_GID` in `catalog/render.py` (992, **measured
2026-09-30 on all three prod nodes**; it is a *systemd dynamic* allocation, not a static GID,
so a node rebuild can move it — re-measure `getent group render` before assuming).
- **`DirectoryOrCreate` is load-bearing — do not tighten it to `Directory`.** A node that comes back
without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the
workload to **software transcoding**, not fail the mount. `apps-wave-0` is `wait: true` and
`apps-wave-1` `dependsOn` it, so a failed hostPath mount stalls the **whole app tier** for the sake
of a feature that is not load-bearing. `/dev` is tmpfs, so the empty directory the kubelet creates
on such a node leaves nothing durable behind.
- **The cost of that choice, stated plainly:** a missing device is **silent**. The workload
software-transcodes and nothing reports it. There is no metric to alert on — media apps here
render no `metricsEndpoint`, so no ServiceMonitor exists — and an alert that cannot fire is worse
than none (see §`observability`'s 2026-09-13 correction). The only proof of hardware transcode is
an operator observing it in the application's own dashboard.
- **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno
`restrict-host-namespaces` is Enforce on app namespaces and would reject it, and the render node
does not need it); no `nodeSelector` (placement is a separate concern with its own justification —
this field is about the device, not about where the pod lands); no resource requests/limits (there
is no app-container resource render path, and inventing one here would be scope creep).
- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical.
Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`.
- **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need).
### `ingressRoutes[]` (`host` / `port`) — HTTPRoute + Certificate + DNS ### `ingressRoutes[]` (`host` / `port`) — HTTPRoute + Certificate + DNS
- Each entry renders an **HTTPRoute** attaching `host` → the service `port` on the shared Gateway. - Each entry renders an **HTTPRoute** attaching `host` → the service `port` on the shared Gateway.
- TLS: the Gateway terminates with the cert-manager **wildcard `*.home.keithjackson.dev`** certificate - TLS: the Gateway terminates with the cert-manager **wildcard `*.home.keithjackson.dev`** certificate
@@ -286,6 +325,11 @@ platform must render, and cites the requirement each mapping satisfies. Changing
`external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it. `external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it.
Derived from `exposure` alone; enforced on every gate run by Derived from `exposure` alone; enforced on every gate run by
`conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03). `conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03).
8. **`hardware` is inert until opted into, and its mechanism is renderer-local.** An SD with no
`hardware` key renders no volume, no volumeMount and no `securityContext` — proven by
`render.py --selftest`. The `/dev/dri` hostPath, the `DirectoryOrCreate` type and
`INTEL_RENDER_GID` appear in `catalog/render.py` and nowhere else, so the capability's
implementation can be replaced without touching a single ServiceDefinition.
## Requirement traceability (contract-level) ## Requirement traceability (contract-level)
Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02 Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02
+14 -1
View File
@@ -2,7 +2,7 @@
"$schema": "https://json-schema.org/draft/2020-12/schema", "$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json", "$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
"title": "ServiceDefinition", "title": "ServiceDefinition",
"description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 / R-IDP-02).", "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with a /dev/dri hostPath volume (`DirectoryOrCreate`), a read-write /dev/dri volumeMount on the app container, and the host `render` GID merged into podSecurityContext.supplementalGroups, and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 / R-IDP-02).",
"type": "object", "type": "object",
"additionalProperties": false, "additionalProperties": false,
"required": ["name", "source"], "required": ["name", "source"],
@@ -139,6 +139,19 @@
"supplementalGroups": { "type": "array", "items": { "type": "integer" }, "description": "Additional GIDs added to the first process in each container." } "supplementalGroups": { "type": "array", "items": { "type": "integer" }, "description": "Additional GIDs added to the first process in each container." }
} }
}, },
"hardware": {
"type": "object",
"description": "Optional host HARDWARE CAPABILITY the workload needs, for the container kind (docs/13). This field declares INTENT (`what device class this service needs`), never MECHANISM (`which host path to bind-mount`, `which extended resource to request`). That separation is the whole point of the enum: the platform currently satisfies `intel-quicksync` with a /dev/dri hostPath volume + the host's `render` GID in supplementalGroups, but a later migration to the Intel GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) must be a catalog/render.py change and NOTHING else - no ServiceDefinition may need editing, and no reviewer may have to re-derive per-service device wiring. Do NOT 'simplify' this into a raw device path, a hostPath, or a resource request; that would push a platform implementation detail into the operator-facing contract and make the migration a catalog-wide edit. Absent => no device wiring at all (existing apps render byte-identical). Ignored for kind=vm (a VM gets devices through KubeVirt's own domain.devices).",
"additionalProperties": false,
"minProperties": 1,
"properties": {
"gpu": {
"type": "string",
"description": "GPU capability class this service needs. `intel-quicksync` = the host's Intel Quick Sync Video render device (the iGPU's /dev/dri/renderD* node) for hardware video transcode/decode. Enum, not a free-form device path, so the satisfying mechanism stays swappable (see the parent description). Extend the enum when a genuinely different capability class appears (e.g. nvidia-nvenc), never to name a different path to the same one.",
"enum": ["intel-quicksync"]
}
}
},
"exposure": { "exposure": {
"type": "string", "type": "string",
"description": "internal = LAN-only (internal DNS + Gateway on LAN); public = additionally edge port-forwarded 80/443 (docs/09; R-DNS/R-NET).", "description": "internal = LAN-only (internal DNS + Gateway on LAN); public = additionally edge port-forwarded 80/443 (docs/09; R-DNS/R-NET).",