fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors

Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.

THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.

FIXED IN THE GENERATOR, not in twelve rendered files:
  * `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
    service's own workload object —
    `kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
    `kube_statefulset_status_replicas_ready` for a StatefulSet, and
    `absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
    KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
    exporter and no new target. The `or absent(...)` half is load-bearing: the signal
    DISAPPEARING now fires the alert instead of silencing it.
  * schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
    ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
    scrape together: the named Service port, the monitor endpoint, and the
    monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
    dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
    kube-state-metrics liveness.
  * schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
    it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
    would have made its `absent()` half fire FOREVER — the other failure mode this repo has
    shipped.
  * renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
    must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
    must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.

PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
  * quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
    the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
    keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
    keycloak-db StatefulSet ready in the same namespace).
  * mutation-checked: six mutations, each turning its suite red — reverting plex to
    `up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
    capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
    inverting plex's `== 0` to `!= 0`. Control run green.
  * live on prod with `promtool query instant`: all ten deployed app Deployments export
    `kube_deployment_status_replicas_available`,
    `kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
    `kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
    SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
    twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
    — its rule is not applied either.

DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.

GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.

Also fixed, found on the way:
  * platform/monitoring/tests/run.sh discovered the SUITE convention only inside
    platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
    was never run and still looked like coverage. Now discovered tree-wide; a misplaced
    *.test.yaml fails loudly instead of vanishing.
  * KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
    apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
    gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
  * renderer-contract.md claimed the default-deny baseline carried a standing monitoring
    scrape allow. It never did; the claim is corrected and the allow is now real, per declared
    metrics endpoint and scoped to that one port.

Co-Authored-By: Claude Opus 5 <[email protected]>

schemas/renderer-contract.md

@@ -26,6 +26,8 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 | `update.channel` | `stable` | R-UPD-01 |
 | `update.autoRollbackOnFailedHealth` | `true` | R-UPD-03 |
 | `observability.metrics` | `true` | R-OBS-01 |
+| `observability.metricsEndpoint` | **absent** → the workload serves no Prometheus endpoint; no ServiceMonitor | R-OBS-01 |
+| `observability.livenessWorkload` | the object this SD renders (`Deployment` named after the service; `VirtualMachineInstance` for `kind: vm`) | R-OBS-01 |
 | backup of irreplaceable state | off-cluster (never `reacquirable`) | R-STATE-03 |
 
 ---
@@ -65,8 +67,11 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 
 ### `dependencies[]` (`to` / `port` / `protocol`) — east-west policy
 - Each namespace first gets the **default-deny baseline** (deny all ingress + egress) applied by
-  GitOps at namespace creation, with a standing allow only for **DNS to the in-cluster resolver** and
-  for **observability scraping** from the monitoring namespace.
+  GitOps at namespace creation, with a standing allow only for **DNS to the in-cluster resolver**.
+  **Observability scraping is NOT in the baseline** — it is an additive allow rendered per declared
+  `observability.metricsEndpoint` (see below), scoped to that one port. This paragraph used to claim
+  the baseline carried a standing monitoring allow; it never did, and an app metrics port would have
+  been dropped by policy even with the Service port and monitor endpoint in agreement.
 - Each `dependencies[]` entry renders one **CiliumNetworkPolicy** egress allow: this service → `to`
   service on `port`/`protocol`. Everything undeclared stays denied.
 - Empty `dependencies` ⇒ the service can reach **only DNS**.
@@ -183,11 +188,35 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 - **Requirements:** **R-UPD-01** (stable only), **R-UPD-02** (no human/AI authoring), **R-UPD-03**
   (health-gated auto-rollback). (docs/11, ADR-0009)
 
-### `observability` (`metrics` / `slis[]`) — ServiceMonitor + alerts
-- `metrics: true` → a **ServiceMonitor/PodMonitor** so Prometheus scrapes the service; logs are
-  shipped to Loki by the node/pod agent by default.
-- each entry in `slis[]` → at least one **alert rule** (e.g. `up` → service-down alert) routed by
-  Alertmanager to email; a per-service Grafana dashboard row.
+### `observability` (`metrics` / `slis[]` / `metricsEndpoint` / `livenessWorkload`) — ServiceMonitor + alerts
+- `metrics: true` → the platform **collects signals** for this service: logs to Loki via the node
+  agent, and kube-state-metrics readiness for the `up` SLI below. Both are platform-wide scrapes;
+  neither needs a per-service CR. **It does not assert that the workload serves a Prometheus
+  endpoint.** *(Corrected 2026-09-13. It used to render a ServiceMonitor unconditionally, hardcoded
+  to a Service port named `metrics`. No rendered Service ever defined such a port, so all twelve app
+  monitors discovered ZERO targets — verified on prod: 24 jobs in `up`, not one of the twelve — and
+  every `up{job=...}` SLI built on them was structurally inert. `metrics: true` could not have been
+  right: Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud serve no
+  Prometheus endpoint at all without a separate exporter.)*
+- `metricsEndpoint: {port, name?, path?, interval?}` → **the workload does serve metrics, here.**
+  Renders all three halves of a working scrape from the one declaration, because any one of them
+  missing makes the scrape silently yield nothing:
+  1. a **named Service port** (`name`, default `metrics`) on that `port`;
+  2. a **ServiceMonitor** endpoint selecting that port name, on `path` at `interval`;
+  3. a **CiliumNetworkPolicy** admitting the `monitoring` namespace to that port — the namespace
+     default-deny baseline is `ingress: []`, so without it the scrape is dropped by policy.
+  The metrics port is deliberately **not** added to the Gateway ingress allow: it is east-west for
+  Prometheus, never north-south.
+- each entry in `slis[]` → at least one **alert rule** routed by Alertmanager to email; a per-service
+  Grafana dashboard row. `up` → a `<name>-down` **critical** alert reading the readiness of
+  `livenessWorkload` from **kube-state-metrics** (`kube_deployment_status_replicas_available`,
+  `kube_statefulset_status_replicas_ready`, `kube_daemonset_status_number_available`) or, for a
+  `VirtualMachineInstance`, the existence of `kubevirt_vmi_info{phase="running"}`. Each rule carries
+  an `or absent(...)` half, so the signal DISAPPEARING fires the alert rather than silencing it.
+- `livenessWorkload: {kind, name}` → which object's readiness **is** this service. Defaults to the
+  object this SD renders; override it when the running workload is a different one (keycloak's server
+  is a StatefulSet owned by the Keycloak operator — a Deployment-shaped rule would `absent()` and
+  fire forever).
 - **Requirements:** **R-OBS-01** (per-service SLIs/alerts). Platform-health alerts (R-OBS-04) are
   rendered platform-wide, not per ServiceDefinition. (docs/10)
 

schemas/servicedefinition.schema.json

@@ -2,7 +2,7 @@
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
   "title": "ServiceDefinition",
-  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 / R-IDP-02).",
+  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 / R-IDP-02).",
   "type": "object",
   "additionalProperties": false,
   "required": ["name", "source"],
@@ -350,7 +350,7 @@
       "properties": {
         "metrics": {
           "type": "boolean",
-          "description": "Whether the platform scrapes metrics for this service.",
+          "description": "Whether the platform collects signals for this service (logs to Loki + kube-state-metrics-derived liveness). It does NOT assert that the workload serves a Prometheus endpoint — say that with metricsEndpoint.",
           "default": true
         },
         "slis": {
@@ -358,6 +358,55 @@
           "description": "Per-service alert intents (e.g. up). Each becomes at least one alert rule (R-OBS-01).",
           "items": { "type": "string", "minLength": 1 },
           "default": []
+        },
+        "metricsEndpoint": {
+          "type": "object",
+          "description": "WHERE this workload serves Prometheus metrics. Absent (the default for every catalog entry today) means it serves none, and NO ServiceMonitor is rendered — the renderer used to emit one naming port `metrics` regardless, which no rendered Service ever defined, so all twelve monitors discovered zero targets and every `up`-based SLI was structurally inert. When present it renders all three halves of a working scrape together: the named Service port, the ServiceMonitor endpoint, and the monitoring-namespace ingress allow that the default-deny baseline would otherwise drop.",
+          "additionalProperties": false,
+          "required": ["port"],
+          "properties": {
+            "port": {
+              "type": "integer",
+              "description": "Container/Service port the metrics endpoint listens on.",
+              "minimum": 1,
+              "maximum": 65535
+            },
+            "name": {
+              "type": "string",
+              "description": "Service port name the ServiceMonitor endpoint selects on.",
+              "minLength": 1,
+              "default": "metrics"
+            },
+            "path": {
+              "type": "string",
+              "description": "HTTP path serving the Prometheus exposition format.",
+              "minLength": 1,
+              "default": "/metrics"
+            },
+            "interval": {
+              "type": "string",
+              "description": "Scrape interval (Prometheus duration).",
+              "pattern": "^[0-9]+[smh]$",
+              "default": "30s"
+            }
+          }
+        },
+        "livenessWorkload": {
+          "type": "object",
+          "description": "WHICH Kubernetes object's readiness constitutes this service being up, for the `up` SLI. Defaults to the object this ServiceDefinition renders (Deployment named after the service for kind: container, VirtualMachineInstance for kind: vm). Override only when the object that actually runs on the cluster is a different one — keycloak's server is a StatefulSet owned by the Keycloak operator, and a Deployment-shaped rule would read an absent series and fire forever.",
+          "additionalProperties": false,
+          "properties": {
+            "kind": {
+              "type": "string",
+              "description": "Workload kind whose kube-state-metrics readiness series the SLI reads.",
+              "enum": ["Deployment", "StatefulSet", "DaemonSet", "VirtualMachineInstance"]
+            },
+            "name": {
+              "type": "string",
+              "description": "Object name (defaults to the service name).",
+              "minLength": 1
+            }
+          }
         }
       }
     },