Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors

Browse Source 
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.

THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.

FIXED IN THE GENERATOR, not in twelve rendered files:
  * `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
    service's own workload object —
    `kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
    `kube_statefulset_status_replicas_ready` for a StatefulSet, and
    `absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
    KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
    exporter and no new target. The `or absent(...)` half is load-bearing: the signal
    DISAPPEARING now fires the alert instead of silencing it.
  * schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
    ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
    scrape together: the named Service port, the monitor endpoint, and the
    monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
    dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
    kube-state-metrics liveness.
  * schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
    it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
    would have made its `absent()` half fire FOREVER — the other failure mode this repo has
    shipped.
  * renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
    must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
    must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.

PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
  * quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
    the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
    keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
    keycloak-db StatefulSet ready in the same namespace).
  * mutation-checked: six mutations, each turning its suite red — reverting plex to
    `up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
    capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
    inverting plex's `== 0` to `!= 0`. Control run green.
  * live on prod with `promtool query instant`: all ten deployed app Deployments export
    `kube_deployment_status_replicas_available`,
    `kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
    `kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
    SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
    twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
    — its rule is not applied either.

DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.

GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.

Also fixed, found on the way:
  * platform/monitoring/tests/run.sh discovered the SUITE convention only inside
    platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
    was never run and still looked like coverage. Now discovered tree-wide; a misplaced
    *.test.yaml fails loudly instead of vanishing.
  * KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
    apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
    gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
  * renderer-contract.md claimed the default-deny baseline carried a standing monitoring
    scrape allow. It never did; the claim is corrected and the allow is now real, per declared
    metrics endpoint and scoped to that one port.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-13 17:11:30 -05:00
1 parent 72f0ba4e1f
commit cf7d97e546
29 files changed
+1632 -428
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

catalog/render.py
+339 -15
View File
@@ -81,6 +81,27 @@ BULK_NFS_MOUNT_OPTIONS = ["nfsvers=4.1", "hard", "noatime", "timeo=600", "retran
GATEWAY_NAME = "shared"
GATEWAY_NS = "gateway"
KP_RELEASE = "kube-prometheus-stack" # ServiceMonitor/PrometheusRule discovery label
MONITORING_NS = "monitoring" # where Prometheus runs; the only admitted scrape source
# The kube-state-metrics series a workload's readiness IS (r_prometheusrule). kube-state-metrics
# is scraped by the bundled kube-prometheus-stack ServiceMonitor, i.e. these exist for EVERY
# workload in the cluster with no per-service exporter — which is the whole point of using them
# for `slis: [up]`. Each entry is (metric, label naming the object). VERIFIED on prod 2026-09-13:
# kube_deployment_status_replicas_available returned a series for all ten deployed app
# Deployments, and kube_statefulset_status_replicas_ready{namespace="keycloak",
# statefulset="keycloak"} returned 2.
LIVENESS_METRICS = {
"Deployment": ("kube_deployment_status_replicas_available", "deployment"),
"StatefulSet": ("kube_statefulset_status_replicas_ready", "statefulset"),
"DaemonSet": ("kube_daemonset_status_number_available", "daemonset"),
}
# A KubeVirt VMI is not a replica count: kubevirt_vmi_info carries the phase as a LABEL, so
# liveness is the EXISTENCE of a running-phase series. The value is lowercase `running` — a
# capital-R matcher selects nothing and is just as inert (the same trap that made
# kubevirt_vmi_phase_count{phase="Failed"} a no-op; verified on prod, where the one live VMI
# reports phase="running").
VMI_METRIC = "kubevirt_vmi_info"
VMI_RUNNING_PHASE = "running"
# egress classes and auth modes fixed by docs/03 / docs/13 + the Kyverno enums.
EGRESS_CLASSES = ("internet", "vpn")
@@ -135,6 +156,26 @@ def apply_defaults(sd: dict) -> dict:
obs = dict(s.get("observability") or {})
obs.setdefault("metrics", True)
obs.setdefault("slis", [])
# metricsEndpoint is NEVER defaulted (schema 1.6.0). `metrics: true` says the platform
# collects signals FOR this service; it cannot say the workload SERVES a Prometheus
# endpoint, because for most of this catalog it does not (see r_servicemonitor). When the
# endpoint IS declared, its name/path/interval carry defaults so the common case is
# `metricsEndpoint: {port: 9100}`.
ep = obs.get("metricsEndpoint")
if ep:
obs["metricsEndpoint"] = {
**{"name": "metrics", "path": "/metrics", "interval": "30s"},
**ep,
}
# livenessWorkload: WHICH Kubernetes object's readiness IS this service being up. Defaulted
# from `kind` — a container renders a Deployment named after the service, a vm renders a
# VirtualMachine whose VMI carries the same name — and overridable for the case where the
# object that actually runs on the cluster is not the one this SD renders (keycloak: the
# Keycloak operator runs a StatefulSet, so a Deployment-shaped rule would `absent()` forever).
lw = dict(obs.get("livenessWorkload") or {})
lw.setdefault("kind", "VirtualMachineInstance" if s["kind"] == "vm" else "Deployment")
lw.setdefault("name", s["name"])
obs["livenessWorkload"] = lw
s["observability"] = obs
# storage entry class default is longhorn (replicated tier); readOnly default is False
@@ -488,7 +529,12 @@ def r_deployment(s: dict) -> dict:
def r_service(s: dict):
ports = _service_ports(s)
if not ports:
# A declared metrics endpoint becomes a NAMED Service port, because that name is the only
# thing a ServiceMonitor endpoint can select on. Appended BELOW rather than inside
# _service_ports so it never leaks into r_gateway_ingress_policy's north-south allow list —
# the metrics port is for the monitoring namespace, not for the Gateway.
ep = s["observability"].get("metricsEndpoint")
if not ports and not ep:
return None
# forward-auth (Option B): the shared Gateway must reach the oauth2-proxy sidecar, so front it
# with a `authproxy` :4180 Service port (the HTTPRoute backendRef targets this). The app UI/API
@@ -498,6 +544,10 @@ def r_service(s: dict):
ports = ports + [
{"name": "authproxy", "port": OAUTH2_PROXY_PORT, "targetPort": OAUTH2_PROXY_PORT}
]
if ep:
ports = ports + [
{"name": ep["name"], "port": ep["port"], "targetPort": ep["port"]}
]
return {
"apiVersion": "v1",
"kind": "Service",
@@ -990,7 +1040,33 @@ def _auth_annotations(s: dict) -> dict:
def r_servicemonitor(s: dict):
if not s["observability"]["metrics"]:
"""ServiceMonitor — rendered ONLY when the SD says WHERE metrics are served.
THIS FUNCTION USED TO RENDER A MONITOR UNCONDITIONALLY FROM `metrics: true`, hardcoding
`endpoints: [{port: metrics}]`. No Service this renderer emits had ever defined a port of
that name, so all twelve monitors discovered ZERO targets: no `up{job="<app>"}` series
existed anywhere, `up{job="<app>"} == 0` could not fire, and `PrometheusTargetsMissing`
could not see the hole either, because a target that was never discovered has no `up` series
to be 0. Verified on prod 2026-09-13: `count by (job) (up)` listed 24 jobs and not one of
the twelve appeared; `up{job="plex"}` and `up{job="gitea"}` both returned nothing.
THE ROOT CAUSE WAS NOT A TYPO — it was that `metrics: true` cannot state a fact the workload
does not have. Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and
Nextcloud serve no Prometheus endpoint at all without a separate exporter; Gitea can serve
one but only once its app.ini enables it. So the port name was never the missing piece. A
monitor is now rendered only from `observability.metricsEndpoint`, which names the port the
workload actually serves — and that same declaration renders the Service port (r_service) and
the monitoring-namespace ingress allow (r_monitoring_ingress_policy), so the three halves of
a working scrape cannot be authored apart.
`metrics: true` still means something without it: logs ship to Loki via the node agent, and
liveness comes from kube-state-metrics (r_prometheusrule) — neither needs a per-service CR.
"""
obs = s["observability"]
if not obs["metrics"]:
return None
ep = obs.get("metricsEndpoint")
if not ep:
return None
return {
"apiVersion": "monitoring.coreos.com/v1",
@@ -1003,7 +1079,108 @@ def r_servicemonitor(s: dict):
"spec": {
"selector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
"namespaceSelector": {"matchNames": [s["name"]]},
"endpoints": [{"port": "metrics", "interval": "30s"}],
"endpoints": [
{"port": ep["name"], "path": ep["path"], "interval": ep["interval"]}
],
},
}
def r_monitoring_ingress_policy(s: dict):
"""Let the monitoring namespace reach a declared metrics port (R-NET-02, R-OBS-01).
renderer-contract.md has always said the default-deny baseline carries "a standing allow ...
for observability scraping from the monitoring namespace" — and no renderer output ever did.
Every app namespace ships `ingress: []`, so a scrape of an app metrics port would be DROPPED
by policy even with the Service port and the monitor endpoint in agreement. That is the same
trap Longhorn's own NetworkPolicies set (platform/longhorn/networkpolicy-metrics.yaml exists
for exactly this reason), and it is a THIRD independent way for a metrics SLI to be inert.
Rendered only alongside a declared metricsEndpoint, and it opens ONLY that port to ONLY the
monitoring namespace: an allow for a port nothing serves would be a standing hole for no
benefit.
"""
ep = s["observability"].get("metricsEndpoint")
if not ep or not s["observability"]["metrics"]:
return None
return {
"apiVersion": "cilium.io/v2",
"kind": "CiliumNetworkPolicy",
"metadata": {
"name": f"{s['name']}-allow-monitoring-scrape",
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": {
"endpointSelector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
"ingress": [
{
"fromEndpoints": [
{"matchLabels": {"k8s:io.kubernetes.pod.namespace": MONITORING_NS}}
],
"toPorts": [
{"ports": [{"port": str(ep["port"]), "protocol": "TCP"}]}
],
}
],
},
}
def _up_sli_rule(s: dict) -> dict:
"""`slis: [up]` -> a `<name>-down` alert against a signal THAT EXISTS on this cluster.
IT USED TO READ `up{job="<name>"} == 0`. That is the textbook expression and it was inert
for all twelve services in the catalog, because `up` only exists for a target Prometheus
actually DISCOVERED and the rendered ServiceMonitor discovered none (see r_servicemonitor).
Worse than a wrong threshold: an inert rule and a correct rule are indistinguishable on a
healthy cluster, and the generic safety net cannot see the difference either — a target that
was never discovered has no `up` series for PrometheusTargetsMissing to find at 0.
Rewriting the SLI against the app's own /metrics was NOT an option for most of this catalog:
Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export no
Prometheus metrics without a separate exporter sidecar per app. So the `up` SLI is expressed
against kube-state-metrics, which IS scraped and DOES carry a series for every workload — the
same idiom KeycloakDown in platform/monitoring/alerts.yaml has used since it was fixed.
The `or absent(...)` half is deliberate and load-bearing: it makes the DISAPPEARANCE of the
signal fire the alert instead of silencing it. A rule that can only fire on `== 0` is a rule
that goes quiet when the workload object (or kube-state-metrics) vanishes, which is precisely
how the old expression failed. It is also why this rule cannot be permanently firing where the
workload exists: the series is present and non-zero on a healthy cluster (verified on prod).
"""
lw = s["observability"]["livenessWorkload"]
kind, obj = lw["kind"], lw["name"]
ns = s["name"]
if kind == "VirtualMachineInstance":
sel = f'{VMI_METRIC}{{namespace="{ns}", name="{obj}", phase="{VMI_RUNNING_PHASE}"}}'
expr = f"absent({sel})"
desc = (
f"{s['name']}: no KubeVirt VMI in phase {VMI_RUNNING_PHASE} for 5m "
f"(SLI: up, via {VMI_METRIC})."
)
else:
try:
metric, label = LIVENESS_METRICS[kind]
except KeyError:
raise ValueError(
f"{s['name']}: observability.livenessWorkload.kind {kind!r} has no liveness "
f"metric; known kinds: {sorted(LIVENESS_METRICS)} + VirtualMachineInstance"
) from None
sel = f'{metric}{{namespace="{ns}", {label}="{obj}"}}'
expr = f"{sel} == 0 or absent({sel})"
desc = (
f"{s['name']}: its {kind} has no available/ready replica for 5m, or the signal is "
f"gone entirely (SLI: up, via {metric})."
)
return {
"alert": f"{s['name']}-down".replace("_", "-"),
"expr": expr,
"for": "5m",
"labels": {"severity": "critical", "service": s["name"]},
"annotations": {
"summary": f"{s['name']} is down",
"description": desc,
},
}
@@ -1016,18 +1193,7 @@ def r_prometheusrule(s: dict):
rules = []
for sli in slis:
if sli == "up":
rules.append(
{
"alert": f"{s['name']}-down".replace("_", "-"),
"expr": f'up{{job="{s["name"]}"}} == 0',
"for": "5m",
"labels": {"severity": "critical", "service": s["name"]},
"annotations": {
"summary": f"{s['name']} is down",
"description": f"{s['name']} has been unreachable for 5m (SLI: up).",
},
}
)
rules.append(_up_sli_rule(s))
else:
rules.append(
{
@@ -1386,6 +1552,16 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
)
)
mon_allow = r_monitoring_ingress_policy(s)
if mon_allow:
docs.append(
(
"CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): "
f"{mon_allow['metadata']['name']}",
mon_allow,
)
)
ie = r_internet_egress(s)
docs.append(
(
@@ -2315,6 +2491,154 @@ def _selftest() -> int:
"the escaped `$${...}` form must render (and reach the shell as `${...}`)",
)
# 9) observability (R-OBS-01): `metrics: true` alone renders NO ServiceMonitor, and the `up`
# SLI reads a signal that exists. This is the regression test for the defect that made all
# twelve `<app>-down` rules structurally inert on prod (see r_servicemonitor).
obs_sd = {
"name": "obsdemo",
"source": {"image": "a:1"},
"ingressRoutes": [{"host": "obsdemo.${BASE_DOMAIN}", "port": 8080}],
"auth": {"mode": "none"},
"observability": {"metrics": True, "slis": ["up"]},
}
obs_docs = render(obs_sd)
check(
not [d for d in obs_docs if d[1]["kind"] == "ServiceMonitor"],
"metrics: true with NO metricsEndpoint must render no ServiceMonitor — a monitor naming a "
"port the Service does not define discovers zero targets and makes the SLI inert",
)
pr = next(d[1] for d in obs_docs if d[1]["kind"] == "PrometheusRule")
up_rule = pr["spec"]["groups"][0]["rules"][0]
check(up_rule["alert"] == "obsdemo-down", "slis: [up] must render <name>-down")
check(
"up{job=" not in up_rule["expr"],
"the up SLI must NOT read up{job=...}: no repo-rendered scrape target produces that job, "
"so the rule can never fire (verified inert on prod for all twelve apps)",
)
check(
up_rule["expr"]
== 'kube_deployment_status_replicas_available{namespace="obsdemo", deployment="obsdemo"} '
'== 0 or absent(kube_deployment_status_replicas_available{namespace="obsdemo", '
'deployment="obsdemo"})',
"container up SLI must read kube_deployment_status_replicas_available with an absent() "
f"half; got: {up_rule['expr']}",
)
# 9a) livenessWorkload override: the object that RUNS is not always the one this SD renders.
sts_docs = render(
dict(
obs_sd,
name="obssts",
observability={
"metrics": True,
"slis": ["up"],
"livenessWorkload": {"kind": "StatefulSet", "name": "obssts"},
},
)
)
sts_expr = next(
d[1] for d in sts_docs if d[1]["kind"] == "PrometheusRule"
)["spec"]["groups"][0]["rules"][0]["expr"]
check(
"kube_statefulset_status_replicas_ready" in sts_expr,
f"livenessWorkload kind StatefulSet must read the statefulset series; got: {sts_expr}",
)
# 9b) vm kind: a VMI has no replica count, so liveness is a running-phase series EXISTING.
vm_pr = next(
d[1]
for d in render(
{
"name": "obsvm",
"kind": "vm",
"source": {"vmTemplate": "quay.io/x/y:1"},
"storage": [{"name": "root", "size": "6Gi"}],
"observability": {"metrics": True, "slis": ["up"]},
}
)
if d[1]["kind"] == "PrometheusRule"
)
check(
vm_pr["spec"]["groups"][0]["rules"][0]["expr"]
== 'absent(kubevirt_vmi_info{namespace="obsvm", name="obsvm", phase="running"})',
"vm up SLI must read kubevirt_vmi_info with the LOWERCASE running phase: "
f"{vm_pr['spec']['groups'][0]['rules'][0]['expr']}",
)
# 9c) metricsEndpoint renders ALL THREE halves of a working scrape, and they AGREE. Any one of
# them missing is an independent way for the scrape to silently discover/deliver nothing.
ep_docs = render(
dict(
obs_sd,
name="obsep",
observability={
"metrics": True,
"slis": ["up"],
"metricsEndpoint": {"port": 9100},
},
)
)
ep_svc = next(d[1] for d in ep_docs if d[1]["kind"] == "Service")
ep_sm = next(d[1] for d in ep_docs if d[1]["kind"] == "ServiceMonitor")
svc_port_names = {p["name"] for p in ep_svc["spec"]["ports"]}
ep_ports = {e["port"] for e in ep_sm["spec"]["endpoints"]}
check(
ep_ports <= svc_port_names,
f"ServiceMonitor endpoint port(s) {sorted(ep_ports)} must all exist on the Service "
f"{sorted(svc_port_names)} — this exact disagreement is what made twelve monitors inert",
)
check(
{"name": "metrics", "port": 9100, "targetPort": 9100} in ep_svc["spec"]["ports"],
"metricsEndpoint must render the named Service port it declares",
)
mon_cnp = [
d[1]
for d in ep_docs
if d[1]["kind"] == "CiliumNetworkPolicy"
and d[1]["metadata"]["name"] == "obsep-allow-monitoring-scrape"
]
check(
len(mon_cnp) == 1,
"a declared metricsEndpoint must render the monitoring-namespace ingress allow — the "
"namespace default-deny has `ingress: []` and would DROP the scrape without it",
)
if mon_cnp:
ing = mon_cnp[0]["spec"]["ingress"][0]
check(
ing["fromEndpoints"][0]["matchLabels"]["k8s:io.kubernetes.pod.namespace"]
== MONITORING_NS
and ing["toPorts"][0]["ports"] == [{"port": "9100", "protocol": "TCP"}],
"the scrape allow must admit ONLY the monitoring namespace to ONLY the metrics port",
)
gw_cnp = next(
d[1]
for d in ep_docs
if d[1]["kind"] == "CiliumNetworkPolicy"
and d[1]["metadata"]["name"] == "obsep-allow-gateway-ingress"
)
gw_ports = {p["port"] for p in gw_cnp["spec"]["ingress"][0]["toPorts"][0]["ports"]}
check(
"9100" not in gw_ports,
"the metrics port must NOT be opened to the Gateway (north-south); it is for the "
f"monitoring namespace only. Gateway allows: {sorted(gw_ports)}",
)
# A metricsEndpoint the renderer cannot express must be REFUSED, never rendered inert.
try:
render(
dict(
obs_sd,
name="obsbad",
observability={
"metrics": True,
"slis": ["up"],
"livenessWorkload": {"kind": "CronJob", "name": "obsbad"},
},
)
)
check(False, "an unknown livenessWorkload.kind must raise, not render a bogus expression")
except ValueError as exc:
check("CronJob" in str(exc), "the refusal must name the unsupported kind")
if failures:
for f in failures:
sys.stderr.write(f"SELFTEST FAIL: {f}\n")