jakman85andClaude Opus 5
cf7d97e546
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.
THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.
FIXED IN THE GENERATOR, not in twelve rendered files:
* `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
service's own workload object —
`kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
`kube_statefulset_status_replicas_ready` for a StatefulSet, and
`absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
exporter and no new target. The `or absent(...)` half is load-bearing: the signal
DISAPPEARING now fires the alert instead of silencing it.
* schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
scrape together: the named Service port, the monitor endpoint, and the
monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
kube-state-metrics liveness.
* schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
would have made its `absent()` half fire FOREVER — the other failure mode this repo has
shipped.
* renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.
PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
* quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
keycloak-db StatefulSet ready in the same namespace).
* mutation-checked: six mutations, each turning its suite red — reverting plex to
`up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
inverting plex's `== 0` to `!= 0`. Control run green.
* live on prod with `promtool query instant`: all ten deployed app Deployments export
`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
`kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
— its rule is not applied either.
DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.
GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.
Also fixed, found on the way:
* platform/monitoring/tests/run.sh discovered the SUITE convention only inside
platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
was never run and still looked like coverage. Now discovered tree-wide; a misplaced
*.test.yaml fails loudly instead of vanishing.
* KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
* renderer-contract.md claimed the default-deny baseline carried a standing monitoring
scrape allow. It never did; the claim is corrected and the allow is now real, per declared
metrics endpoint and scoped to that one port.
Co-Authored-By: Claude Opus 5 <[email protected]>
platform/monitoring
Observability stack — kube-prometheus-stack (Prometheus + Alertmanager + Grafana) + Loki + Grafana Alloy + platform-health PrometheusRules.
- Spec: docs/10 — Observability & Alerting (named in docs/01; no dedicated ADR)
- Serves: R-OBS-01/02/03/04 (supports R-C2-02, R-SEC-03, R-STATE-03)
- Owner: WS-OBS
Contents
| File | What |
|---|---|
namespace.yaml |
monitoring namespace |
helmrepositories.yaml |
prometheus-community + grafana Flux HelmRepositorys |
kube-prometheus-stack.yaml |
HelmRelease — Prometheus, Alertmanager (→ SMTP email), Grafana, kube-state-metrics, node-exporter |
loki.yaml |
HelmRelease — Loki single-binary, filesystem, compactor-enforced retention |
alloy.yaml |
HelmRelease — Grafana Alloy DaemonSet, tails pod logs → Loki |
alerts.yaml |
Platform-health PrometheusRules (R-OBS-04) |
tests/ |
promtool test rules unit tests over the committed PrometheusRules — each rule pinned quiet on the good state and firing on the bad one (tests/run.sh — the repo-wide harness, it also runs other components' companion *.promtool-test.yaml; enforced by the L0 check alert-rules-unit-tested) |
kustomization.yaml |
builds all of the above |
Requirement mapping
- R-OBS-01 (what is monitored): Prometheus + node-exporter + kube-state-metrics for metrics;
Alloy → Loki for logs.
serviceMonitorSelectorNilUsesHelmValues: falseso ServiceMonitors from every namespace (and those rendered from ServiceDefinitions, docs/13) are scraped by default. - R-OBS-02 (alert delivery): Alertmanager routes by
severity(criticalvswarning) to email. SMTP creds are a SOPS secret referenced by name only (see below). - R-OBS-03 (dashboards): Grafana, with the dashboard/datasource sidecar watching
grafana_dashboard-labeled ConfigMaps across all namespaces (per-service dashboards land here). Grafana is part of the C2 UI composition (docs/12), later behind Keycloak SSO (ADR-0014). - R-OBS-04 (platform health):
alerts.yaml— first-class alerts for nodes, Longhorn, Flux, cert-manager, networking/VPN/Hubble, Velero backups, Keycloak/IdP, and the monitor's own capacity. Splitcritical/warningper the docs/10 severity model.
SMTP secret (operator hand-off)
Alertmanager reads the SMTP password from a mounted SOPS-encrypted Secret — referenced by name, never inlined. The operator must provide (out of band, SOPS-encrypted per docs/07):
kind: Secret
metadata:
name: alertmanager-smtp # <-- referenced in kube-prometheus-stack.yaml (alertmanagerSpec.secrets)
namespace: monitoring
stringData:
smtp-password: "<app password>" # mounted at /etc/alertmanager/secrets/alertmanager-smtp/smtp-password
The non-secret SMTP fields (smtp_smarthost, smtp_from, smtp_auth_username, and the
receiver to: address) are REPLACE-ME placeholders in kube-prometheus-stack.yaml for the
operator to fill (same convention as velero/). Grafana's admin bootstrap likewise references a
SOPS secret by name — grafana-admin-credentials (keys admin-user / admin-password).
Dev trims (the ~22 GiB RAM ceiling)
This subtree is authored to stay light on the single dev workstation:
- Loki
chunksCache/resultsCacheDISABLED — the grafana/loki chart's default memcached caches request ~8-9 GiB each; leaving them on would blow the budget by itself. Also disabled: the nginxgateway,lokiCanary, andtesthooks. Loki runs SingleBinary (one pod). - Single replicas everywhere (Prometheus, Alertmanager, Grafana, Loki).
- Modest requests/limits on every component (see values). Ephemeral (emptyDir) state for Alertmanager and Grafana in dev.
- k3s control-plane scrape jobs disabled (
kubeControllerManager/kubeScheduler/kubeEtcd/kubeProxy) — embedded in k3s, not separately scrapable; keeps targets green and RAM down.
Approximate steady-state memory requests (what the scheduler reserves):
| Component | Req | Notes |
|---|---|---|
| Prometheus | 384Mi | limit 1Gi |
| Alertmanager | 32Mi | |
| Grafana | 128Mi | |
| prometheus-operator | 96Mi | |
| kube-state-metrics | 64Mi | |
| node-exporter ×3 | 96Mi | 32Mi/node DaemonSet |
| Loki (single binary) | 192Mi | limit 512Mi |
| Alloy ×3 | 288Mi | 96Mi/node DaemonSet |
| Total requests | ~1.3 GiB | working set typically ~1.5-2.5 GiB; limit ceiling ~3 GiB |
Comfortably within the 22 GiB box alongside the rest of the platform.
Environment knobs (R-ENV-02)
Only three substitution vars touch this subtree; the dev overlay (clusters/dev) patches
cluster-vars to dev values:
| Var | base | dev |
|---|---|---|
${MONITORING_PROMETHEUS_RETENTION} |
7d |
3d |
${MONITORING_PROMETHEUS_PVC_SIZE} |
10Gi |
8Gi |
${MONITORING_PROMETHEUS_RETENTION_SIZE} |
8GiB |
5GiB |
${MONITORING_LOKI_PVC_SIZE} |
10Gi |
5Gi |
MONITORING_PROMETHEUS_RETENTION_SIZE is a size-based retention cap (Prometheus *iB units,
distinct from the K8s Gi PVC knob) and must stay below MONITORING_PROMETHEUS_PVC_SIZE. Time-only
retention prunes nothing until the age window elapses, so a young/busy cluster fills /prometheus first;
a full disk wedges every WAL commit (scrapes still 200 but samples are dropped, the head freezes) and
makes every absent()/==0 platform alert fire falsely. The size cap drops the oldest blocks before the
disk fills. (prod overlay: PVC 50Gi, cap 45GiB.)
PVCs use the replicated longhorn StorageClass (ADR-0003).
Notes / known gaps for bring-up
- Loki log retention has no dedicated knob. Only the three vars above exist. Loki's
retention_periodis hardcoded to72h(compactor-enforced) to mirror dev metrics retention. Promote to acluster-varsknob (inclusters/, out of this subtree) if per-env log retention is wanted. - Several R-OBS-04 rules depend on exporters/ServiceMonitors owned by other workstreams (Flux,
cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble). Rules are authored to fire once those
series arrive; metric names marked
TODO(WS-*)inalerts.yaml(notably the egress VPN metric) need confirming against the deployed exporter. - Hubble network-flow visibility itself is delivered by
platform/cilium(docs/03); this subtree only consumes its metrics for denied-flow alerts.
Validate (no deploy)
kubectl kustomize platform/monitoring
Author-only — nothing here is applied to a cluster; Flux reconciles it at deploy time.