jakman85andClaude Opus 4.8
af4f6c07ae
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
platform/monitoring
Observability stack — kube-prometheus-stack (Prometheus + Alertmanager + Grafana) + Loki + Grafana Alloy.
- Spec: docs/10 — Observability & Alerting (named in docs/01; no dedicated ADR)
- Serves: R-OBS-01..04
- Owner: WS-OBS
What this stub is
Three Flux HelmReleases (kube-prometheus-stack, loki, alloy) with placeholder values. Retention and
PVC sizes are overlay knobs (${MONITORING_*}, R-ENV-02): short/small in dev, long/large in prod.
Persistent data uses the replicated longhorn StorageClass.
Contract (observability)
The "observability contract" (roadmap/workstreams.md): how a service declares metrics/SLIs/alerts
and gets dashboards. ServiceMonitors/dashboards are rendered from ServiceDefinitions by WS-APPS/WS-VM
against this stack. Grafana is part of the C2 UI composition (docs/12), behind Keycloak SSO
(ADR-0014).
TODO (later phases)
- Phase 4 (WS-OBS): Alertmanager → Gmail SMTP receiver (creds SOPS-encrypted, name only);
Loki retention via compactor (
retention_enabled: true— a period alone does nothing); Alloy log scrape config → Loki; platform-health alerts. Reconciliation errors + drift are alerting conditions (R-OBS-04). - Exit gate (Phase 4): an injected fault emails an alert; dashboards render (R-OBS-01/02/03/04).
- Grafana OIDC client via Keycloak (WS-SEC/WS-C2).