# 10 — Observability & Alerting

Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**.

## Stack

- **kube-prometheus-stack** — Prometheus (metrics), Alertmanager (routing/notify), Grafana
  (dashboards), and the standard Kubernetes exporters/rules.
- **Loki** — log aggregation.
- **Grafana Alloy** — ships node/pod/VM logs to Loki.
- **Hubble** — network flow visibility from Cilium ([03](03-networking-isolation-and-egress.md)).
- **Alertmanager → SMTP** — issues reported by **email** (R-OBS-02).
- **Grafana** — dashboards for services and platform health (R-OBS-03).

## What is monitored (R-OBS-01/04)

Application services:
- Up/health, resource use, request/error signals where exposed, and per-service SLIs declared in the
  ServiceDefinition (a service can declare the alerts it cares about).
- Each forward-auth gate ([15](15-identity-and-access.md)) is scraped in its own right, and the
  platform alerts on a gate that nothing observes: an auth failure otherwise looks like a healthy
  response.

Platform health (R-OBS-04) — first-class alerts:
- **Nodes**: down/unreachable, CPU, memory and I/O pressure, disk fill, disk latency.
- **Storage (Longhorn + NFS bulk tier)**: degraded/faulted volumes, replica rebuilds, capacity
  thresholds, PVC capacity, NFS server latency and client timeouts.
- **Databases (CloudNativePG)**: archiving failures, missed backups, primary placement.
- **GitOps (Flux)**: reconciliation failures, drift, suspended resources.
- **Automated updates**: image automation failures, stalled image policies, failed health gates,
  node reboot automation.
- **Certificates (cert-manager)**: upcoming expiry, issuance failures.
- **Networking**: VPN tunnel down, egress-gateway unhealthy, spikes in denied flows (Hubble),
  NetworkPolicy reconcile errors, public ingress tunnel down.
- **Backups (Velero/Longhorn)**: backup job failure, missed schedule, restore-test failures.
- **KubeVirt**: VM crash-loops / failed launches.
- **Identity (Keycloak)**: IdP down/unhealthy or its DB unavailable (SSO is a critical dependency —
  [15](15-identity-and-access.md)); certificate/login-endpoint failures; a failing or stale
  identity-config reconciler.
- **The monitor itself**: monitoring volume capacity, missing scrape targets.

Coverage against this list is tracked in [known gaps](16-known-gaps.md).

## Alert delivery (R-OBS-02)

- Alertmanager routes by severity to **email** (SMTP creds are SOPS-encrypted, [07](07-security-and-secrets.md)).
- Severity model: `critical` (page-worthy: data risk, outage, VPN leak potential, cert about to
  expire) vs `warning` (degraded, capacity, needs attention). Both go to email; only routing/subject
  differs.
- **Deploy notices** are a separate, non-severity notification class, also by email: one message
  when Flux starts applying a revision and one when every Kustomization has reached it, so a routine
  deploy never reads as a platform fault.
- Alerts include enough context (runbook link, affected object) to act without spelunking. Every
  alert's runbook link resolves to a runbook served under `/runbooks` on the apex, and every rule has
  a unit test; the merge gate checks both.

## Dashboards (R-OBS-03)

Grafana dashboards, surfaced in the C2 UI ([12](12-command-center-ui.md)):
- **Platform overview**: nodes, cluster capacity, Flux status, cert expiry, backup status.
- **Storage**: Longhorn volume health, capacity, replica state.
- **Network**: Hubble flows, egress/VPN health, denied-flow trends.
- **Per-service**: one row/dashboard per catalog service with its declared SLIs.

## Observability as onboarding output

The ServiceDefinition ([13](13-service-catalog.md)) declares observability expectations; the
platform renders **ServiceMonitors/PodMonitors** for services that declare a metrics endpoint, and
dashboards and alert rules from its declared SLIs. Log collection needs no per-service rendering:
the log agent discovers every pod. A newly onboarded service is monitored **by default**, not as a
follow-up task (part of the golden path, [01](01-architecture-overview.md)).

## Retention & sizing
- Metrics/log retention is set in the environment overlay. Metrics retention is capped by size as
  well as age, so a young or busy TSDB cannot fill its volume and blind the alerts that depend on
  it. Loki retention requires a compactor with retention enabled — do not rely on a period setting
  alone.
- Monitoring storage lives on Longhorn; ensure its own capacity is alerted on (don't let the monitor
  blind itself).

## Auditability tie-in (R-SEC-03)
Hubble + Prometheus + cert status provide the evidence for the isolation/egress/TLS/backup proofs
required by [07](07-security-and-secrets.md).

## Acceptance gates
- Every catalog service emits metrics + logs and has at least one meaningful alert that fires
  correctly in a fault test.
- Platform-health alerts (node down, Longhorn degraded, Flux failure, cert expiry, VPN down, backup
  failure) each fire and email correctly in an injected-fault test.
- Grafana dashboards render for platform and each service; the C2 UI links them.