# 10 — Observability & Alerting

Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**.

## Stack

- **kube-prometheus-stack** — Prometheus (metrics), Alertmanager (routing/notify), Grafana
  (dashboards), and the standard Kubernetes exporters/rules.
- **Loki** — log aggregation.
- **Grafana Alloy** (or Promtail-class agent) — ships node/pod/VM logs to Loki.
- **Hubble** — network flow visibility from Cilium ([03](03-networking-isolation-and-egress.md)).
- **Alertmanager → SMTP** — issues reported by **email** (R-OBS-02).
- **Grafana** — dashboards for services and platform health (R-OBS-03).

## What is monitored (R-OBS-01/04)

Application services:
- Up/health, resource use, request/error signals where exposed, and per-service SLIs declared in the
  ServiceDefinition (a service can declare the alerts it cares about).

Platform health (R-OBS-04) — first-class alerts:
- **Nodes**: down/unreachable, resource pressure, disk fill.
- **Storage (Longhorn)**: degraded/faulted volumes, replica rebuilds, capacity thresholds.
- **GitOps (Flux)**: reconciliation failures, drift, suspended resources.
- **Certificates (cert-manager)**: upcoming expiry, issuance failures.
- **Networking**: VPN tunnel down, egress-gateway unhealthy, spikes in denied flows (Hubble),
  NetworkPolicy reconcile errors.
- **Backups (Velero/Longhorn)**: backup job failure, missed schedule, restore-test failures.
- **KubeVirt**: VM crash-loops / failed launches.

## Alert delivery (R-OBS-02)

- Alertmanager routes by severity to **email** (SMTP creds are SOPS-encrypted, [07](07-security-and-secrets.md)).
- Severity model: `critical` (page-worthy: data risk, outage, VPN leak potential, cert about to
  expire) vs `warning` (degraded, capacity, needs attention). Both go to email; only routing/subject
  differs.
- Alerts include enough context (runbook link, affected object) to act without spelunking.

## Dashboards (R-OBS-03)

Grafana dashboards, surfaced in the C2 UI ([12](12-command-center-ui.md)):
- **Platform overview**: nodes, cluster capacity, Flux status, cert expiry, backup status.
- **Storage**: Longhorn volume health, capacity, replica state.
- **Network**: Hubble flows, egress/VPN health, denied-flow trends.
- **Per-service**: one row/dashboard per catalog service with its declared SLIs.

## Observability as onboarding output

The ServiceDefinition ([13](13-service-catalog.md)) declares observability expectations; the
platform renders **ServiceMonitors/PodMonitors**, log scraping, dashboards, and alert rules from it.
A newly onboarded service is monitored **by default**, not as a follow-up task (part of the golden
path, [01](01-architecture-overview.md)).

## Retention & sizing
- Metrics/log retention set per environment overlay (dev shorter, prod longer). Loki retention
  requires a compactor with retention enabled — do not rely on a period setting alone.
- Monitoring storage lives on Longhorn; ensure its own capacity is alerted on (don't let the monitor
  blind itself).

## Auditability tie-in (R-SEC-03)
Hubble + Prometheus + cert status provide the evidence for the isolation/egress/TLS/backup proofs
required by [07](07-security-and-secrets.md).

## Acceptance gates
- Every catalog service emits metrics + logs and has at least one meaningful alert that fires
  correctly in a fault test.
- Platform-health alerts (node down, Longhorn degraded, Flux failure, cert expiry, VPN down, backup
  failure) each fire and email correctly in an injected-fault test.
- Grafana dashboards render for platform and each service; the C2 UI links them.