jakman85andClaude Opus 4.8
e8c555cfeb
Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision & requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap (phases + workstreams), and AI-agent coordination (roles, protocol, definition-of-done, task board). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
70 lines
3.5 KiB
Markdown
70 lines
3.5 KiB
Markdown
# 10 — Observability & Alerting
|
|
|
|
Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**.
|
|
|
|
## Stack
|
|
|
|
- **kube-prometheus-stack** — Prometheus (metrics), Alertmanager (routing/notify), Grafana
|
|
(dashboards), and the standard Kubernetes exporters/rules.
|
|
- **Loki** — log aggregation.
|
|
- **Grafana Alloy** (or Promtail-class agent) — ships node/pod/VM logs to Loki.
|
|
- **Hubble** — network flow visibility from Cilium ([03](03-networking-isolation-and-egress.md)).
|
|
- **Alertmanager → SMTP** — issues reported by **email** (R-OBS-02).
|
|
- **Grafana** — dashboards for services and platform health (R-OBS-03).
|
|
|
|
## What is monitored (R-OBS-01/04)
|
|
|
|
Application services:
|
|
- Up/health, resource use, request/error signals where exposed, and per-service SLIs declared in the
|
|
ServiceDefinition (a service can declare the alerts it cares about).
|
|
|
|
Platform health (R-OBS-04) — first-class alerts:
|
|
- **Nodes**: down/unreachable, resource pressure, disk fill.
|
|
- **Storage (Longhorn)**: degraded/faulted volumes, replica rebuilds, capacity thresholds.
|
|
- **GitOps (Flux)**: reconciliation failures, drift, suspended resources.
|
|
- **Certificates (cert-manager)**: upcoming expiry, issuance failures.
|
|
- **Networking**: VPN tunnel down, egress-gateway unhealthy, spikes in denied flows (Hubble),
|
|
NetworkPolicy reconcile errors.
|
|
- **Backups (Velero/Longhorn)**: backup job failure, missed schedule, restore-test failures.
|
|
- **KubeVirt**: VM crash-loops / failed launches.
|
|
|
|
## Alert delivery (R-OBS-02)
|
|
|
|
- Alertmanager routes by severity to **email** (SMTP creds are SOPS-encrypted, [07](07-security-and-secrets.md)).
|
|
- Severity model: `critical` (page-worthy: data risk, outage, VPN leak potential, cert about to
|
|
expire) vs `warning` (degraded, capacity, needs attention). Both go to email; only routing/subject
|
|
differs.
|
|
- Alerts include enough context (runbook link, affected object) to act without spelunking.
|
|
|
|
## Dashboards (R-OBS-03)
|
|
|
|
Grafana dashboards, surfaced in the C2 UI ([12](12-command-center-ui.md)):
|
|
- **Platform overview**: nodes, cluster capacity, Flux status, cert expiry, backup status.
|
|
- **Storage**: Longhorn volume health, capacity, replica state.
|
|
- **Network**: Hubble flows, egress/VPN health, denied-flow trends.
|
|
- **Per-service**: one row/dashboard per catalog service with its declared SLIs.
|
|
|
|
## Observability as onboarding output
|
|
|
|
The ServiceDefinition ([13](13-service-catalog.md)) declares observability expectations; the
|
|
platform renders **ServiceMonitors/PodMonitors**, log scraping, dashboards, and alert rules from it.
|
|
A newly onboarded service is monitored **by default**, not as a follow-up task (part of the golden
|
|
path, [01](01-architecture-overview.md)).
|
|
|
|
## Retention & sizing
|
|
- Metrics/log retention set per environment overlay (dev shorter, prod longer). Loki retention
|
|
requires a compactor with retention enabled — do not rely on a period setting alone.
|
|
- Monitoring storage lives on Longhorn; ensure its own capacity is alerted on (don't let the monitor
|
|
blind itself).
|
|
|
|
## Auditability tie-in (R-SEC-03)
|
|
Hubble + Prometheus + cert status provide the evidence for the isolation/egress/TLS/backup proofs
|
|
required by [07](07-security-and-secrets.md).
|
|
|
|
## Acceptance gates
|
|
- Every catalog service emits metrics + logs and has at least one meaningful alert that fires
|
|
correctly in a fault test.
|
|
- Platform-health alerts (node down, Longhorn degraded, Flux failure, cert expiry, VPN down, backup
|
|
failure) each fire and email correctly in an injected-fault test.
|
|
- Grafana dashboards render for platform and each service; the C2 UI links them.
|