jakman85andClaude Opus 4.8
e8c555cfeb
Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision & requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap (phases + workstreams), and AI-agent coordination (roles, protocol, definition-of-done, task board). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
3.5 KiB
3.5 KiB
10 — Observability & Alerting
Serves: R-OBS-01/02/03/04, supports R-C2-02, R-SEC-03.
Stack
- kube-prometheus-stack — Prometheus (metrics), Alertmanager (routing/notify), Grafana (dashboards), and the standard Kubernetes exporters/rules.
- Loki — log aggregation.
- Grafana Alloy (or Promtail-class agent) — ships node/pod/VM logs to Loki.
- Hubble — network flow visibility from Cilium (03).
- Alertmanager → SMTP — issues reported by email (R-OBS-02).
- Grafana — dashboards for services and platform health (R-OBS-03).
What is monitored (R-OBS-01/04)
Application services:
- Up/health, resource use, request/error signals where exposed, and per-service SLIs declared in the ServiceDefinition (a service can declare the alerts it cares about).
Platform health (R-OBS-04) — first-class alerts:
- Nodes: down/unreachable, resource pressure, disk fill.
- Storage (Longhorn): degraded/faulted volumes, replica rebuilds, capacity thresholds.
- GitOps (Flux): reconciliation failures, drift, suspended resources.
- Certificates (cert-manager): upcoming expiry, issuance failures.
- Networking: VPN tunnel down, egress-gateway unhealthy, spikes in denied flows (Hubble), NetworkPolicy reconcile errors.
- Backups (Velero/Longhorn): backup job failure, missed schedule, restore-test failures.
- KubeVirt: VM crash-loops / failed launches.
Alert delivery (R-OBS-02)
- Alertmanager routes by severity to email (SMTP creds are SOPS-encrypted, 07).
- Severity model:
critical(page-worthy: data risk, outage, VPN leak potential, cert about to expire) vswarning(degraded, capacity, needs attention). Both go to email; only routing/subject differs. - Alerts include enough context (runbook link, affected object) to act without spelunking.
Dashboards (R-OBS-03)
Grafana dashboards, surfaced in the C2 UI (12):
- Platform overview: nodes, cluster capacity, Flux status, cert expiry, backup status.
- Storage: Longhorn volume health, capacity, replica state.
- Network: Hubble flows, egress/VPN health, denied-flow trends.
- Per-service: one row/dashboard per catalog service with its declared SLIs.
Observability as onboarding output
The ServiceDefinition (13) declares observability expectations; the platform renders ServiceMonitors/PodMonitors, log scraping, dashboards, and alert rules from it. A newly onboarded service is monitored by default, not as a follow-up task (part of the golden path, 01).
Retention & sizing
- Metrics/log retention set per environment overlay (dev shorter, prod longer). Loki retention requires a compactor with retention enabled — do not rely on a period setting alone.
- Monitoring storage lives on Longhorn; ensure its own capacity is alerted on (don't let the monitor blind itself).
Auditability tie-in (R-SEC-03)
Hubble + Prometheus + cert status provide the evidence for the isolation/egress/TLS/backup proofs required by 07.
Acceptance gates
- Every catalog service emits metrics + logs and has at least one meaningful alert that fires correctly in a fault test.
- Platform-health alerts (node down, Longhorn degraded, Flux failure, cert expiry, VPN down, backup failure) each fire and email correctly in an injected-fault test.
- Grafana dashboards render for platform and each service; the C2 UI links them.