jakman85andClaude Opus 5.5
250fb54154
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
91 lines
5.0 KiB
Markdown
91 lines
5.0 KiB
Markdown
# 10 — Observability & Alerting
|
|
|
|
Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**.
|
|
|
|
## Stack
|
|
|
|
- **kube-prometheus-stack** — Prometheus (metrics), Alertmanager (routing/notify), Grafana
|
|
(dashboards), and the standard Kubernetes exporters/rules.
|
|
- **Loki** — log aggregation.
|
|
- **Grafana Alloy** — ships node/pod/VM logs to Loki.
|
|
- **Hubble** — network flow visibility from Cilium ([03](03-networking-isolation-and-egress.md)).
|
|
- **Alertmanager → SMTP** — issues reported by **email** (R-OBS-02).
|
|
- **Grafana** — dashboards for services and platform health (R-OBS-03).
|
|
|
|
## What is monitored (R-OBS-01/04)
|
|
|
|
Application services:
|
|
- Up/health, resource use, request/error signals where exposed, and per-service SLIs declared in the
|
|
ServiceDefinition (a service can declare the alerts it cares about).
|
|
- Each forward-auth gate ([15](15-identity-and-access.md)) is scraped in its own right, and the
|
|
platform alerts on a gate that nothing observes: an auth failure otherwise looks like a healthy
|
|
response.
|
|
|
|
Platform health (R-OBS-04) — first-class alerts:
|
|
- **Nodes**: down/unreachable, CPU, memory and I/O pressure, disk fill, disk latency.
|
|
- **Storage (Longhorn + NFS bulk tier)**: degraded/faulted volumes, replica rebuilds, capacity
|
|
thresholds, PVC capacity, NFS server latency and client timeouts.
|
|
- **Databases (CloudNativePG)**: archiving failures, missed backups, primary placement.
|
|
- **GitOps (Flux)**: reconciliation failures, drift, suspended resources.
|
|
- **Automated updates**: image automation failures, stalled image policies, failed health gates,
|
|
node reboot automation.
|
|
- **Certificates (cert-manager)**: upcoming expiry, issuance failures.
|
|
- **Networking**: VPN tunnel down, egress-gateway unhealthy, spikes in denied flows (Hubble),
|
|
NetworkPolicy reconcile errors, public ingress tunnel down.
|
|
- **Backups (Velero/Longhorn)**: backup job failure, missed schedule, restore-test failures.
|
|
- **KubeVirt**: VM crash-loops / failed launches.
|
|
- **Identity (Keycloak)**: IdP down/unhealthy or its DB unavailable (SSO is a critical dependency —
|
|
[15](15-identity-and-access.md)); certificate/login-endpoint failures; a failing or stale
|
|
identity-config reconciler.
|
|
- **The monitor itself**: monitoring volume capacity, missing scrape targets.
|
|
|
|
Coverage against this list is tracked in [known gaps](16-known-gaps.md).
|
|
|
|
## Alert delivery (R-OBS-02)
|
|
|
|
- Alertmanager routes by severity to **email** (SMTP creds are SOPS-encrypted, [07](07-security-and-secrets.md)).
|
|
- Severity model: `critical` (page-worthy: data risk, outage, VPN leak potential, cert about to
|
|
expire) vs `warning` (degraded, capacity, needs attention). Both go to email; only routing/subject
|
|
differs.
|
|
- **Deploy notices** are a separate, non-severity notification class, also by email: one message
|
|
when Flux starts applying a revision and one when every Kustomization has reached it, so a routine
|
|
deploy never reads as a platform fault.
|
|
- Alerts include enough context (runbook link, affected object) to act without spelunking. Every
|
|
alert's runbook link resolves to a runbook served under `/runbooks` on the apex, and every rule has
|
|
a unit test; the merge gate checks both.
|
|
|
|
## Dashboards (R-OBS-03)
|
|
|
|
Grafana dashboards, surfaced in the C2 UI ([12](12-command-center-ui.md)):
|
|
- **Platform overview**: nodes, cluster capacity, Flux status, cert expiry, backup status.
|
|
- **Storage**: Longhorn volume health, capacity, replica state.
|
|
- **Network**: Hubble flows, egress/VPN health, denied-flow trends.
|
|
- **Per-service**: one row/dashboard per catalog service with its declared SLIs.
|
|
|
|
## Observability as onboarding output
|
|
|
|
The ServiceDefinition ([13](13-service-catalog.md)) declares observability expectations; the
|
|
platform renders **ServiceMonitors/PodMonitors** for services that declare a metrics endpoint, and
|
|
dashboards and alert rules from its declared SLIs. Log collection needs no per-service rendering:
|
|
the log agent discovers every pod. A newly onboarded service is monitored **by default**, not as a
|
|
follow-up task (part of the golden path, [01](01-architecture-overview.md)).
|
|
|
|
## Retention & sizing
|
|
- Metrics/log retention is set in the environment overlay. Metrics retention is capped by size as
|
|
well as age, so a young or busy TSDB cannot fill its volume and blind the alerts that depend on
|
|
it. Loki retention requires a compactor with retention enabled — do not rely on a period setting
|
|
alone.
|
|
- Monitoring storage lives on Longhorn; ensure its own capacity is alerted on (don't let the monitor
|
|
blind itself).
|
|
|
|
## Auditability tie-in (R-SEC-03)
|
|
Hubble + Prometheus + cert status provide the evidence for the isolation/egress/TLS/backup proofs
|
|
required by [07](07-security-and-secrets.md).
|
|
|
|
## Acceptance gates
|
|
- Every catalog service emits metrics + logs and has at least one meaningful alert that fires
|
|
correctly in a fault test.
|
|
- Platform-health alerts (node down, Longhorn degraded, Flux failure, cert expiry, VPN down, backup
|
|
failure) each fire and email correctly in an injected-fault test.
|
|
- Grafana dashboards render for platform and each service; the C2 UI links them.
|