jakman85andClaude Opus 5.5
250fb54154
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
5.0 KiB
5.0 KiB
10 — Observability & Alerting
Serves: R-OBS-01/02/03/04, supports R-C2-02, R-SEC-03.
Stack
- kube-prometheus-stack — Prometheus (metrics), Alertmanager (routing/notify), Grafana (dashboards), and the standard Kubernetes exporters/rules.
- Loki — log aggregation.
- Grafana Alloy — ships node/pod/VM logs to Loki.
- Hubble — network flow visibility from Cilium (03).
- Alertmanager → SMTP — issues reported by email (R-OBS-02).
- Grafana — dashboards for services and platform health (R-OBS-03).
What is monitored (R-OBS-01/04)
Application services:
- Up/health, resource use, request/error signals where exposed, and per-service SLIs declared in the ServiceDefinition (a service can declare the alerts it cares about).
- Each forward-auth gate (15) is scraped in its own right, and the platform alerts on a gate that nothing observes: an auth failure otherwise looks like a healthy response.
Platform health (R-OBS-04) — first-class alerts:
- Nodes: down/unreachable, CPU, memory and I/O pressure, disk fill, disk latency.
- Storage (Longhorn + NFS bulk tier): degraded/faulted volumes, replica rebuilds, capacity thresholds, PVC capacity, NFS server latency and client timeouts.
- Databases (CloudNativePG): archiving failures, missed backups, primary placement.
- GitOps (Flux): reconciliation failures, drift, suspended resources.
- Automated updates: image automation failures, stalled image policies, failed health gates, node reboot automation.
- Certificates (cert-manager): upcoming expiry, issuance failures.
- Networking: VPN tunnel down, egress-gateway unhealthy, spikes in denied flows (Hubble), NetworkPolicy reconcile errors, public ingress tunnel down.
- Backups (Velero/Longhorn): backup job failure, missed schedule, restore-test failures.
- KubeVirt: VM crash-loops / failed launches.
- Identity (Keycloak): IdP down/unhealthy or its DB unavailable (SSO is a critical dependency — 15); certificate/login-endpoint failures; a failing or stale identity-config reconciler.
- The monitor itself: monitoring volume capacity, missing scrape targets.
Coverage against this list is tracked in known gaps.
Alert delivery (R-OBS-02)
- Alertmanager routes by severity to email (SMTP creds are SOPS-encrypted, 07).
- Severity model:
critical(page-worthy: data risk, outage, VPN leak potential, cert about to expire) vswarning(degraded, capacity, needs attention). Both go to email; only routing/subject differs. - Deploy notices are a separate, non-severity notification class, also by email: one message when Flux starts applying a revision and one when every Kustomization has reached it, so a routine deploy never reads as a platform fault.
- Alerts include enough context (runbook link, affected object) to act without spelunking. Every
alert's runbook link resolves to a runbook served under
/runbookson the apex, and every rule has a unit test; the merge gate checks both.
Dashboards (R-OBS-03)
Grafana dashboards, surfaced in the C2 UI (12):
- Platform overview: nodes, cluster capacity, Flux status, cert expiry, backup status.
- Storage: Longhorn volume health, capacity, replica state.
- Network: Hubble flows, egress/VPN health, denied-flow trends.
- Per-service: one row/dashboard per catalog service with its declared SLIs.
Observability as onboarding output
The ServiceDefinition (13) declares observability expectations; the platform renders ServiceMonitors/PodMonitors for services that declare a metrics endpoint, and dashboards and alert rules from its declared SLIs. Log collection needs no per-service rendering: the log agent discovers every pod. A newly onboarded service is monitored by default, not as a follow-up task (part of the golden path, 01).
Retention & sizing
- Metrics/log retention is set in the environment overlay. Metrics retention is capped by size as well as age, so a young or busy TSDB cannot fill its volume and blind the alerts that depend on it. Loki retention requires a compactor with retention enabled — do not rely on a period setting alone.
- Monitoring storage lives on Longhorn; ensure its own capacity is alerted on (don't let the monitor blind itself).
Auditability tie-in (R-SEC-03)
Hubble + Prometheus + cert status provide the evidence for the isolation/egress/TLS/backup proofs required by 07.
Acceptance gates
- Every catalog service emits metrics + logs and has at least one meaningful alert that fires correctly in a fault test.
- Platform-health alerts (node down, Longhorn degraded, Flux failure, cert expiry, VPN down, backup failure) each fire and email correctly in an injected-fault test.
- Grafana dashboards render for platform and each service; the C2 UI links them.