docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
1ddc14b918
commit
250fb54154
43 files changed
+1672
-575
No files matched your search
@@ -7,7 +7,7 @@ Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**.
|
||||
- **kube-prometheus-stack** — Prometheus (metrics), Alertmanager (routing/notify), Grafana
|
||||
(dashboards), and the standard Kubernetes exporters/rules.
|
||||
- **Loki** — log aggregation.
|
||||
- **Grafana Alloy** (or Promtail-class agent) — ships node/pod/VM logs to Loki.
|
||||
- **Grafana Alloy** — ships node/pod/VM logs to Loki.
|
||||
- **Hubble** — network flow visibility from Cilium ([03](03-networking-isolation-and-egress.md)).
|
||||
- **Alertmanager → SMTP** — issues reported by **email** (R-OBS-02).
|
||||
- **Grafana** — dashboards for services and platform health (R-OBS-03).
|
||||
@@ -17,18 +17,29 @@ Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**.
|
||||
Application services:
|
||||
- Up/health, resource use, request/error signals where exposed, and per-service SLIs declared in the
|
||||
ServiceDefinition (a service can declare the alerts it cares about).
|
||||
- Each forward-auth gate ([15](15-identity-and-access.md)) is scraped in its own right, and the
|
||||
platform alerts on a gate that nothing observes: an auth failure otherwise looks like a healthy
|
||||
response.
|
||||
|
||||
Platform health (R-OBS-04) — first-class alerts:
|
||||
- **Nodes**: down/unreachable, resource pressure, disk fill.
|
||||
- **Storage (Longhorn)**: degraded/faulted volumes, replica rebuilds, capacity thresholds.
|
||||
- **Nodes**: down/unreachable, CPU, memory and I/O pressure, disk fill, disk latency.
|
||||
- **Storage (Longhorn + NFS bulk tier)**: degraded/faulted volumes, replica rebuilds, capacity
|
||||
thresholds, PVC capacity, NFS server latency and client timeouts.
|
||||
- **Databases (CloudNativePG)**: archiving failures, missed backups, primary placement.
|
||||
- **GitOps (Flux)**: reconciliation failures, drift, suspended resources.
|
||||
- **Automated updates**: image automation failures, stalled image policies, failed health gates,
|
||||
node reboot automation.
|
||||
- **Certificates (cert-manager)**: upcoming expiry, issuance failures.
|
||||
- **Networking**: VPN tunnel down, egress-gateway unhealthy, spikes in denied flows (Hubble),
|
||||
NetworkPolicy reconcile errors.
|
||||
NetworkPolicy reconcile errors, public ingress tunnel down.
|
||||
- **Backups (Velero/Longhorn)**: backup job failure, missed schedule, restore-test failures.
|
||||
- **KubeVirt**: VM crash-loops / failed launches.
|
||||
- **Identity (Keycloak)**: IdP down/unhealthy or its DB unavailable (SSO is a critical dependency —
|
||||
[15](15-identity-and-access.md)); certificate/login-endpoint failures.
|
||||
[15](15-identity-and-access.md)); certificate/login-endpoint failures; a failing or stale
|
||||
identity-config reconciler.
|
||||
- **The monitor itself**: monitoring volume capacity, missing scrape targets.
|
||||
|
||||
Coverage against this list is tracked in [known gaps](16-known-gaps.md).
|
||||
|
||||
## Alert delivery (R-OBS-02)
|
||||
|
||||
@@ -36,7 +47,12 @@ Platform health (R-OBS-04) — first-class alerts:
|
||||
- Severity model: `critical` (page-worthy: data risk, outage, VPN leak potential, cert about to
|
||||
expire) vs `warning` (degraded, capacity, needs attention). Both go to email; only routing/subject
|
||||
differs.
|
||||
- Alerts include enough context (runbook link, affected object) to act without spelunking.
|
||||
- **Deploy notices** are a separate, non-severity notification class, also by email: one message
|
||||
when Flux starts applying a revision and one when every Kustomization has reached it, so a routine
|
||||
deploy never reads as a platform fault.
|
||||
- Alerts include enough context (runbook link, affected object) to act without spelunking. Every
|
||||
alert's runbook link resolves to a runbook served under `/runbooks` on the apex, and every rule has
|
||||
a unit test; the merge gate checks both.
|
||||
|
||||
## Dashboards (R-OBS-03)
|
||||
|
||||
@@ -49,13 +65,16 @@ Grafana dashboards, surfaced in the C2 UI ([12](12-command-center-ui.md)):
|
||||
## Observability as onboarding output
|
||||
|
||||
The ServiceDefinition ([13](13-service-catalog.md)) declares observability expectations; the
|
||||
platform renders **ServiceMonitors/PodMonitors**, log scraping, dashboards, and alert rules from it.
|
||||
A newly onboarded service is monitored **by default**, not as a follow-up task (part of the golden
|
||||
path, [01](01-architecture-overview.md)).
|
||||
platform renders **ServiceMonitors/PodMonitors** for services that declare a metrics endpoint, and
|
||||
dashboards and alert rules from its declared SLIs. Log collection needs no per-service rendering:
|
||||
the log agent discovers every pod. A newly onboarded service is monitored **by default**, not as a
|
||||
follow-up task (part of the golden path, [01](01-architecture-overview.md)).
|
||||
|
||||
## Retention & sizing
|
||||
- Metrics/log retention set per environment overlay (dev shorter, prod longer). Loki retention
|
||||
requires a compactor with retention enabled — do not rely on a period setting alone.
|
||||
- Metrics/log retention is set in the environment overlay. Metrics retention is capped by size as
|
||||
well as age, so a young or busy TSDB cannot fill its volume and blind the alerts that depend on
|
||||
it. Loki retention requires a compactor with retention enabled — do not rely on a period setting
|
||||
alone.
|
||||
- Monitoring storage lives on Longhorn; ensure its own capacity is alerted on (don't let the monitor
|
||||
blind itself).
|
||||
|
||||
|
||||
Reference in new issue
Block a user