docs: bring the specification in line with the accepted implementation

The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-11-08 22:46:00 -06:00
1 parent 1ddc14b918
commit 250fb54154
43 files changed
+1672 -575

No files matched your search

+30 -11
View File
@@ -7,7 +7,7 @@ Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**.
- **kube-prometheus-stack** — Prometheus (metrics), Alertmanager (routing/notify), Grafana
(dashboards), and the standard Kubernetes exporters/rules.
- **Loki** — log aggregation.
- **Grafana Alloy** (or Promtail-class agent) — ships node/pod/VM logs to Loki.
- **Grafana Alloy** — ships node/pod/VM logs to Loki.
- **Hubble** — network flow visibility from Cilium ([03](03-networking-isolation-and-egress.md)).
- **Alertmanager → SMTP** — issues reported by **email** (R-OBS-02).
- **Grafana** — dashboards for services and platform health (R-OBS-03).
@@ -17,18 +17,29 @@ Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**.
Application services:
- Up/health, resource use, request/error signals where exposed, and per-service SLIs declared in the
ServiceDefinition (a service can declare the alerts it cares about).
- Each forward-auth gate ([15](15-identity-and-access.md)) is scraped in its own right, and the
platform alerts on a gate that nothing observes: an auth failure otherwise looks like a healthy
response.
Platform health (R-OBS-04) — first-class alerts:
- **Nodes**: down/unreachable, resource pressure, disk fill.
- **Storage (Longhorn)**: degraded/faulted volumes, replica rebuilds, capacity thresholds.
- **Nodes**: down/unreachable, CPU, memory and I/O pressure, disk fill, disk latency.
- **Storage (Longhorn + NFS bulk tier)**: degraded/faulted volumes, replica rebuilds, capacity
thresholds, PVC capacity, NFS server latency and client timeouts.
- **Databases (CloudNativePG)**: archiving failures, missed backups, primary placement.
- **GitOps (Flux)**: reconciliation failures, drift, suspended resources.
- **Automated updates**: image automation failures, stalled image policies, failed health gates,
node reboot automation.
- **Certificates (cert-manager)**: upcoming expiry, issuance failures.
- **Networking**: VPN tunnel down, egress-gateway unhealthy, spikes in denied flows (Hubble),
NetworkPolicy reconcile errors.
NetworkPolicy reconcile errors, public ingress tunnel down.
- **Backups (Velero/Longhorn)**: backup job failure, missed schedule, restore-test failures.
- **KubeVirt**: VM crash-loops / failed launches.
- **Identity (Keycloak)**: IdP down/unhealthy or its DB unavailable (SSO is a critical dependency —
[15](15-identity-and-access.md)); certificate/login-endpoint failures.
[15](15-identity-and-access.md)); certificate/login-endpoint failures; a failing or stale
identity-config reconciler.
- **The monitor itself**: monitoring volume capacity, missing scrape targets.
Coverage against this list is tracked in [known gaps](16-known-gaps.md).
## Alert delivery (R-OBS-02)
@@ -36,7 +47,12 @@ Platform health (R-OBS-04) — first-class alerts:
- Severity model: `critical` (page-worthy: data risk, outage, VPN leak potential, cert about to
expire) vs `warning` (degraded, capacity, needs attention). Both go to email; only routing/subject
differs.
- Alerts include enough context (runbook link, affected object) to act without spelunking.
- **Deploy notices** are a separate, non-severity notification class, also by email: one message
when Flux starts applying a revision and one when every Kustomization has reached it, so a routine
deploy never reads as a platform fault.
- Alerts include enough context (runbook link, affected object) to act without spelunking. Every
alert's runbook link resolves to a runbook served under `/runbooks` on the apex, and every rule has
a unit test; the merge gate checks both.
## Dashboards (R-OBS-03)
@@ -49,13 +65,16 @@ Grafana dashboards, surfaced in the C2 UI ([12](12-command-center-ui.md)):
## Observability as onboarding output
The ServiceDefinition ([13](13-service-catalog.md)) declares observability expectations; the
platform renders **ServiceMonitors/PodMonitors**, log scraping, dashboards, and alert rules from it.
A newly onboarded service is monitored **by default**, not as a follow-up task (part of the golden
path, [01](01-architecture-overview.md)).
platform renders **ServiceMonitors/PodMonitors** for services that declare a metrics endpoint, and
dashboards and alert rules from its declared SLIs. Log collection needs no per-service rendering:
the log agent discovers every pod. A newly onboarded service is monitored **by default**, not as a
follow-up task (part of the golden path, [01](01-architecture-overview.md)).
## Retention & sizing
- Metrics/log retention set per environment overlay (dev shorter, prod longer). Loki retention
requires a compactor with retention enabled — do not rely on a period setting alone.
- Metrics/log retention is set in the environment overlay. Metrics retention is capped by size as
well as age, so a young or busy TSDB cannot fill its volume and blind the alerts that depend on
it. Loki retention requires a compactor with retention enabled — do not rely on a period setting
alone.
- Monitoring storage lives on Longhorn; ensure its own capacity is alerted on (don't let the monitor
blind itself).