Autonomous session: Phase-2 platform authoring + Flux bootstrap progress

Author-only Phase-2 components (platform suspended; cluster untouched): - platform/longhorn: 3-replica SC + bulk external-mount indirection (R-STATE) - platform/monitoring: kube-prometheus-stack + Loki + Alloy (dev-trimmed) + 8 alert rules (R-OBS) - platform/cilium: parity fix (k8sServiceHost=10.73.0.11 + masquerade) safe to adopt Phase 2 - conformance: partial L2 tests (worker hit session limit; incomplete — see DECISIONS-LOG K2) - bootstrap/gitea: fix rootless-podman perms (host bind-mount -> named volume) Flux installed on the cluster; host Gitea up + repo pushed; GitRepository BLOCKED by host firewalld (VM->10.73.0.1:3000) — needs one operator sudo command (DECISIONS-LOG). Known issue K1: cluster-vars substitution-var case mismatch (fix before un-suspend). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 14:32:47 -05:00
1 parent 397f29e99a
commit cc89b4b830
@@ -1,31 +1,117 @@
 # platform/monitoring

 **Observability stack** — kube-prometheus-stack (Prometheus + Alertmanager + Grafana) + **Loki** +
-**Grafana Alloy**.
+**Grafana Alloy** + platform-health **PrometheusRules**.

 - **Spec:** [docs/10 — Observability & Alerting](../../../home-network-k3s/docs/10-observability-and-alerting.md)
  (named in [docs/01](../../../home-network-k3s/docs/01-architecture-overview.md); no dedicated ADR)
- **Serves:** R-OBS-01..04
+- **Serves:** R-OBS-01/02/03/04 (supports R-C2-02, R-SEC-03, R-STATE-03)
 - **Owner:** WS-OBS

-## What this stub is
+## Contents

-Three Flux `HelmRelease`s (kube-prometheus-stack, loki, alloy) with placeholder values. Retention and
-PVC sizes are overlay knobs (`${MONITORING_*}`, R-ENV-02): short/small in dev, long/large in prod.
-Persistent data uses the replicated `longhorn` StorageClass.
+| File | What |
+|---|---|
+| `namespace.yaml` | `monitoring` namespace |
+| `helmrepositories.yaml` | `prometheus-community` + `grafana` Flux `HelmRepository`s |
+| `kube-prometheus-stack.yaml` | `HelmRelease` — Prometheus, Alertmanager (→ SMTP email), Grafana, kube-state-metrics, node-exporter |
+| `loki.yaml` | `HelmRelease` — Loki single-binary, filesystem, compactor-enforced retention |
+| `alloy.yaml` | `HelmRelease` — Grafana Alloy DaemonSet, tails pod logs → Loki |
+| `alerts.yaml` | Platform-health `PrometheusRule`s (R-OBS-04) |
+| `kustomization.yaml` | builds all of the above |

-## Contract (observability)
+## Requirement mapping

-The "observability contract" (`roadmap/workstreams.md`): how a service declares metrics/SLIs/alerts
-and gets dashboards. ServiceMonitors/dashboards are rendered from ServiceDefinitions by WS-APPS/WS-VM
-against this stack. Grafana is part of the C2 UI composition (docs/12), behind Keycloak SSO
-(ADR-0014).
+- **R-OBS-01 (what is monitored):** Prometheus + node-exporter + kube-state-metrics for metrics;
+  Alloy → Loki for logs. `serviceMonitorSelectorNilUsesHelmValues: false` so ServiceMonitors from
+  every namespace (and those rendered from ServiceDefinitions, docs/13) are scraped by default.
+- **R-OBS-02 (alert delivery):** Alertmanager routes by `severity` (`critical` vs `warning`) to
+  **email**. SMTP creds are a SOPS secret referenced **by name only** (see below).
+- **R-OBS-03 (dashboards):** Grafana, with the dashboard/datasource sidecar watching
+  `grafana_dashboard`-labeled ConfigMaps across all namespaces (per-service dashboards land here).
+  Grafana is part of the C2 UI composition (docs/12), later behind Keycloak SSO (ADR-0014).
+- **R-OBS-04 (platform health):** `alerts.yaml` — first-class alerts for **nodes**, **Longhorn**,
+  **Flux**, **cert-manager**, **networking/VPN/Hubble**, **Velero backups**, **Keycloak/IdP**, and
+  the **monitor's own capacity**. Split `critical`/`warning` per the docs/10 severity model.

-## TODO (later phases)
+## SMTP secret (operator hand-off)

- **Phase 4 (WS-OBS):** Alertmanager → Gmail SMTP receiver (creds SOPS-encrypted, name only);
-  Loki retention via compactor (`retention_enabled: true` — a period alone does nothing); Alloy log
-  scrape config → Loki; platform-health alerts. **Reconciliation errors + drift are alerting
-  conditions (R-OBS-04).**
- **Exit gate (Phase 4):** an injected fault emails an alert; dashboards render (R-OBS-01/02/03/04).
- **Grafana OIDC** client via Keycloak (WS-SEC/WS-C2).
+Alertmanager reads the SMTP **password** from a mounted SOPS-encrypted Secret — referenced by name,
+never inlined. The operator must provide (out of band, SOPS-encrypted per docs/07):
+
+```
+kind: Secret
+metadata:
+  name: alertmanager-smtp        # <-- referenced in kube-prometheus-stack.yaml (alertmanagerSpec.secrets)
+  namespace: monitoring
+stringData:
+  smtp-password: "<app password>"   # mounted at /etc/alertmanager/secrets/alertmanager-smtp/smtp-password
+```
+
+The **non-secret** SMTP fields (`smtp_smarthost`, `smtp_from`, `smtp_auth_username`, and the
+receiver `to:` address) are `REPLACE-ME` placeholders in `kube-prometheus-stack.yaml` for the
+operator to fill (same convention as `velero/`). Grafana's admin bootstrap likewise references a
+SOPS secret **by name** — `grafana-admin-credentials` (keys `admin-user` / `admin-password`).
+
+## Dev trims (the ~22 GiB RAM ceiling)
+
+This subtree is authored to stay light on the single dev workstation:
+
+- **Loki `chunksCache`/`resultsCache` DISABLED** — the grafana/loki chart's default memcached caches
+  request **~8-9 GiB each**; leaving them on would blow the budget by itself. Also disabled: the
+  nginx `gateway`, `lokiCanary`, and `test` hooks. Loki runs **SingleBinary** (one pod).
+- **Single replicas** everywhere (Prometheus, Alertmanager, Grafana, Loki).
+- **Modest requests/limits** on every component (see values). Ephemeral (emptyDir) state for
+  Alertmanager and Grafana in dev.
+- **k3s control-plane scrape jobs disabled** (`kubeControllerManager`/`kubeScheduler`/`kubeEtcd`/
+  `kubeProxy`) — embedded in k3s, not separately scrapable; keeps targets green and RAM down.
+
+Approximate steady-state **memory requests** (what the scheduler reserves):
+
+| Component | Req | Notes |
+|---|---:|---|
+| Prometheus | 384Mi | limit 1Gi |
+| Alertmanager | 32Mi | |
+| Grafana | 128Mi | |
+| prometheus-operator | 96Mi | |
+| kube-state-metrics | 64Mi | |
+| node-exporter ×3 | 96Mi | 32Mi/node DaemonSet |
+| Loki (single binary) | 192Mi | limit 512Mi |
+| Alloy ×3 | 288Mi | 96Mi/node DaemonSet |
+| **Total requests** | **~1.3 GiB** | working set typically ~1.5-2.5 GiB; limit ceiling ~3 GiB |
+
+Comfortably within the 22 GiB box alongside the rest of the platform.
+
+## Environment knobs (R-ENV-02)
+
+Only three substitution vars touch this subtree; the dev overlay (`clusters/dev`) patches
+`cluster-vars` to dev values:
+
+| Var | base | dev |
+|---|---|---|
+| `${MONITORING_PROMETHEUS_RETENTION}` | `7d` | `3d` |
+| `${MONITORING_PROMETHEUS_PVC_SIZE}` | `10Gi` | `5Gi` |
+| `${MONITORING_LOKI_PVC_SIZE}` | `10Gi` | `5Gi` |
+
+PVCs use the replicated `longhorn` StorageClass (ADR-0003).
+
+## Notes / known gaps for bring-up
+
+- **Loki log retention has no dedicated knob.** Only the three vars above exist. Loki's
+  `retention_period` is hardcoded to `72h` (compactor-enforced) to mirror dev metrics retention.
+  Promote to a `cluster-vars` knob (in `clusters/`, out of this subtree) if per-env log retention is
+  wanted.
+- Several R-OBS-04 rules depend on exporters/ServiceMonitors owned by **other** workstreams (Flux,
+  cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble). Rules are authored to fire once those
+  series arrive; metric names marked `TODO(WS-*)` in `alerts.yaml` (notably the **egress VPN**
+  metric) need confirming against the deployed exporter.
+- **Hubble** network-flow visibility itself is delivered by `platform/cilium` (docs/03); this subtree
+  only consumes its metrics for denied-flow alerts.
+
+## Validate (no deploy)
+
+```
+kubectl kustomize platform/monitoring
+```
+
+Author-only — nothing here is applied to a cluster; Flux reconciles it at deploy time.