# platform/monitoring

**Observability stack** — kube-prometheus-stack (Prometheus + Alertmanager + Grafana) + **Loki** +
**Grafana Alloy** + platform-health **PrometheusRules**.

- **Spec:** [docs/10 — Observability & Alerting](../../../home-network-k3s/docs/10-observability-and-alerting.md)
  (named in [docs/01](../../../home-network-k3s/docs/01-architecture-overview.md); no dedicated ADR)
- **Serves:** R-OBS-01/02/03/04 (supports R-C2-02, R-SEC-03, R-STATE-03)
- **Owner:** WS-OBS

## Contents

| File | What |
|---|---|
| `namespace.yaml` | `monitoring` namespace |
| `helmrepositories.yaml` | `prometheus-community` + `grafana` Flux `HelmRepository`s |
| `kube-prometheus-stack.yaml` | `HelmRelease` — Prometheus, Alertmanager (→ SMTP email), Grafana, kube-state-metrics, node-exporter |
| `loki.yaml` | `HelmRelease` — Loki single-binary, filesystem, compactor-enforced retention |
| `alloy.yaml` | `HelmRelease` — Grafana Alloy DaemonSet, tails pod logs → Loki |
| `alerts.yaml` | Platform-health `PrometheusRule`s (R-OBS-04) |
| `kustomization.yaml` | builds all of the above |

## Requirement mapping

- **R-OBS-01 (what is monitored):** Prometheus + node-exporter + kube-state-metrics for metrics;
  Alloy → Loki for logs. `serviceMonitorSelectorNilUsesHelmValues: false` so ServiceMonitors from
  every namespace (and those rendered from ServiceDefinitions, docs/13) are scraped by default.
- **R-OBS-02 (alert delivery):** Alertmanager routes by `severity` (`critical` vs `warning`) to
  **email**. SMTP creds are a SOPS secret referenced **by name only** (see below).
- **R-OBS-03 (dashboards):** Grafana, with the dashboard/datasource sidecar watching
  `grafana_dashboard`-labeled ConfigMaps across all namespaces (per-service dashboards land here).
  Grafana is part of the C2 UI composition (docs/12), later behind Keycloak SSO (ADR-0014).
- **R-OBS-04 (platform health):** `alerts.yaml` — first-class alerts for **nodes**, **Longhorn**,
  **Flux**, **cert-manager**, **networking/VPN/Hubble**, **Velero backups**, **Keycloak/IdP**, and
  the **monitor's own capacity**. Split `critical`/`warning` per the docs/10 severity model.

## SMTP secret (operator hand-off)

Alertmanager reads the SMTP **password** from a mounted SOPS-encrypted Secret — referenced by name,
never inlined. The operator must provide (out of band, SOPS-encrypted per docs/07):

```
kind: Secret
metadata:
  name: alertmanager-smtp        # <-- referenced in kube-prometheus-stack.yaml (alertmanagerSpec.secrets)
  namespace: monitoring
stringData:
  smtp-password: "<app password>"   # mounted at /etc/alertmanager/secrets/alertmanager-smtp/smtp-password
```

The **non-secret** SMTP fields (`smtp_smarthost`, `smtp_from`, `smtp_auth_username`, and the
receiver `to:` address) are `REPLACE-ME` placeholders in `kube-prometheus-stack.yaml` for the
operator to fill (same convention as `velero/`). Grafana's admin bootstrap likewise references a
SOPS secret **by name** — `grafana-admin-credentials` (keys `admin-user` / `admin-password`).

## Dev trims (the ~22 GiB RAM ceiling)

This subtree is authored to stay light on the single dev workstation:

- **Loki `chunksCache`/`resultsCache` DISABLED** — the grafana/loki chart's default memcached caches
  request **~8-9 GiB each**; leaving them on would blow the budget by itself. Also disabled: the
  nginx `gateway`, `lokiCanary`, and `test` hooks. Loki runs **SingleBinary** (one pod).
- **Single replicas** everywhere (Prometheus, Alertmanager, Grafana, Loki).
- **Modest requests/limits** on every component (see values). Ephemeral (emptyDir) state for
  Alertmanager and Grafana in dev.
- **k3s control-plane scrape jobs disabled** (`kubeControllerManager`/`kubeScheduler`/`kubeEtcd`/
  `kubeProxy`) — embedded in k3s, not separately scrapable; keeps targets green and RAM down.

Approximate steady-state **memory requests** (what the scheduler reserves):

| Component | Req | Notes |
|---|---:|---|
| Prometheus | 384Mi | limit 1Gi |
| Alertmanager | 32Mi | |
| Grafana | 128Mi | |
| prometheus-operator | 96Mi | |
| kube-state-metrics | 64Mi | |
| node-exporter ×3 | 96Mi | 32Mi/node DaemonSet |
| Loki (single binary) | 192Mi | limit 512Mi |
| Alloy ×3 | 288Mi | 96Mi/node DaemonSet |
| **Total requests** | **~1.3 GiB** | working set typically ~1.5-2.5 GiB; limit ceiling ~3 GiB |

Comfortably within the 22 GiB box alongside the rest of the platform.

## Environment knobs (R-ENV-02)

Only three substitution vars touch this subtree; the dev overlay (`clusters/dev`) patches
`cluster-vars` to dev values:

| Var | base | dev |
|---|---|---|
| `${MONITORING_PROMETHEUS_RETENTION}` | `7d` | `3d` |
| `${MONITORING_PROMETHEUS_PVC_SIZE}` | `10Gi` | `8Gi` |
| `${MONITORING_PROMETHEUS_RETENTION_SIZE}` | `8GiB` | `5GiB` |
| `${MONITORING_LOKI_PVC_SIZE}` | `10Gi` | `5Gi` |

`MONITORING_PROMETHEUS_RETENTION_SIZE` is a **size-based** retention cap (Prometheus `*iB` units,
distinct from the K8s `Gi` PVC knob) and must stay **below** `MONITORING_PROMETHEUS_PVC_SIZE`. Time-only
retention prunes nothing until the age window elapses, so a young/busy cluster fills `/prometheus` first;
a full disk wedges every WAL commit (scrapes still 200 but samples are dropped, the head freezes) and
makes every `absent()`/`==0` platform alert fire falsely. The size cap drops the oldest blocks before the
disk fills. (prod overlay: PVC `50Gi`, cap `45GiB`.)

PVCs use the replicated `longhorn` StorageClass (ADR-0003).

## Notes / known gaps for bring-up

- **Loki log retention has no dedicated knob.** Only the three vars above exist. Loki's
  `retention_period` is hardcoded to `72h` (compactor-enforced) to mirror dev metrics retention.
  Promote to a `cluster-vars` knob (in `clusters/`, out of this subtree) if per-env log retention is
  wanted.
- Several R-OBS-04 rules depend on exporters/ServiceMonitors owned by **other** workstreams (Flux,
  cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble). Rules are authored to fire once those
  series arrive; metric names marked `TODO(WS-*)` in `alerts.yaml` (notably the **egress VPN**
  metric) need confirming against the deployed exporter.
- **Hubble** network-flow visibility itself is delivered by `platform/cilium` (docs/03); this subtree
  only consumes its metrics for denied-flow alerts.

## Validate (no deploy)

```
kubectl kustomize platform/monitoring
```

Author-only — nothing here is applied to a cluster; Flux reconciles it at deploy time.