jakman85andClaude Opus 4.8
cc89b4b830
Author-only Phase-2 components (platform suspended; cluster untouched): - platform/longhorn: 3-replica SC + bulk external-mount indirection (R-STATE) - platform/monitoring: kube-prometheus-stack + Loki + Alloy (dev-trimmed) + 8 alert rules (R-OBS) - platform/cilium: parity fix (k8sServiceHost=10.73.0.11 + masquerade) safe to adopt Phase 2 - conformance: partial L2 tests (worker hit session limit; incomplete — see DECISIONS-LOG K2) - bootstrap/gitea: fix rootless-podman perms (host bind-mount -> named volume) Flux installed on the cluster; host Gitea up + repo pushed; GitRepository BLOCKED by host firewalld (VM->10.73.0.1:3000) — needs one operator sudo command (DECISIONS-LOG). Known issue K1: cluster-vars substitution-var case mismatch (fix before un-suspend). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
118 lines
5.7 KiB
Markdown
118 lines
5.7 KiB
Markdown
# platform/monitoring
|
||
|
||
**Observability stack** — kube-prometheus-stack (Prometheus + Alertmanager + Grafana) + **Loki** +
|
||
**Grafana Alloy** + platform-health **PrometheusRules**.
|
||
|
||
- **Spec:** [docs/10 — Observability & Alerting](../../../home-network-k3s/docs/10-observability-and-alerting.md)
|
||
(named in [docs/01](../../../home-network-k3s/docs/01-architecture-overview.md); no dedicated ADR)
|
||
- **Serves:** R-OBS-01/02/03/04 (supports R-C2-02, R-SEC-03, R-STATE-03)
|
||
- **Owner:** WS-OBS
|
||
|
||
## Contents
|
||
|
||
| File | What |
|
||
|---|---|
|
||
| `namespace.yaml` | `monitoring` namespace |
|
||
| `helmrepositories.yaml` | `prometheus-community` + `grafana` Flux `HelmRepository`s |
|
||
| `kube-prometheus-stack.yaml` | `HelmRelease` — Prometheus, Alertmanager (→ SMTP email), Grafana, kube-state-metrics, node-exporter |
|
||
| `loki.yaml` | `HelmRelease` — Loki single-binary, filesystem, compactor-enforced retention |
|
||
| `alloy.yaml` | `HelmRelease` — Grafana Alloy DaemonSet, tails pod logs → Loki |
|
||
| `alerts.yaml` | Platform-health `PrometheusRule`s (R-OBS-04) |
|
||
| `kustomization.yaml` | builds all of the above |
|
||
|
||
## Requirement mapping
|
||
|
||
- **R-OBS-01 (what is monitored):** Prometheus + node-exporter + kube-state-metrics for metrics;
|
||
Alloy → Loki for logs. `serviceMonitorSelectorNilUsesHelmValues: false` so ServiceMonitors from
|
||
every namespace (and those rendered from ServiceDefinitions, docs/13) are scraped by default.
|
||
- **R-OBS-02 (alert delivery):** Alertmanager routes by `severity` (`critical` vs `warning`) to
|
||
**email**. SMTP creds are a SOPS secret referenced **by name only** (see below).
|
||
- **R-OBS-03 (dashboards):** Grafana, with the dashboard/datasource sidecar watching
|
||
`grafana_dashboard`-labeled ConfigMaps across all namespaces (per-service dashboards land here).
|
||
Grafana is part of the C2 UI composition (docs/12), later behind Keycloak SSO (ADR-0014).
|
||
- **R-OBS-04 (platform health):** `alerts.yaml` — first-class alerts for **nodes**, **Longhorn**,
|
||
**Flux**, **cert-manager**, **networking/VPN/Hubble**, **Velero backups**, **Keycloak/IdP**, and
|
||
the **monitor's own capacity**. Split `critical`/`warning` per the docs/10 severity model.
|
||
|
||
## SMTP secret (operator hand-off)
|
||
|
||
Alertmanager reads the SMTP **password** from a mounted SOPS-encrypted Secret — referenced by name,
|
||
never inlined. The operator must provide (out of band, SOPS-encrypted per docs/07):
|
||
|
||
```
|
||
kind: Secret
|
||
metadata:
|
||
name: alertmanager-smtp # <-- referenced in kube-prometheus-stack.yaml (alertmanagerSpec.secrets)
|
||
namespace: monitoring
|
||
stringData:
|
||
smtp-password: "<app password>" # mounted at /etc/alertmanager/secrets/alertmanager-smtp/smtp-password
|
||
```
|
||
|
||
The **non-secret** SMTP fields (`smtp_smarthost`, `smtp_from`, `smtp_auth_username`, and the
|
||
receiver `to:` address) are `REPLACE-ME` placeholders in `kube-prometheus-stack.yaml` for the
|
||
operator to fill (same convention as `velero/`). Grafana's admin bootstrap likewise references a
|
||
SOPS secret **by name** — `grafana-admin-credentials` (keys `admin-user` / `admin-password`).
|
||
|
||
## Dev trims (the ~22 GiB RAM ceiling)
|
||
|
||
This subtree is authored to stay light on the single dev workstation:
|
||
|
||
- **Loki `chunksCache`/`resultsCache` DISABLED** — the grafana/loki chart's default memcached caches
|
||
request **~8-9 GiB each**; leaving them on would blow the budget by itself. Also disabled: the
|
||
nginx `gateway`, `lokiCanary`, and `test` hooks. Loki runs **SingleBinary** (one pod).
|
||
- **Single replicas** everywhere (Prometheus, Alertmanager, Grafana, Loki).
|
||
- **Modest requests/limits** on every component (see values). Ephemeral (emptyDir) state for
|
||
Alertmanager and Grafana in dev.
|
||
- **k3s control-plane scrape jobs disabled** (`kubeControllerManager`/`kubeScheduler`/`kubeEtcd`/
|
||
`kubeProxy`) — embedded in k3s, not separately scrapable; keeps targets green and RAM down.
|
||
|
||
Approximate steady-state **memory requests** (what the scheduler reserves):
|
||
|
||
| Component | Req | Notes |
|
||
|---|---:|---|
|
||
| Prometheus | 384Mi | limit 1Gi |
|
||
| Alertmanager | 32Mi | |
|
||
| Grafana | 128Mi | |
|
||
| prometheus-operator | 96Mi | |
|
||
| kube-state-metrics | 64Mi | |
|
||
| node-exporter ×3 | 96Mi | 32Mi/node DaemonSet |
|
||
| Loki (single binary) | 192Mi | limit 512Mi |
|
||
| Alloy ×3 | 288Mi | 96Mi/node DaemonSet |
|
||
| **Total requests** | **~1.3 GiB** | working set typically ~1.5-2.5 GiB; limit ceiling ~3 GiB |
|
||
|
||
Comfortably within the 22 GiB box alongside the rest of the platform.
|
||
|
||
## Environment knobs (R-ENV-02)
|
||
|
||
Only three substitution vars touch this subtree; the dev overlay (`clusters/dev`) patches
|
||
`cluster-vars` to dev values:
|
||
|
||
| Var | base | dev |
|
||
|---|---|---|
|
||
| `${MONITORING_PROMETHEUS_RETENTION}` | `7d` | `3d` |
|
||
| `${MONITORING_PROMETHEUS_PVC_SIZE}` | `10Gi` | `5Gi` |
|
||
| `${MONITORING_LOKI_PVC_SIZE}` | `10Gi` | `5Gi` |
|
||
|
||
PVCs use the replicated `longhorn` StorageClass (ADR-0003).
|
||
|
||
## Notes / known gaps for bring-up
|
||
|
||
- **Loki log retention has no dedicated knob.** Only the three vars above exist. Loki's
|
||
`retention_period` is hardcoded to `72h` (compactor-enforced) to mirror dev metrics retention.
|
||
Promote to a `cluster-vars` knob (in `clusters/`, out of this subtree) if per-env log retention is
|
||
wanted.
|
||
- Several R-OBS-04 rules depend on exporters/ServiceMonitors owned by **other** workstreams (Flux,
|
||
cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble). Rules are authored to fire once those
|
||
series arrive; metric names marked `TODO(WS-*)` in `alerts.yaml` (notably the **egress VPN**
|
||
metric) need confirming against the deployed exporter.
|
||
- **Hubble** network-flow visibility itself is delivered by `platform/cilium` (docs/03); this subtree
|
||
only consumes its metrics for denied-flow alerts.
|
||
|
||
## Validate (no deploy)
|
||
|
||
```
|
||
kubectl kustomize platform/monitoring
|
||
```
|
||
|
||
Author-only — nothing here is applied to a cluster; Flux reconciles it at deploy time.
|