jakman85andClaude Opus 4.8
eb31d2ef58
The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown,
KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown,
CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches:
the Prometheus PVC is 100% full ("no space left on device" on every WAL commit).
Evidence from the live dev cluster:
- /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit
failed ... write to WAL: no space left on device". Scrapes still return 200
(targets show healthy, lastScrape=now) but samples are never stored, so the
TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC.
- Instant queries at "now" therefore return an EMPTY vector, so every
absent()/==0 platform rule trips. Evaluated at the last-good-sample time each
expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet
up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default
present) — the rules are correct; the monitor was simply blind.
`retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev
retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus
a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based
retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each
env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the
dev PVC 5Gi->8Gi for real WAL/head/compaction headroom.
- platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob}
- clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC)
- clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB
- clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC)
- platform/monitoring/README.md: knob table + rationale
Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune
until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion
(operator PVC expansion / manual patch); once free space returns, retentionSize
keeps it healthy. Watchdog left untouched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
126 lines
6.3 KiB
Markdown
126 lines
6.3 KiB
Markdown
# platform/monitoring
|
||
|
||
**Observability stack** — kube-prometheus-stack (Prometheus + Alertmanager + Grafana) + **Loki** +
|
||
**Grafana Alloy** + platform-health **PrometheusRules**.
|
||
|
||
- **Spec:** [docs/10 — Observability & Alerting](../../../home-network-k3s/docs/10-observability-and-alerting.md)
|
||
(named in [docs/01](../../../home-network-k3s/docs/01-architecture-overview.md); no dedicated ADR)
|
||
- **Serves:** R-OBS-01/02/03/04 (supports R-C2-02, R-SEC-03, R-STATE-03)
|
||
- **Owner:** WS-OBS
|
||
|
||
## Contents
|
||
|
||
| File | What |
|
||
|---|---|
|
||
| `namespace.yaml` | `monitoring` namespace |
|
||
| `helmrepositories.yaml` | `prometheus-community` + `grafana` Flux `HelmRepository`s |
|
||
| `kube-prometheus-stack.yaml` | `HelmRelease` — Prometheus, Alertmanager (→ SMTP email), Grafana, kube-state-metrics, node-exporter |
|
||
| `loki.yaml` | `HelmRelease` — Loki single-binary, filesystem, compactor-enforced retention |
|
||
| `alloy.yaml` | `HelmRelease` — Grafana Alloy DaemonSet, tails pod logs → Loki |
|
||
| `alerts.yaml` | Platform-health `PrometheusRule`s (R-OBS-04) |
|
||
| `kustomization.yaml` | builds all of the above |
|
||
|
||
## Requirement mapping
|
||
|
||
- **R-OBS-01 (what is monitored):** Prometheus + node-exporter + kube-state-metrics for metrics;
|
||
Alloy → Loki for logs. `serviceMonitorSelectorNilUsesHelmValues: false` so ServiceMonitors from
|
||
every namespace (and those rendered from ServiceDefinitions, docs/13) are scraped by default.
|
||
- **R-OBS-02 (alert delivery):** Alertmanager routes by `severity` (`critical` vs `warning`) to
|
||
**email**. SMTP creds are a SOPS secret referenced **by name only** (see below).
|
||
- **R-OBS-03 (dashboards):** Grafana, with the dashboard/datasource sidecar watching
|
||
`grafana_dashboard`-labeled ConfigMaps across all namespaces (per-service dashboards land here).
|
||
Grafana is part of the C2 UI composition (docs/12), later behind Keycloak SSO (ADR-0014).
|
||
- **R-OBS-04 (platform health):** `alerts.yaml` — first-class alerts for **nodes**, **Longhorn**,
|
||
**Flux**, **cert-manager**, **networking/VPN/Hubble**, **Velero backups**, **Keycloak/IdP**, and
|
||
the **monitor's own capacity**. Split `critical`/`warning` per the docs/10 severity model.
|
||
|
||
## SMTP secret (operator hand-off)
|
||
|
||
Alertmanager reads the SMTP **password** from a mounted SOPS-encrypted Secret — referenced by name,
|
||
never inlined. The operator must provide (out of band, SOPS-encrypted per docs/07):
|
||
|
||
```
|
||
kind: Secret
|
||
metadata:
|
||
name: alertmanager-smtp # <-- referenced in kube-prometheus-stack.yaml (alertmanagerSpec.secrets)
|
||
namespace: monitoring
|
||
stringData:
|
||
smtp-password: "<app password>" # mounted at /etc/alertmanager/secrets/alertmanager-smtp/smtp-password
|
||
```
|
||
|
||
The **non-secret** SMTP fields (`smtp_smarthost`, `smtp_from`, `smtp_auth_username`, and the
|
||
receiver `to:` address) are `REPLACE-ME` placeholders in `kube-prometheus-stack.yaml` for the
|
||
operator to fill (same convention as `velero/`). Grafana's admin bootstrap likewise references a
|
||
SOPS secret **by name** — `grafana-admin-credentials` (keys `admin-user` / `admin-password`).
|
||
|
||
## Dev trims (the ~22 GiB RAM ceiling)
|
||
|
||
This subtree is authored to stay light on the single dev workstation:
|
||
|
||
- **Loki `chunksCache`/`resultsCache` DISABLED** — the grafana/loki chart's default memcached caches
|
||
request **~8-9 GiB each**; leaving them on would blow the budget by itself. Also disabled: the
|
||
nginx `gateway`, `lokiCanary`, and `test` hooks. Loki runs **SingleBinary** (one pod).
|
||
- **Single replicas** everywhere (Prometheus, Alertmanager, Grafana, Loki).
|
||
- **Modest requests/limits** on every component (see values). Ephemeral (emptyDir) state for
|
||
Alertmanager and Grafana in dev.
|
||
- **k3s control-plane scrape jobs disabled** (`kubeControllerManager`/`kubeScheduler`/`kubeEtcd`/
|
||
`kubeProxy`) — embedded in k3s, not separately scrapable; keeps targets green and RAM down.
|
||
|
||
Approximate steady-state **memory requests** (what the scheduler reserves):
|
||
|
||
| Component | Req | Notes |
|
||
|---|---:|---|
|
||
| Prometheus | 384Mi | limit 1Gi |
|
||
| Alertmanager | 32Mi | |
|
||
| Grafana | 128Mi | |
|
||
| prometheus-operator | 96Mi | |
|
||
| kube-state-metrics | 64Mi | |
|
||
| node-exporter ×3 | 96Mi | 32Mi/node DaemonSet |
|
||
| Loki (single binary) | 192Mi | limit 512Mi |
|
||
| Alloy ×3 | 288Mi | 96Mi/node DaemonSet |
|
||
| **Total requests** | **~1.3 GiB** | working set typically ~1.5-2.5 GiB; limit ceiling ~3 GiB |
|
||
|
||
Comfortably within the 22 GiB box alongside the rest of the platform.
|
||
|
||
## Environment knobs (R-ENV-02)
|
||
|
||
Only three substitution vars touch this subtree; the dev overlay (`clusters/dev`) patches
|
||
`cluster-vars` to dev values:
|
||
|
||
| Var | base | dev |
|
||
|---|---|---|
|
||
| `${MONITORING_PROMETHEUS_RETENTION}` | `7d` | `3d` |
|
||
| `${MONITORING_PROMETHEUS_PVC_SIZE}` | `10Gi` | `8Gi` |
|
||
| `${MONITORING_PROMETHEUS_RETENTION_SIZE}` | `8GiB` | `5GiB` |
|
||
| `${MONITORING_LOKI_PVC_SIZE}` | `10Gi` | `5Gi` |
|
||
|
||
`MONITORING_PROMETHEUS_RETENTION_SIZE` is a **size-based** retention cap (Prometheus `*iB` units,
|
||
distinct from the K8s `Gi` PVC knob) and must stay **below** `MONITORING_PROMETHEUS_PVC_SIZE`. Time-only
|
||
retention prunes nothing until the age window elapses, so a young/busy cluster fills `/prometheus` first;
|
||
a full disk wedges every WAL commit (scrapes still 200 but samples are dropped, the head freezes) and
|
||
makes every `absent()`/`==0` platform alert fire falsely. The size cap drops the oldest blocks before the
|
||
disk fills. (prod overlay: PVC `50Gi`, cap `45GiB`.)
|
||
|
||
PVCs use the replicated `longhorn` StorageClass (ADR-0003).
|
||
|
||
## Notes / known gaps for bring-up
|
||
|
||
- **Loki log retention has no dedicated knob.** Only the three vars above exist. Loki's
|
||
`retention_period` is hardcoded to `72h` (compactor-enforced) to mirror dev metrics retention.
|
||
Promote to a `cluster-vars` knob (in `clusters/`, out of this subtree) if per-env log retention is
|
||
wanted.
|
||
- Several R-OBS-04 rules depend on exporters/ServiceMonitors owned by **other** workstreams (Flux,
|
||
cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble). Rules are authored to fire once those
|
||
series arrive; metric names marked `TODO(WS-*)` in `alerts.yaml` (notably the **egress VPN**
|
||
metric) need confirming against the deployed exporter.
|
||
- **Hubble** network-flow visibility itself is delivered by `platform/cilium` (docs/03); this subtree
|
||
only consumes its metrics for denied-flow alerts.
|
||
|
||
## Validate (no deploy)
|
||
|
||
```
|
||
kubectl kustomize platform/monitoring
|
||
```
|
||
|
||
Author-only — nothing here is applied to a cluster; Flux reconciles it at deploy time.
|