jakman85andClaude Opus 5
7057bbb711
VeleroNoRecentBackup could not fire for the failure mode that actually happened.
`(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 24*3600` is the
right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once
a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series.
So it detects "backed up once, then stopped" and is structurally INERT on "never
backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused
and never-successful: no volume backups at all, nothing alerted, and it was masked
because `velero-daily-resources-only` completed cleanly every 03:00, so every other
backup signal read green while only Kubernetes objects were protected.
A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports
35 velero_* metric names and every one carrying a `schedule` label is created by a
backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on
prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is
not in the TSDB, so the Schedule CRs are exported via kube-state-metrics'
custom-resource-state collector — the path this repo already uses for
kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}.
Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD
list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the
CRS metrics as an empty set, silently.
Three rules, all keyed on PROPERTIES rather than schedule names:
* VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no
successful backup ever. The 36h grace comes from creationTimestamp so a newly
added schedule is not paged before its first run is even due.
* VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup
schedule must never be silently indefinite.
* VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some
schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a
healthy resources-only schedule can never make the platform look protected.
Reading the property that DEFINES volume protection (not a name list) matches
platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is
load-bearing: `count(<empty>) == 0` returns no series and would be inert —
absence of the exporter is deliberately the alarm.
Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/,
wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs —
never a copy — and pins every rule in the group both ways: quiet on today's healthy
prod state, firing on the reconstructed bad state, with `for:` durations asserted.
Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and
breaking the `unless on (schedule)` join each turn the suite red. This is the tool
that makes both of this repo's recurring alert bugs — permanently-firing and
structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy
cluster can do.
Proof of the fix is by construction; the live metrics cannot show it any more, because
`velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items.
The throwaway probe Schedule and the probe KSM deployment were deleted and the
deletion confirmed; `velero-daily` remains unpaused.
Co-Authored-By: Claude Opus 5 <[email protected]>
127 lines
6.5 KiB
Markdown
127 lines
6.5 KiB
Markdown
# platform/monitoring
|
||
|
||
**Observability stack** — kube-prometheus-stack (Prometheus + Alertmanager + Grafana) + **Loki** +
|
||
**Grafana Alloy** + platform-health **PrometheusRules**.
|
||
|
||
- **Spec:** [docs/10 — Observability & Alerting](../../../home-network-k3s/docs/10-observability-and-alerting.md)
|
||
(named in [docs/01](../../../home-network-k3s/docs/01-architecture-overview.md); no dedicated ADR)
|
||
- **Serves:** R-OBS-01/02/03/04 (supports R-C2-02, R-SEC-03, R-STATE-03)
|
||
- **Owner:** WS-OBS
|
||
|
||
## Contents
|
||
|
||
| File | What |
|
||
|---|---|
|
||
| `namespace.yaml` | `monitoring` namespace |
|
||
| `helmrepositories.yaml` | `prometheus-community` + `grafana` Flux `HelmRepository`s |
|
||
| `kube-prometheus-stack.yaml` | `HelmRelease` — Prometheus, Alertmanager (→ SMTP email), Grafana, kube-state-metrics, node-exporter |
|
||
| `loki.yaml` | `HelmRelease` — Loki single-binary, filesystem, compactor-enforced retention |
|
||
| `alloy.yaml` | `HelmRelease` — Grafana Alloy DaemonSet, tails pod logs → Loki |
|
||
| `alerts.yaml` | Platform-health `PrometheusRule`s (R-OBS-04) |
|
||
| `tests/` | `promtool test rules` unit tests over the committed `PrometheusRule`s — each rule pinned quiet on the good state and firing on the bad one (`tests/run.sh`, or `make -C ci prom-rules`) |
|
||
| `kustomization.yaml` | builds all of the above |
|
||
|
||
## Requirement mapping
|
||
|
||
- **R-OBS-01 (what is monitored):** Prometheus + node-exporter + kube-state-metrics for metrics;
|
||
Alloy → Loki for logs. `serviceMonitorSelectorNilUsesHelmValues: false` so ServiceMonitors from
|
||
every namespace (and those rendered from ServiceDefinitions, docs/13) are scraped by default.
|
||
- **R-OBS-02 (alert delivery):** Alertmanager routes by `severity` (`critical` vs `warning`) to
|
||
**email**. SMTP creds are a SOPS secret referenced **by name only** (see below).
|
||
- **R-OBS-03 (dashboards):** Grafana, with the dashboard/datasource sidecar watching
|
||
`grafana_dashboard`-labeled ConfigMaps across all namespaces (per-service dashboards land here).
|
||
Grafana is part of the C2 UI composition (docs/12), later behind Keycloak SSO (ADR-0014).
|
||
- **R-OBS-04 (platform health):** `alerts.yaml` — first-class alerts for **nodes**, **Longhorn**,
|
||
**Flux**, **cert-manager**, **networking/VPN/Hubble**, **Velero backups**, **Keycloak/IdP**, and
|
||
the **monitor's own capacity**. Split `critical`/`warning` per the docs/10 severity model.
|
||
|
||
## SMTP secret (operator hand-off)
|
||
|
||
Alertmanager reads the SMTP **password** from a mounted SOPS-encrypted Secret — referenced by name,
|
||
never inlined. The operator must provide (out of band, SOPS-encrypted per docs/07):
|
||
|
||
```
|
||
kind: Secret
|
||
metadata:
|
||
name: alertmanager-smtp # <-- referenced in kube-prometheus-stack.yaml (alertmanagerSpec.secrets)
|
||
namespace: monitoring
|
||
stringData:
|
||
smtp-password: "<app password>" # mounted at /etc/alertmanager/secrets/alertmanager-smtp/smtp-password
|
||
```
|
||
|
||
The **non-secret** SMTP fields (`smtp_smarthost`, `smtp_from`, `smtp_auth_username`, and the
|
||
receiver `to:` address) are `REPLACE-ME` placeholders in `kube-prometheus-stack.yaml` for the
|
||
operator to fill (same convention as `velero/`). Grafana's admin bootstrap likewise references a
|
||
SOPS secret **by name** — `grafana-admin-credentials` (keys `admin-user` / `admin-password`).
|
||
|
||
## Dev trims (the ~22 GiB RAM ceiling)
|
||
|
||
This subtree is authored to stay light on the single dev workstation:
|
||
|
||
- **Loki `chunksCache`/`resultsCache` DISABLED** — the grafana/loki chart's default memcached caches
|
||
request **~8-9 GiB each**; leaving them on would blow the budget by itself. Also disabled: the
|
||
nginx `gateway`, `lokiCanary`, and `test` hooks. Loki runs **SingleBinary** (one pod).
|
||
- **Single replicas** everywhere (Prometheus, Alertmanager, Grafana, Loki).
|
||
- **Modest requests/limits** on every component (see values). Ephemeral (emptyDir) state for
|
||
Alertmanager and Grafana in dev.
|
||
- **k3s control-plane scrape jobs disabled** (`kubeControllerManager`/`kubeScheduler`/`kubeEtcd`/
|
||
`kubeProxy`) — embedded in k3s, not separately scrapable; keeps targets green and RAM down.
|
||
|
||
Approximate steady-state **memory requests** (what the scheduler reserves):
|
||
|
||
| Component | Req | Notes |
|
||
|---|---:|---|
|
||
| Prometheus | 384Mi | limit 1Gi |
|
||
| Alertmanager | 32Mi | |
|
||
| Grafana | 128Mi | |
|
||
| prometheus-operator | 96Mi | |
|
||
| kube-state-metrics | 64Mi | |
|
||
| node-exporter ×3 | 96Mi | 32Mi/node DaemonSet |
|
||
| Loki (single binary) | 192Mi | limit 512Mi |
|
||
| Alloy ×3 | 288Mi | 96Mi/node DaemonSet |
|
||
| **Total requests** | **~1.3 GiB** | working set typically ~1.5-2.5 GiB; limit ceiling ~3 GiB |
|
||
|
||
Comfortably within the 22 GiB box alongside the rest of the platform.
|
||
|
||
## Environment knobs (R-ENV-02)
|
||
|
||
Only three substitution vars touch this subtree; the dev overlay (`clusters/dev`) patches
|
||
`cluster-vars` to dev values:
|
||
|
||
| Var | base | dev |
|
||
|---|---|---|
|
||
| `${MONITORING_PROMETHEUS_RETENTION}` | `7d` | `3d` |
|
||
| `${MONITORING_PROMETHEUS_PVC_SIZE}` | `10Gi` | `8Gi` |
|
||
| `${MONITORING_PROMETHEUS_RETENTION_SIZE}` | `8GiB` | `5GiB` |
|
||
| `${MONITORING_LOKI_PVC_SIZE}` | `10Gi` | `5Gi` |
|
||
|
||
`MONITORING_PROMETHEUS_RETENTION_SIZE` is a **size-based** retention cap (Prometheus `*iB` units,
|
||
distinct from the K8s `Gi` PVC knob) and must stay **below** `MONITORING_PROMETHEUS_PVC_SIZE`. Time-only
|
||
retention prunes nothing until the age window elapses, so a young/busy cluster fills `/prometheus` first;
|
||
a full disk wedges every WAL commit (scrapes still 200 but samples are dropped, the head freezes) and
|
||
makes every `absent()`/`==0` platform alert fire falsely. The size cap drops the oldest blocks before the
|
||
disk fills. (prod overlay: PVC `50Gi`, cap `45GiB`.)
|
||
|
||
PVCs use the replicated `longhorn` StorageClass (ADR-0003).
|
||
|
||
## Notes / known gaps for bring-up
|
||
|
||
- **Loki log retention has no dedicated knob.** Only the three vars above exist. Loki's
|
||
`retention_period` is hardcoded to `72h` (compactor-enforced) to mirror dev metrics retention.
|
||
Promote to a `cluster-vars` knob (in `clusters/`, out of this subtree) if per-env log retention is
|
||
wanted.
|
||
- Several R-OBS-04 rules depend on exporters/ServiceMonitors owned by **other** workstreams (Flux,
|
||
cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble). Rules are authored to fire once those
|
||
series arrive; metric names marked `TODO(WS-*)` in `alerts.yaml` (notably the **egress VPN**
|
||
metric) need confirming against the deployed exporter.
|
||
- **Hubble** network-flow visibility itself is delivered by `platform/cilium` (docs/03); this subtree
|
||
only consumes its metrics for denied-flow alerts.
|
||
|
||
## Validate (no deploy)
|
||
|
||
```
|
||
kubectl kustomize platform/monitoring
|
||
```
|
||
|
||
Author-only — nothing here is applied to a cluster; Flux reconciles it at deploy time.
|