jakman85andClaude Opus 5
4687521d65
LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.
DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.
AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
count(longhorn_volume_robustness == 2) => (empty)
count(longhorn_volume_robustness == 3) => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.
The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.
PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.
REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.
Co-Authored-By: Claude Opus 5 <[email protected]>
platform/monitoring
Observability stack — kube-prometheus-stack (Prometheus + Alertmanager + Grafana) + Loki + Grafana Alloy + platform-health PrometheusRules.
- Spec: docs/10 — Observability & Alerting (named in docs/01; no dedicated ADR)
- Serves: R-OBS-01/02/03/04 (supports R-C2-02, R-SEC-03, R-STATE-03)
- Owner: WS-OBS
Contents
| File | What |
|---|---|
namespace.yaml |
monitoring namespace |
helmrepositories.yaml |
prometheus-community + grafana Flux HelmRepositorys |
kube-prometheus-stack.yaml |
HelmRelease — Prometheus, Alertmanager (→ SMTP email), Grafana, kube-state-metrics, node-exporter |
loki.yaml |
HelmRelease — Loki single-binary, filesystem, compactor-enforced retention |
alloy.yaml |
HelmRelease — Grafana Alloy DaemonSet, tails pod logs → Loki |
alerts.yaml |
Platform-health PrometheusRules (R-OBS-04) |
tests/ |
promtool test rules unit tests over the committed PrometheusRules — each rule pinned quiet on the good state and firing on the bad one (tests/run.sh — the repo-wide harness, it also runs other components' companion *.promtool-test.yaml; enforced by the L0 check alert-rules-unit-tested) |
kustomization.yaml |
builds all of the above |
Requirement mapping
- R-OBS-01 (what is monitored): Prometheus + node-exporter + kube-state-metrics for metrics;
Alloy → Loki for logs.
serviceMonitorSelectorNilUsesHelmValues: falseso ServiceMonitors from every namespace (and those rendered from ServiceDefinitions, docs/13) are scraped by default. - R-OBS-02 (alert delivery): Alertmanager routes by
severity(criticalvswarning) to email. SMTP creds are a SOPS secret referenced by name only (see below). - R-OBS-03 (dashboards): Grafana, with the dashboard/datasource sidecar watching
grafana_dashboard-labeled ConfigMaps across all namespaces (per-service dashboards land here). Grafana is part of the C2 UI composition (docs/12), later behind Keycloak SSO (ADR-0014). - R-OBS-04 (platform health):
alerts.yaml— first-class alerts for nodes, Longhorn, Flux, cert-manager, networking/VPN/Hubble, Velero backups, Keycloak/IdP, and the monitor's own capacity. Splitcritical/warningper the docs/10 severity model.
SMTP secret (operator hand-off)
Alertmanager reads the SMTP password from a mounted SOPS-encrypted Secret — referenced by name, never inlined. The operator must provide (out of band, SOPS-encrypted per docs/07):
kind: Secret
metadata:
name: alertmanager-smtp # <-- referenced in kube-prometheus-stack.yaml (alertmanagerSpec.secrets)
namespace: monitoring
stringData:
smtp-password: "<app password>" # mounted at /etc/alertmanager/secrets/alertmanager-smtp/smtp-password
The non-secret SMTP fields (smtp_smarthost, smtp_from, smtp_auth_username, and the
receiver to: address) are REPLACE-ME placeholders in kube-prometheus-stack.yaml for the
operator to fill (same convention as velero/). Grafana's admin bootstrap likewise references a
SOPS secret by name — grafana-admin-credentials (keys admin-user / admin-password).
Dev trims (the ~22 GiB RAM ceiling)
This subtree is authored to stay light on the single dev workstation:
- Loki
chunksCache/resultsCacheDISABLED — the grafana/loki chart's default memcached caches request ~8-9 GiB each; leaving them on would blow the budget by itself. Also disabled: the nginxgateway,lokiCanary, andtesthooks. Loki runs SingleBinary (one pod). - Single replicas everywhere (Prometheus, Alertmanager, Grafana, Loki).
- Modest requests/limits on every component (see values). Ephemeral (emptyDir) state for Alertmanager and Grafana in dev.
- k3s control-plane scrape jobs disabled (
kubeControllerManager/kubeScheduler/kubeEtcd/kubeProxy) — embedded in k3s, not separately scrapable; keeps targets green and RAM down.
Approximate steady-state memory requests (what the scheduler reserves):
| Component | Req | Notes |
|---|---|---|
| Prometheus | 384Mi | limit 1Gi |
| Alertmanager | 32Mi | |
| Grafana | 128Mi | |
| prometheus-operator | 96Mi | |
| kube-state-metrics | 64Mi | |
| node-exporter ×3 | 96Mi | 32Mi/node DaemonSet |
| Loki (single binary) | 192Mi | limit 512Mi |
| Alloy ×3 | 288Mi | 96Mi/node DaemonSet |
| Total requests | ~1.3 GiB | working set typically ~1.5-2.5 GiB; limit ceiling ~3 GiB |
Comfortably within the 22 GiB box alongside the rest of the platform.
Environment knobs (R-ENV-02)
Only three substitution vars touch this subtree; the dev overlay (clusters/dev) patches
cluster-vars to dev values:
| Var | base | dev |
|---|---|---|
${MONITORING_PROMETHEUS_RETENTION} |
7d |
3d |
${MONITORING_PROMETHEUS_PVC_SIZE} |
10Gi |
8Gi |
${MONITORING_PROMETHEUS_RETENTION_SIZE} |
8GiB |
5GiB |
${MONITORING_LOKI_PVC_SIZE} |
10Gi |
5Gi |
MONITORING_PROMETHEUS_RETENTION_SIZE is a size-based retention cap (Prometheus *iB units,
distinct from the K8s Gi PVC knob) and must stay below MONITORING_PROMETHEUS_PVC_SIZE. Time-only
retention prunes nothing until the age window elapses, so a young/busy cluster fills /prometheus first;
a full disk wedges every WAL commit (scrapes still 200 but samples are dropped, the head freezes) and
makes every absent()/==0 platform alert fire falsely. The size cap drops the oldest blocks before the
disk fills. (prod overlay: PVC 50Gi, cap 45GiB.)
PVCs use the replicated longhorn StorageClass (ADR-0003).
Notes / known gaps for bring-up
- Loki log retention has no dedicated knob. Only the three vars above exist. Loki's
retention_periodis hardcoded to72h(compactor-enforced) to mirror dev metrics retention. Promote to acluster-varsknob (inclusters/, out of this subtree) if per-env log retention is wanted. - Several R-OBS-04 rules depend on exporters/ServiceMonitors owned by other workstreams (Flux,
cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble). Rules are authored to fire once those
series arrive; metric names marked
TODO(WS-*)inalerts.yaml(notably the egress VPN metric) need confirming against the deployed exporter. - Hubble network-flow visibility itself is delivered by
platform/cilium(docs/03); this subtree only consumes its metrics for denied-flow alerts.
Validate (no deploy)
kubectl kustomize platform/monitoring
Author-only — nothing here is applied to a cluster; Flux reconciles it at deploy time.