home-network-k3s-impl/platform/monitoring

platform/monitoring

Observability stack — kube-prometheus-stack (Prometheus + Alertmanager + Grafana) + Loki + Grafana Alloy + platform-health PrometheusRules.

• Spec: docs/10 — Observability & Alerting (named in docs/01; no dedicated ADR)
• Serves: R-OBS-01/02/03/04 (supports R-C2-02, R-SEC-03, R-STATE-03)
• Owner: WS-OBS

Contents

File	What
namespace.yaml	monitoring namespace
helmrepositories.yaml	prometheus-community + grafana Flux HelmRepositorys
kube-prometheus-stack.yaml	HelmRelease — Prometheus, Alertmanager (→ SMTP email), Grafana, kube-state-metrics, node-exporter
loki.yaml	HelmRelease — Loki single-binary, filesystem, compactor-enforced retention
alloy.yaml	HelmRelease — Grafana Alloy DaemonSet, tails pod logs → Loki
alerts.yaml	Platform-health PrometheusRules (R-OBS-04)
tests/	promtool test rules unit tests over the committed PrometheusRules — each rule pinned quiet on the good state and firing on the bad one (tests/run.sh — the repo-wide harness, it also runs other components' companion *.promtool-test.yaml; enforced by the L0 check alert-rules-unit-tested)
kustomization.yaml	builds all of the above

Requirement mapping

• R-OBS-01 (what is monitored): Prometheus + node-exporter + kube-state-metrics for metrics; Alloy → Loki for logs. serviceMonitorSelectorNilUsesHelmValues: false so ServiceMonitors from every namespace (and those rendered from ServiceDefinitions, docs/13) are scraped by default.
• R-OBS-02 (alert delivery): Alertmanager routes by severity (critical vs warning) to email. SMTP creds are a SOPS secret referenced by name only (see below).
• R-OBS-03 (dashboards): Grafana, with the dashboard/datasource sidecar watching grafana_dashboard-labeled ConfigMaps across all namespaces (per-service dashboards land here). Grafana is part of the C2 UI composition (docs/12), later behind Keycloak SSO (ADR-0014).
• R-OBS-04 (platform health): alerts.yaml — first-class alerts for nodes, Longhorn, Flux, cert-manager, networking/VPN/Hubble, Velero backups, Keycloak/IdP, and the monitor's own capacity. Split critical/warning per the docs/10 severity model.

SMTP secret (operator hand-off)

Alertmanager reads the SMTP password from a mounted SOPS-encrypted Secret — referenced by name, never inlined. The operator must provide (out of band, SOPS-encrypted per docs/07):

kind: Secret
metadata:
  name: alertmanager-smtp        # <-- referenced in kube-prometheus-stack.yaml (alertmanagerSpec.secrets)
  namespace: monitoring
stringData:
  smtp-password: "<app password>"   # mounted at /etc/alertmanager/secrets/alertmanager-smtp/smtp-password

The non-secret SMTP fields (smtp_smarthost, smtp_from, smtp_auth_username, and the receiver to: address) are REPLACE-ME placeholders in kube-prometheus-stack.yaml for the operator to fill (same convention as velero/). Grafana's admin bootstrap likewise references a SOPS secret by name — grafana-admin-credentials (keys admin-user / admin-password).

Dev trims (the ~22 GiB RAM ceiling)

This subtree is authored to stay light on the single dev workstation:

• Loki chunksCache/resultsCache DISABLED — the grafana/loki chart's default memcached caches request ~8-9 GiB each; leaving them on would blow the budget by itself. Also disabled: the nginx gateway, lokiCanary, and test hooks. Loki runs SingleBinary (one pod).
• Single replicas everywhere (Prometheus, Alertmanager, Grafana, Loki).
• Modest requests/limits on every component (see values). Ephemeral (emptyDir) state for Alertmanager and Grafana in dev.
• k3s control-plane scrape jobs disabled (kubeControllerManager/kubeScheduler/kubeEtcd/ kubeProxy) — embedded in k3s, not separately scrapable; keeps targets green and RAM down.

Approximate steady-state memory requests (what the scheduler reserves):

Component	Req	Notes
Prometheus	384Mi	limit 1Gi
Alertmanager	32Mi
Grafana	128Mi
prometheus-operator	96Mi
kube-state-metrics	64Mi
node-exporter ×3	96Mi	32Mi/node DaemonSet
Loki (single binary)	192Mi	limit 512Mi
Alloy ×3	288Mi	96Mi/node DaemonSet
Total requests	~1.3 GiB	working set typically ~1.5-2.5 GiB; limit ceiling ~3 GiB

Comfortably within the 22 GiB box alongside the rest of the platform.

Environment knobs (R-ENV-02)

Only three substitution vars touch this subtree; the dev overlay (clusters/dev) patches cluster-vars to dev values:

Var	base	dev
${MONITORING_PROMETHEUS_RETENTION}	7d	3d
${MONITORING_PROMETHEUS_PVC_SIZE}	10Gi	8Gi
${MONITORING_PROMETHEUS_RETENTION_SIZE}	8GiB	5GiB
${MONITORING_LOKI_PVC_SIZE}	10Gi	5Gi

MONITORING_PROMETHEUS_RETENTION_SIZE is a size-based retention cap (Prometheus *iB units, distinct from the K8s Gi PVC knob) and must stay below MONITORING_PROMETHEUS_PVC_SIZE. Time-only retention prunes nothing until the age window elapses, so a young/busy cluster fills /prometheus first; a full disk wedges every WAL commit (scrapes still 200 but samples are dropped, the head freezes) and makes every absent()/==0 platform alert fire falsely. The size cap drops the oldest blocks before the disk fills. (prod overlay: PVC 50Gi, cap 45GiB.)

PVCs use the replicated longhorn StorageClass (ADR-0003).

Notes / known gaps for bring-up

• Loki log retention has no dedicated knob. Only the three vars above exist. Loki's retention_period is hardcoded to 72h (compactor-enforced) to mirror dev metrics retention. Promote to a cluster-vars knob (in clusters/, out of this subtree) if per-env log retention is wanted.
• Several R-OBS-04 rules depend on exporters/ServiceMonitors owned by other workstreams (Flux, cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble). Rules are authored to fire once those series arrive; metric names marked TODO(WS-*) in alerts.yaml (notably the egress VPN metric) need confirming against the deployed exporter.
• Hubble network-flow visibility itself is delivered by platform/cilium (docs/03); this subtree only consumes its metrics for denied-flow alerts.

Validate (no deploy)

kubectl kustomize platform/monitoring

Author-only — nothing here is applied to a cluster; Flux reconciles it at deploy time.