jakman85andClaude Opus 5
b9f3143f42
NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.
Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.
NodeDiskDeviceSaturated device saturation >0.85 busy, 10m
NodeDiskWriteLatencyHigh device latency >20ms/write, 10m
NodeDiskWriteLatencyCritical device latency >50ms/write, 5m
NodeDiskReadLatencyHigh device latency >250ms/read, 15m
NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m
NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent
NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s
ContainerIOStallHigh which workload per-container PSI >15%
Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.
Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.
etcd and NFS metric availability, verified on prod rather than assumed:
* etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
member instead, and says in its own description that it is a proxy.
* node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
/proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
retransmissions and NO latency, so it can see a dropping export but not a slow
one. The mountstats collector is enabled here to close that gap; NO rule
references node_mountstats_* yet, because the series do not exist until the
HelmRelease rolls out and the collector's visibility through the node-exporter
mount namespace must be verified on prod first. The runbook records the check
and says to revert if it comes back empty.
Proof, per the repo's two shipped alerting failure modes:
* platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
latency, and the `unless` semantics.
* 10 mutations of the shipped rules, all killed. The mutation run found a real
gap: the etcd threshold could be changed without failing any test, because it
was only pinned by a promql_expr_test that restated the threshold in the test
file. An alert_rule_test now pins it.
* All 11 referenced metrics verified to return series on prod with
`promtool query instant`, and all 8 expressions verified quiet on today's state.
Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
platform/monitoring
Observability stack — kube-prometheus-stack (Prometheus + Alertmanager + Grafana) + Loki + Grafana Alloy + platform-health PrometheusRules.
- Spec: docs/10 — Observability & Alerting (named in docs/01; no dedicated ADR)
- Serves: R-OBS-01/02/03/04 (supports R-C2-02, R-SEC-03, R-STATE-03)
- Owner: WS-OBS
Contents
| File | What |
|---|---|
namespace.yaml |
monitoring namespace |
helmrepositories.yaml |
prometheus-community + grafana Flux HelmRepositorys |
kube-prometheus-stack.yaml |
HelmRelease — Prometheus, Alertmanager (→ SMTP email), Grafana, kube-state-metrics, node-exporter |
loki.yaml |
HelmRelease — Loki single-binary, filesystem, compactor-enforced retention |
alloy.yaml |
HelmRelease — Grafana Alloy DaemonSet, tails pod logs → Loki |
alerts.yaml |
Platform-health PrometheusRules (R-OBS-04) |
tests/ |
promtool test rules unit tests over the committed PrometheusRules — each rule pinned quiet on the good state and firing on the bad one (tests/run.sh — the repo-wide harness, it also runs other components' companion *.promtool-test.yaml; enforced by the L0 check alert-rules-unit-tested) |
kustomization.yaml |
builds all of the above |
Requirement mapping
- R-OBS-01 (what is monitored): Prometheus + node-exporter + kube-state-metrics for metrics;
Alloy → Loki for logs.
serviceMonitorSelectorNilUsesHelmValues: falseso ServiceMonitors from every namespace (and those rendered from ServiceDefinitions, docs/13) are scraped by default. - R-OBS-02 (alert delivery): Alertmanager routes by
severity(criticalvswarning) to email. SMTP creds are a SOPS secret referenced by name only (see below). - R-OBS-03 (dashboards): Grafana, with the dashboard/datasource sidecar watching
grafana_dashboard-labeled ConfigMaps across all namespaces (per-service dashboards land here). Grafana is part of the C2 UI composition (docs/12), later behind Keycloak SSO (ADR-0014). - R-OBS-04 (platform health):
alerts.yaml— first-class alerts for nodes, Longhorn, Flux, cert-manager, networking/VPN/Hubble, Velero backups, Keycloak/IdP, and the monitor's own capacity. Splitcritical/warningper the docs/10 severity model.
SMTP secret (operator hand-off)
Alertmanager reads the SMTP password from a mounted SOPS-encrypted Secret — referenced by name, never inlined. The operator must provide (out of band, SOPS-encrypted per docs/07):
kind: Secret
metadata:
name: alertmanager-smtp # <-- referenced in kube-prometheus-stack.yaml (alertmanagerSpec.secrets)
namespace: monitoring
stringData:
smtp-password: "<app password>" # mounted at /etc/alertmanager/secrets/alertmanager-smtp/smtp-password
The non-secret SMTP fields (smtp_smarthost, smtp_from, smtp_auth_username, and the
receiver to: address) are REPLACE-ME placeholders in kube-prometheus-stack.yaml for the
operator to fill (same convention as velero/). Grafana's admin bootstrap likewise references a
SOPS secret by name — grafana-admin-credentials (keys admin-user / admin-password).
Dev trims (the ~22 GiB RAM ceiling)
This subtree is authored to stay light on the single dev workstation:
- Loki
chunksCache/resultsCacheDISABLED — the grafana/loki chart's default memcached caches request ~8-9 GiB each; leaving them on would blow the budget by itself. Also disabled: the nginxgateway,lokiCanary, andtesthooks. Loki runs SingleBinary (one pod). - Single replicas everywhere (Prometheus, Alertmanager, Grafana, Loki).
- Modest requests/limits on every component (see values). Ephemeral (emptyDir) state for Alertmanager and Grafana in dev.
- k3s control-plane scrape jobs disabled (
kubeControllerManager/kubeScheduler/kubeEtcd/kubeProxy) — embedded in k3s, not separately scrapable; keeps targets green and RAM down.
Approximate steady-state memory requests (what the scheduler reserves):
| Component | Req | Notes |
|---|---|---|
| Prometheus | 384Mi | limit 1Gi |
| Alertmanager | 32Mi | |
| Grafana | 128Mi | |
| prometheus-operator | 96Mi | |
| kube-state-metrics | 64Mi | |
| node-exporter ×3 | 96Mi | 32Mi/node DaemonSet |
| Loki (single binary) | 192Mi | limit 512Mi |
| Alloy ×3 | 288Mi | 96Mi/node DaemonSet |
| Total requests | ~1.3 GiB | working set typically ~1.5-2.5 GiB; limit ceiling ~3 GiB |
Comfortably within the 22 GiB box alongside the rest of the platform.
Environment knobs (R-ENV-02)
Only three substitution vars touch this subtree; the dev overlay (clusters/dev) patches
cluster-vars to dev values:
| Var | base | dev |
|---|---|---|
${MONITORING_PROMETHEUS_RETENTION} |
7d |
3d |
${MONITORING_PROMETHEUS_PVC_SIZE} |
10Gi |
8Gi |
${MONITORING_PROMETHEUS_RETENTION_SIZE} |
8GiB |
5GiB |
${MONITORING_LOKI_PVC_SIZE} |
10Gi |
5Gi |
MONITORING_PROMETHEUS_RETENTION_SIZE is a size-based retention cap (Prometheus *iB units,
distinct from the K8s Gi PVC knob) and must stay below MONITORING_PROMETHEUS_PVC_SIZE. Time-only
retention prunes nothing until the age window elapses, so a young/busy cluster fills /prometheus first;
a full disk wedges every WAL commit (scrapes still 200 but samples are dropped, the head freezes) and
makes every absent()/==0 platform alert fire falsely. The size cap drops the oldest blocks before the
disk fills. (prod overlay: PVC 50Gi, cap 45GiB.)
PVCs use the replicated longhorn StorageClass (ADR-0003).
Notes / known gaps for bring-up
- Loki log retention has no dedicated knob. Only the three vars above exist. Loki's
retention_periodis hardcoded to72h(compactor-enforced) to mirror dev metrics retention. Promote to acluster-varsknob (inclusters/, out of this subtree) if per-env log retention is wanted. - Several R-OBS-04 rules depend on exporters/ServiceMonitors owned by other workstreams (Flux,
cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble). Rules are authored to fire once those
series arrive; metric names marked
TODO(WS-*)inalerts.yaml(notably the egress VPN metric) need confirming against the deployed exporter. - Hubble network-flow visibility itself is delivered by
platform/cilium(docs/03); this subtree only consumes its metrics for denied-flow alerts.
Validate (no deploy)
kubectl kustomize platform/monitoring
Author-only — nothing here is applied to a cluster; Flux reconciles it at deploy time.