jakman85andClaude Opus 5
78d0f1b0f6
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.
PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:
kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
-> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless)
ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.
NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.
ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern b9f8807 had just removed from ci/l0.sh, and the
declarative replacement gives what a hand-written stage cannot — selection by
`conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a
requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence,
and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3.
The scope declaration is load-bearing here: touched-requirements.sh keys apps/* to nothing
and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule
would otherwise be gated by nothing.
The check asserts the pins and their lockstep, that the image bakes and records promtool,
that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine
still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that
every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting
a synthetic rule through it twice with one expectation mutated.
MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of
58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests
exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG
tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only
register, printed with counts every run (13 alert files nothing tests at all; 35 named
alerts inside covered files that nothing asserts on), and it cannot grow: a new
PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale
entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one
genuine environment gap, promtool being absent.
PROVEN, in the rebuilt image (promtool 3.14.0):
* both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS.
* the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files.
* NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it
red (pending expected at 25m/29m, got firing), reverted. Adding an untested
PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet,
removed. The harness's own mutation self-test runs on every invocation.
* the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99
--include-scope whole-repo --print-selection` lists alert-rules-unit-tested.
Co-Authored-By: Claude Opus 5 <[email protected]>
platform/monitoring
Observability stack — kube-prometheus-stack (Prometheus + Alertmanager + Grafana) + Loki + Grafana Alloy + platform-health PrometheusRules.
- Spec: docs/10 — Observability & Alerting (named in docs/01; no dedicated ADR)
- Serves: R-OBS-01/02/03/04 (supports R-C2-02, R-SEC-03, R-STATE-03)
- Owner: WS-OBS
Contents
| File | What |
|---|---|
namespace.yaml |
monitoring namespace |
helmrepositories.yaml |
prometheus-community + grafana Flux HelmRepositorys |
kube-prometheus-stack.yaml |
HelmRelease — Prometheus, Alertmanager (→ SMTP email), Grafana, kube-state-metrics, node-exporter |
loki.yaml |
HelmRelease — Loki single-binary, filesystem, compactor-enforced retention |
alloy.yaml |
HelmRelease — Grafana Alloy DaemonSet, tails pod logs → Loki |
alerts.yaml |
Platform-health PrometheusRules (R-OBS-04) |
tests/ |
promtool test rules unit tests over the committed PrometheusRules — each rule pinned quiet on the good state and firing on the bad one (tests/run.sh — the repo-wide harness, it also runs other components' companion *.promtool-test.yaml; enforced by the L0 check alert-rules-unit-tested) |
kustomization.yaml |
builds all of the above |
Requirement mapping
- R-OBS-01 (what is monitored): Prometheus + node-exporter + kube-state-metrics for metrics;
Alloy → Loki for logs.
serviceMonitorSelectorNilUsesHelmValues: falseso ServiceMonitors from every namespace (and those rendered from ServiceDefinitions, docs/13) are scraped by default. - R-OBS-02 (alert delivery): Alertmanager routes by
severity(criticalvswarning) to email. SMTP creds are a SOPS secret referenced by name only (see below). - R-OBS-03 (dashboards): Grafana, with the dashboard/datasource sidecar watching
grafana_dashboard-labeled ConfigMaps across all namespaces (per-service dashboards land here). Grafana is part of the C2 UI composition (docs/12), later behind Keycloak SSO (ADR-0014). - R-OBS-04 (platform health):
alerts.yaml— first-class alerts for nodes, Longhorn, Flux, cert-manager, networking/VPN/Hubble, Velero backups, Keycloak/IdP, and the monitor's own capacity. Splitcritical/warningper the docs/10 severity model.
SMTP secret (operator hand-off)
Alertmanager reads the SMTP password from a mounted SOPS-encrypted Secret — referenced by name, never inlined. The operator must provide (out of band, SOPS-encrypted per docs/07):
kind: Secret
metadata:
name: alertmanager-smtp # <-- referenced in kube-prometheus-stack.yaml (alertmanagerSpec.secrets)
namespace: monitoring
stringData:
smtp-password: "<app password>" # mounted at /etc/alertmanager/secrets/alertmanager-smtp/smtp-password
The non-secret SMTP fields (smtp_smarthost, smtp_from, smtp_auth_username, and the
receiver to: address) are REPLACE-ME placeholders in kube-prometheus-stack.yaml for the
operator to fill (same convention as velero/). Grafana's admin bootstrap likewise references a
SOPS secret by name — grafana-admin-credentials (keys admin-user / admin-password).
Dev trims (the ~22 GiB RAM ceiling)
This subtree is authored to stay light on the single dev workstation:
- Loki
chunksCache/resultsCacheDISABLED — the grafana/loki chart's default memcached caches request ~8-9 GiB each; leaving them on would blow the budget by itself. Also disabled: the nginxgateway,lokiCanary, andtesthooks. Loki runs SingleBinary (one pod). - Single replicas everywhere (Prometheus, Alertmanager, Grafana, Loki).
- Modest requests/limits on every component (see values). Ephemeral (emptyDir) state for Alertmanager and Grafana in dev.
- k3s control-plane scrape jobs disabled (
kubeControllerManager/kubeScheduler/kubeEtcd/kubeProxy) — embedded in k3s, not separately scrapable; keeps targets green and RAM down.
Approximate steady-state memory requests (what the scheduler reserves):
| Component | Req | Notes |
|---|---|---|
| Prometheus | 384Mi | limit 1Gi |
| Alertmanager | 32Mi | |
| Grafana | 128Mi | |
| prometheus-operator | 96Mi | |
| kube-state-metrics | 64Mi | |
| node-exporter ×3 | 96Mi | 32Mi/node DaemonSet |
| Loki (single binary) | 192Mi | limit 512Mi |
| Alloy ×3 | 288Mi | 96Mi/node DaemonSet |
| Total requests | ~1.3 GiB | working set typically ~1.5-2.5 GiB; limit ceiling ~3 GiB |
Comfortably within the 22 GiB box alongside the rest of the platform.
Environment knobs (R-ENV-02)
Only three substitution vars touch this subtree; the dev overlay (clusters/dev) patches
cluster-vars to dev values:
| Var | base | dev |
|---|---|---|
${MONITORING_PROMETHEUS_RETENTION} |
7d |
3d |
${MONITORING_PROMETHEUS_PVC_SIZE} |
10Gi |
8Gi |
${MONITORING_PROMETHEUS_RETENTION_SIZE} |
8GiB |
5GiB |
${MONITORING_LOKI_PVC_SIZE} |
10Gi |
5Gi |
MONITORING_PROMETHEUS_RETENTION_SIZE is a size-based retention cap (Prometheus *iB units,
distinct from the K8s Gi PVC knob) and must stay below MONITORING_PROMETHEUS_PVC_SIZE. Time-only
retention prunes nothing until the age window elapses, so a young/busy cluster fills /prometheus first;
a full disk wedges every WAL commit (scrapes still 200 but samples are dropped, the head freezes) and
makes every absent()/==0 platform alert fire falsely. The size cap drops the oldest blocks before the
disk fills. (prod overlay: PVC 50Gi, cap 45GiB.)
PVCs use the replicated longhorn StorageClass (ADR-0003).
Notes / known gaps for bring-up
- Loki log retention has no dedicated knob. Only the three vars above exist. Loki's
retention_periodis hardcoded to72h(compactor-enforced) to mirror dev metrics retention. Promote to acluster-varsknob (inclusters/, out of this subtree) if per-env log retention is wanted. - Several R-OBS-04 rules depend on exporters/ServiceMonitors owned by other workstreams (Flux,
cert-manager, Longhorn, Velero, Keycloak, Cilium/Hubble). Rules are authored to fire once those
series arrive; metric names marked
TODO(WS-*)inalerts.yaml(notably the egress VPN metric) need confirming against the deployed exporter. - Hubble network-flow visibility itself is delivered by
platform/cilium(docs/03); this subtree only consumes its metrics for denied-flow alerts.
Validate (no deploy)
kubectl kustomize platform/monitoring
Author-only — nothing here is applied to a cluster; Flux reconciles it at deploy time.