home-network-k3s-impl/runbooks/cnpg-exporter.md

cnpg-exporter — The CNPG metrics collector is erroring (a database is invisible)

Alert: CNPGMetricsCollectorErroring (warning, 6h) File: platform/cnpg/alerts.yaml, group platform-health.cnpg Expression: max by (namespace, pod) (cnpg_last_error) == 1

What this means here

This is the monitor's own blind spot, for databases — a failure that looks exactly like health.

CloudNativePG publishes two independent metric families and only one of them depends on SQL:

• cnpg_collector_* comes from the instance manager and is always present.
• cnpg_pg_stat_* (archiver, database, replication) comes from the exporter logging in to the application database as the cnpg_metrics_exporter role.

When that login fails, cnpg_last_error goes to 1 and the entire cnpg_pg_stat_* family blanks for that instance — while up stays 1 and cnpg_collector_up stays 1. Nothing looks broken. The scrape target is green. The dashboards have gaps that read as "no data" rather than "broken".

This is not hypothetical, and it shaped the whole alert file. Measured live on 2026-09-06 on both clusters, cnpg_errors_total carried:

pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter database=nextcloud`:
FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)

nextcloud-db had been publishing no pg_stat_archiver / pg_stat_database metrics at all. The consequence recorded verbatim: an archiving alert built on pg_stat_archiver — the obvious choice — would have been silently blind to one of three prod databases. That is why every DR rule in platform/cnpg/alerts.yaml keys on cnpg_collector_* instead, and why this rule exists to alert on the blind spot itself.

Scope: pod is kept here, deliberately and unchanged. cnpg_last_error is genuinely per-instance — each instance's own exporter reports its own SQL-collection health — and prod proved the metric discriminates rather than tracking role: all three nextcloud-db pods (including primary -2) reported 1 while all six keycloak/semaphore pods reported 0. There is no primary-only defect to fix, and per-pod is the right granularity because the remedy is a GRANT applied per database and you want to see it clear on every instance.

for: 6h keeps a pod restart or a brief connection blip quiet. Severity stays warning so it never becomes the permanently-firing critical this repo has been burned by.

Live state, measured 2026-09-13: max by (namespace,pod) (cnpg_last_error) == 1 returned empty across the 9 instances that publish cnpg_last_error — so the nextcloud grant appears to have been addressed. Re-measure rather than assuming either way.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git. main is a protected branch — a direct push is rejected. A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Which instances are erroring:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s -G --data-urlencode "query=max by (namespace,pod) (cnpg_last_error)" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "pod|\"1\""'

The message itself — cnpg_errors_total carries it in an errorUserQueries label. This is the one command that tells you the cause outright:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s -G --data-urlencode "query=cnpg_errors_total" localhost:19090/api/v1/query | python3 -m json.tool | head -60'

Or straight from the instance's metrics endpoint:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n nextcloud-db exec nextcloud-db-2 -c postgres -- \
     curl -s localhost:9187/metrics | grep -E "^cnpg_last_error|^cnpg_errors_total|^cnpg_collector_up" | head -20'

Confirm which family is actually missing — this is what makes the blind spot visible:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   for q in "count(cnpg_collector_up)" "count(cnpg_last_error)" "count(cnpg_pg_stat_archiver_archived_count)" "count(cnpg_pg_stat_database_xact_commit)"; do \
     printf "%-50s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \
       | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done'

The exporter role's actual grants:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n nextcloud-db exec nextcloud-db-2 -c postgres -- \
     psql -tAc "select rolname, rolcanlogin from pg_roles where rolname = \"cnpg_metrics_exporter\";" \
     -c "select datname, has_database_privilege(\"cnpg_metrics_exporter\", datname, \"CONNECT\") from pg_database where datistemplate = false;"'

Likely causes, most likely first

1. The exporter role lacks CONNECT on the application database

The recorded cause. Signature: SQLSTATE 42501, permission denied for database "<name>" in cnpg_errors_total. Remedy: grant the cnpg_metrics_exporter role CONNECT on the application database. Do it declaratively — CNPG can manage roles and the managed.roles / bootstrap surface in the Cluster spec is where this belongs, so the grant survives a re-seed. A psql GRANT by hand fixes the symptom on one instance and vanishes on the next bootstrap.

2. Custom monitoring queries reference something that does not exist

CNPG lets a Cluster attach extra collector queries via a ConfigMap. A query against a missing table, extension or column fails the whole collection pass for that instance. cnpg_errors_total's label names the failing query.

3. The database is not accepting connections

A read-only/recovery state, max_connections exhausted, or the instance mid-restart. Cross-read /runbooks/keycloak-db if the instance is keycloak-db-*. Note the deliberate setting dataDurability: preferred (never required) exists so a degraded standby cannot block writes here.

4. An upgrade changed the exporter's expectations

CNPG 1.30.0 is the verified operator version on both clusters. A major operator or Postgres bump can change the collector's queries; if this started at an upgrade, read /runbooks/update-rollback.

5. The instance is fine and the whole scrape is gone

Then up for that job would be 0 and this is the wrong runbook — /runbooks/targets-down. The precise value of this alert is the case where up is 1 and the data is still missing.

When to escalate or stop

• Understand what you are currently blind to. While this fires, cnpg_pg_stat_archiver, cnpg_pg_stat_database and cnpg_pg_stat_replication are missing for that instance. The DR alerts in platform/cnpg/alerts.yaml are deliberately built on cnpg_collector_* so they still work — but do not build any new alert or dashboard on pg_stat_* without checking this first. That is the trap this alert exists to mark.
• Do not silence it. It is a warning with a 6h for precisely so it can be left visible without becoming noise, and it was a true positive for months.
• Do not grant the exporter role broad privileges to make it go away. CONNECT on the application database is the whole requirement; anything more widens the blast radius of a metrics credential.
• Do not restart the database to clear it — the error is a grant, not a state.
• Escalate to the operator if the fix requires changing the Cluster's managed roles or bootstrap, since that touches the database's declarative definition and its DR path.

---

Source of truth: runbooks/cnpg-exporter.md. Index: /runbooks/ Records: platform/cnpg/alerts.yaml (the 2026-09-06 SQLSTATE 42501 measurement on nextcloud-db, the two-metric-families finding, and the 2026-09-12 scope audit that deliberately kept pod).