home-network-k3s-impl/runbooks/cnpg-exporter.md

# cnpg-exporter — The CNPG metrics collector is erroring (a database is invisible)

**Alert:** `CNPGMetricsCollectorErroring` (warning, 6h)
**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
**Expression:** `max by (namespace, pod) (cnpg_last_error) == 1`

## What this means here

**This is the monitor's own blind spot, for databases — a failure that looks exactly like health.**

CloudNativePG publishes **two independent metric families** and only one of them depends on SQL:

- `cnpg_collector_*` comes from the instance manager and is **always** present.
- `cnpg_pg_stat_*` (archiver, database, replication) comes from the exporter logging in to the
  application database as the `cnpg_metrics_exporter` role.

When that login fails, `cnpg_last_error` goes to 1 and **the entire `cnpg_pg_stat_*` family blanks for
that instance — while `up` stays 1 and `cnpg_collector_up` stays 1.** Nothing looks broken. The scrape
target is green. The dashboards have gaps that read as "no data" rather than "broken".

**This is not hypothetical, and it shaped the whole alert file.** Measured live on 2026-09-06 on both
clusters, `cnpg_errors_total` carried:

```
pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter database=nextcloud`:
FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
```

`nextcloud-db` had been publishing **no** `pg_stat_archiver` / `pg_stat_database` metrics at all. The
consequence recorded verbatim: *an archiving alert built on `pg_stat_archiver` — the obvious choice —
would have been silently blind to one of three prod databases.* That is why every DR rule in
`platform/cnpg/alerts.yaml` keys on `cnpg_collector_*` instead, and why this rule exists to alert on the
blind spot itself.

**Scope: `pod` is kept here, deliberately and unchanged.** `cnpg_last_error` is genuinely per-instance —
each instance's own exporter reports its own SQL-collection health — and prod proved the metric
discriminates rather than tracking role: all **three** `nextcloud-db` pods (including primary `-2`)
reported 1 while all six `keycloak`/`semaphore` pods reported 0. There is no primary-only defect to fix,
and per-pod is the right granularity because the remedy is a **GRANT applied per database** and you want
to see it clear on every instance.

`for: 6h` keeps a pod restart or a brief connection blip quiet. Severity stays `warning` so it never
becomes the permanently-firing `critical` this repo has been burned by.

**Live state, measured 2026-09-13:** `max by (namespace,pod) (cnpg_last_error) == 1` returned **empty**
across the 9 instances that publish `cnpg_last_error` — so the `nextcloud` grant appears to have been
addressed. Re-measure rather than assuming either way.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.

## First commands

Which instances are erroring:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s -G --data-urlencode "query=max by (namespace,pod) (cnpg_last_error)" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "pod|\"1\""'
```

**The message itself** — `cnpg_errors_total` carries it in an `errorUserQueries` label. This is the one
command that tells you the cause outright:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s -G --data-urlencode "query=cnpg_errors_total" localhost:19090/api/v1/query | python3 -m json.tool | head -60'
```

Or straight from the instance's metrics endpoint:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n nextcloud-db exec nextcloud-db-2 -c postgres -- \
     curl -s localhost:9187/metrics | grep -E "^cnpg_last_error|^cnpg_errors_total|^cnpg_collector_up" | head -20'
```

Confirm which family is actually missing — this is what makes the blind spot visible:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   for q in "count(cnpg_collector_up)" "count(cnpg_last_error)" "count(cnpg_pg_stat_archiver_archived_count)" "count(cnpg_pg_stat_database_xact_commit)"; do \
     printf "%-50s " "$q"; curl -s -G --data-urlencode "query=$q" localhost:19090/api/v1/query \
       | python3 -c "import json,sys;print([x[\"value\"][1] for x in json.load(sys.stdin)[\"data\"][\"result\"]] or \"ABSENT\")"; done'
```

The exporter role's actual grants:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n nextcloud-db exec nextcloud-db-2 -c postgres -- \
     psql -tAc "select rolname, rolcanlogin from pg_roles where rolname = \"cnpg_metrics_exporter\";" \
     -c "select datname, has_database_privilege(\"cnpg_metrics_exporter\", datname, \"CONNECT\") from pg_database where datistemplate = false;"'
```

## Likely causes, most likely first

### 1. The exporter role lacks CONNECT on the application database
The recorded cause. Signature: `SQLSTATE 42501`, `permission denied for database "<name>"` in
`cnpg_errors_total`. Remedy: **grant the `cnpg_metrics_exporter` role CONNECT on the application
database.** Do it declaratively — CNPG can manage roles and the `managed.roles` / bootstrap surface in the
Cluster spec is where this belongs, so the grant survives a re-seed. A `psql` GRANT by hand fixes the
symptom on one instance and vanishes on the next bootstrap.

### 2. Custom monitoring queries reference something that does not exist
CNPG lets a Cluster attach extra collector queries via a ConfigMap. A query against a missing table,
extension or column fails the whole collection pass for that instance. `cnpg_errors_total`'s label names
the failing query.

### 3. The database is not accepting connections
A read-only/recovery state, `max_connections` exhausted, or the instance mid-restart. Cross-read
`/runbooks/keycloak-db` if the instance is `keycloak-db-*`. Note the deliberate setting
`dataDurability: preferred` (never `required`) exists so a degraded standby cannot block writes here.

### 4. An upgrade changed the exporter's expectations
CNPG **1.30.0** is the verified operator version on both clusters. A major operator or Postgres bump can
change the collector's queries; if this started at an upgrade, read `/runbooks/update-rollback`.

### 5. The instance is fine and the whole scrape is gone
Then `up` for that `job` would be 0 and this is the wrong runbook — `/runbooks/targets-down`. The precise
value of *this* alert is the case where `up` is **1** and the data is still missing.

## When to escalate or stop

- **Understand what you are currently blind to.** While this fires, `cnpg_pg_stat_archiver`,
  `cnpg_pg_stat_database` and `cnpg_pg_stat_replication` are missing for that instance. The DR alerts in
  `platform/cnpg/alerts.yaml` are deliberately built on `cnpg_collector_*` so they still work — **but do
  not build any new alert or dashboard on `pg_stat_*` without checking this first.** That is the trap
  this alert exists to mark.
- **Do not silence it.** It is a warning with a 6h `for` precisely so it can be left visible without
  becoming noise, and it was a **true positive** for months.
- **Do not grant the exporter role broad privileges** to make it go away. CONNECT on the application
  database is the whole requirement; anything more widens the blast radius of a metrics credential.
- **Do not restart the database** to clear it — the error is a grant, not a state.
- Escalate to the operator if the fix requires changing the Cluster's managed roles or bootstrap, since
  that touches the database's declarative definition and its DR path.

---
Source of truth: `runbooks/cnpg-exporter.md`. Index: `/runbooks/`
Records: `platform/cnpg/alerts.yaml` (the 2026-09-06 `SQLSTATE 42501` measurement on `nextcloud-db`, the
two-metric-families finding, and the 2026-09-12 scope audit that deliberately kept `pod`).