merge: the nextcloud database has never been monitored

cnpg_metrics_exporter lacked CONNECT on the nextcloud database: an
out-of-band REVOKE materialized its datacl and dropped PUBLIC's implicit
CONNECT, which keycloak and semaphore still inherit. CNPG's API cannot
express a database-level grant, so this follows the repo's existing
level-triggered CronJob reconciler pattern rather than a one-shot Job that
a rebuild would silently skip. CONNECT only; pg_monitor already carries the
read access.

DECISIONS-LOG.md

@@ -2518,3 +2518,46 @@ source. Confirmed live: managedFields moved from `kubectl-client-side-apply` to
   not, and there is no urgency.
 - Testing HTTPS from inside a pod is misleading: nextcloud's Cilium egress policy returned envoy
   `403 Access denied` for other apps' FQDNs. Test from outside with `--resolve`.
+
+---
+
+## 2026-09-13 — `kubectl get backup` LIES on this cluster (ambiguous short name)
+
+**Finding.** `kubectl get backup` / `get backup -A` prints **"No resources found"** on prod while
+backups demonstrably exist. Three API groups each register a `Backup` kind:
+
+| group | kind | populated? |
+|---|---|---|
+| `longhorn.io/v1beta2` | Backup (`lhb`) | **no** — 0 objects |
+| `postgresql.cnpg.io/v1` | Backup | yes — 27 (9 per CNPG cluster) |
+| `velero.io/v1` | Backup | yes |
+
+The unqualified name resolves to Longhorn's, which is empty, so kubectl reports absence and exits 0.
+Nothing warns about the ambiguity.
+
+**Why it matters more than a papercut.** This is a **false negative on the one question you ask
+during a disaster**: "do I have backups?" The honest-looking answer is "No resources found". An
+operator who trusts it concludes backups never ran and may act destructively — re-bootstrapping a
+database, or re-seeding a volume — while good backups sit in R2 the whole time. It is the inverse of
+this repo's recurring "a thing that looks applied and isn't": here, a thing that looks ABSENT and isn't.
+
+I hit it twice in one session and both times my first conclusion was wrong:
+- On CNPG I wrote "(none — no base backup has ever been taken)" and started reasoning about an
+  unrecoverable platform. The Cluster `.status` contradicted me within a minute
+  (`lastSuccessfulBackup` hours old, `firstRecoverabilityPoint` 2026-09-07, `LastBackupSucceeded=True`),
+  and `get backups.postgresql.cnpg.io -A` then showed 27 completed backups.
+- On Velero I read `get backup` as empty while `velero-daily-20260913040106` was *executing*, its
+  PodVolumeBackups logging "Async fs backup completed".
+
+**Rule adopted.** Always use fully-qualified resource names for any kind this cluster multiply
+registers — `backups.velero.io`, `backups.postgresql.cnpg.io`, `backups.longhorn.io`. Never the bare
+short name in a runbook, an alert annotation, a script, or an incident check. The same hazard applies
+to `Cluster` (CNPG vs others) — prefer fully-qualified names generally.
+
+**Corollary for status-checking in general.** `.status` on the owning object was the reliable source
+both times, and the object list was the misleading one. When the two disagree, trust the controller's
+own status conditions and find out why the list is lying before acting.
+
+**Verified same session:** BSL `default` Available (validated 04:13Z); `velero-daily` unpaused and a
+backup kicked off immediately; two Failed backups from 2026-09-12 (`prod-proof-20260912`,
+`prod-vp-probe-20260912`) with orphaned kopia blobs still awaiting owner consent to delete.

apps/wave-0/nextcloud-db.yaml

@@ -158,3 +158,234 @@ spec:
   cluster:
     name: nextcloud-db
   method: barmanObjectStore
+---
+# =============================================================================
+# CNPG METRICS-EXPORTER CONNECT RECONCILER (R-OBS-01)
+#
+# THE BUG THIS CLOSES (true positive, verified live on prod 2026-09-12, and the
+# condition platform/cnpg/alerts.yaml already recorded on BOTH clusters 2026-09-06):
+# the `nextcloud` database's ACL had been narrowed out-of-band so that PUBLIC held
+# nothing, and CNPG's metrics exporter role therefore could not even CONNECT:
+#
+#   nextcloud-db-1/-2/-3, every scrape:
+#     FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
+#     DETAIL: User does not have CONNECT privilege.
+#     msg="Error collecting user query" targetDatabase=nextcloud
+#
+# CONSEQUENCE: `cnpg_last_error` = 1 on all three instances and the ENTIRE
+# cnpg_pg_stat_* / pg_settings / pg_postmaster family MISSING for this database,
+# while the scrape target still reported up=1 — a blind spot that looks like health.
+# Measured before the fix: cnpg_pg_settings_setting had 765 series for keycloak and
+# 765 for semaphore-db and ZERO for nextcloud-db. Nextcloud's database had never
+# been monitored.
+#
+# ROOT CAUSE, AND WHY ONLY THIS DATABASE. All three Cluster specs are identical in
+# their bootstrap; the difference is entirely in the live ACL:
+#
+#   nextcloud : datacl = {nextcloud=CTc/nextcloud}          <-- explicit, no PUBLIC entry
+#   keycloak  : datacl = NULL
+#   semaphore : datacl = NULL
+#
+# A NULL datacl means "PostgreSQL defaults", which include CONNECT and TEMP for
+# PUBLIC — so keycloak and semaphore let the exporter in only by inheriting PUBLIC's
+# implicit CONNECT. On `nextcloud` something ran the equivalent of
+# `REVOKE ALL ON DATABASE nextcloud FROM PUBLIC`, which MATERIALISES datacl and
+# drops the implicit PUBLIC entry with it. That REVOKE is not in this repo and not
+# in DECISIONS-LOG.md — it was applied out of band, which is exactly why a
+# git-declared fix is needed rather than another hand-run psql.
+#
+# WHY THE GRANT IS EXPLICIT RATHER THAN "RE-GRANT CONNECT TO PUBLIC": restoring
+# PUBLIC's CONNECT would undo whatever hardening that REVOKE intended and would
+# leave the exporter depending on an implicit default again — the precise fragility
+# that broke here and that keycloak/semaphore are still relying on. Naming the
+# exporter role makes the dependency explicit and survives a future REVOKE FROM
+# PUBLIC on any of the three databases.
+#
+# WHY A CronJob AND NOT A CNPG FIELD — CNPG 1.30.0's API GENUINELY CANNOT EXPRESS
+# THIS. Checked against the live CRDs on prod:
+#   * `Database` (databases.postgresql.cnpg.io) has no database-ACL field at all.
+#     `allowConnections` is the pg_database.datallowconn FLAG (open/closed to
+#     everyone), not a per-role GRANT; its only grant/revoke verbs are the `usage`
+#     lists on `fdws` and `servers`.
+#   * `spec.managed.roles` and `DatabaseRole` manage ROLE ATTRIBUTES (login,
+#     superuser, connectionLimit, ...) plus `inRoles` MEMBERSHIP. Membership is the
+#     wrong tool: the only local role holding CONNECT here is the OWNER, so
+#     `inRoles: [nextcloud]` would hand the exporter the owner's CREATE and TEMP on
+#     the database and read/write on every application table — the opposite of
+#     least privilege, to buy one CONNECT.
+#   * `bootstrap.initdb.postInitApplicationSQL` runs ONCE, at bootstrap. It cannot
+#     repair these three already-bootstrapped instances (the actual bug), and on a
+#     FRESH cluster the grant is unnecessary anyway (datacl starts NULL). Adding it
+#     would be a second mechanism that fixes nothing this one does not.
+# So the SQL has to be applied by something, and this repo already has the pattern
+# for state Flux cannot reach: the level-triggered CronJob reconciler in
+# platform/keycloak/client-secret-reconciler.yaml, whose reasoning transfers
+# verbatim — the drifted state lives inside PostgreSQL, not in a Kubernetes object,
+# so no admission or mutation policy can see it. A one-shot Job is deliberately NOT
+# used: a completed Job is never re-run, so a DB rebuild or a PITR restore would
+# silently reopen the blind spot. A loop converges instead.
+#
+# PRIVILEGE GRANTED — EXACTLY ONE, AND NOTHING ELSE:
+#   GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
+# Resulting ACL entry is `cnpg_metrics_exporter=c/nextcloud` — the `c` is CONNECT;
+# there is no `C` (CREATE) and no `T` (TEMPORARY). No ownership, no superuser, no
+# write, no table or schema grants. Nothing further is needed because CNPG already
+# creates this role as a member of the built-in `pg_monitor` role (verified on all
+# three prod clusters), and pg_monitor already carries the read access every
+# collector query wants — they read pg_catalog views (pg_stat_database,
+# pg_stat_archiver, pg_settings, pg_postmaster) and no application tables. Proof
+# that CONNECT alone is sufficient: after this grant, `Error collecting user query`
+# went to ZERO on all three pods, i.e. every collector query now succeeds.
+#
+# IT RUNS AS THE DATABASE OWNER, NOT AS A SUPERUSER. `enableSuperuserAccess` is off
+# on this Cluster (there is no nextcloud-db-superuser Secret), and it stays off: the
+# owner of a database holds the grant option on that database's privileges, so
+# `nextcloud` can issue this GRANT itself. Verified live before committing, by
+# running the statement under `SET ROLE nextcloud`. Credentials come from the
+# in-namespace SOPS Secret nextcloud-db-app that already exists for the app.
+#
+# NETWORKING: nextcloud-db is intentionally NOT an app-namespace (see this file's
+# header), so Kyverno generates no default-deny baseline here and this same-namespace
+# pod -> nextcloud-db-rw:5432 connection needs no rendered allow.
+#
+# THE POD LABEL IS DELIBERATELY *NOT* app.kubernetes.io/name=nextcloud-db. That
+# label is what `inheritedMetadata` stamps on the CNPG instance pods and what the
+# rendered nextcloud egress CiliumNetworkPolicy selects; reusing it here would quietly
+# make this reconciler a permitted destination for the Nextcloud app. It gets its own
+# name so that selector keeps meaning "the database".
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: nextcloud-db-exporter-grant
+  namespace: nextcloud-db
+  labels:
+    app.kubernetes.io/name: nextcloud-db-exporter-grant
+    app.kubernetes.io/part-of: nextcloud-db
+spec:
+  # Every 15 minutes, matching the keycloak reconciler's cadence. A converged
+  # database makes this one idempotent GRANT and exits; the cost is negligible, and
+  # after a rebuild or PITR restore monitoring is blind for at most one interval
+  # (well inside the alert's `for: 6h`).
+  schedule: "*/15 * * * *"
+  # A run still waiting on the primary must not be lapped by the next tick.
+  concurrencyPolicy: Forbid
+  startingDeadlineSeconds: 300
+  successfulJobsHistoryLimit: 1
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 2
+      activeDeadlineSeconds: 300
+      ttlSecondsAfterFinished: 3600
+      template:
+        metadata:
+          labels:
+            app.kubernetes.io/name: nextcloud-db-exporter-grant
+            app.kubernetes.io/part-of: nextcloud-db
+        spec:
+          restartPolicy: Never
+          # Talks only to PostgreSQL; its one input is a mounted Secret, so it needs
+          # no Kubernetes API access at all (R-SEC-02, least privilege).
+          automountServiceAccountToken: false
+          securityContext:
+            runAsNonRoot: true
+            # uid/gid 26 is `postgres` in the CNPG PostgreSQL image.
+            runAsUser: 26
+            runAsGroup: 26
+            seccompProfile:
+              type: RuntimeDefault
+          containers:
+            - name: grant
+              # The SAME image the instances above run, pinned to the same tag. psql
+              # is already in it and it is already present on every node, so this
+              # adds no new image and no new supply-chain surface. Keep the tag in
+              # step with spec.imageName above when Renovate bumps it.
+              image: ghcr.io/cloudnative-pg/postgresql:16.4
+              imagePullPolicy: IfNotPresent
+              command:
+                - psql
+                # -w never prompts; -X skips ~/.psqlrc; ON_ERROR_STOP makes the Job
+                # FAIL LOUDLY rather than reporting success on a refused grant.
+                - "-w"
+                - "-X"
+                - "-v"
+                - "ON_ERROR_STOP=1"
+                - "-f"
+                - /etc/grant/grant.sql
+              env:
+                # libpq connection, assembled from the owner Secret. Target the -rw
+                # Service: GRANT is a write and must reach the primary.
+                - name: PGHOST
+                  value: nextcloud-db-rw.nextcloud-db.svc.cluster.local
+                - name: PGPORT
+                  value: "5432"
+                # Connecting to the application database itself is deliberate: the
+                # owner has CONNECT on it, and a successful connection is the same
+                # check the exporter was failing.
+                - name: PGDATABASE
+                  value: nextcloud
+                - name: PGUSER
+                  valueFrom:
+                    secretKeyRef:
+                      name: nextcloud-db-app
+                      key: username
+                # Read by libpq from the environment and never logged; psql echoes
+                # no connection parameters.
+                - name: PGPASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: nextcloud-db-app
+                      key: password
+                - name: PGCONNECT_TIMEOUT
+                  value: "10"
+              volumeMounts:
+                - name: sql
+                  mountPath: /etc/grant
+                  readOnly: true
+              securityContext:
+                allowPrivilegeEscalation: false
+                readOnlyRootFilesystem: true
+                capabilities:
+                  drop:
+                    - ALL
+              resources:
+                requests:
+                  cpu: 10m
+                  memory: 32Mi
+                limits:
+                  memory: 128Mi
+          volumes:
+            - name: sql
+              configMap:
+                name: nextcloud-db-exporter-grant
+                defaultMode: 0444
+---
+# The desired state, as SQL. Idempotent: re-granting an existing privilege is a
+# no-op, so a converged run changes nothing. The DO block is the ASSERTION — it
+# turns "the grant did not take" into a failed Job instead of a silent success.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: nextcloud-db-exporter-grant
+  namespace: nextcloud-db
+  labels:
+    app.kubernetes.io/name: nextcloud-db-exporter-grant
+    app.kubernetes.io/part-of: nextcloud-db
+data:
+  grant.sql: |
+    -- The ONE privilege the CNPG metrics exporter is missing on this database.
+    -- CONNECT only: no CREATE, no TEMPORARY, no ownership, no table/schema grants.
+    -- Read access for the collector's pg_catalog queries already comes from the
+    -- built-in pg_monitor role, which CNPG grants this role when it creates it.
+    GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
+
+    -- Fail the Job if the privilege is somehow still absent, so a broken
+    -- reconciler surfaces as a failed Job rather than as a quiet blind spot.
+    DO $$
+    BEGIN
+      IF NOT has_database_privilege('cnpg_metrics_exporter', 'nextcloud', 'CONNECT') THEN
+        RAISE EXCEPTION 'cnpg_metrics_exporter still lacks CONNECT on database nextcloud';
+      END IF;
+      RAISE NOTICE 'converged: cnpg_metrics_exporter has CONNECT on database nextcloud';
+    END
+    $$;

platform/cnpg/alerts.yaml

@@ -48,6 +48,33 @@
 #   cnpg_collector_pg_wal{value="count"} vs {value="max"} prod: 17 vs 64  dev: 36 vs 64  (neither fires yet)
 #   cnpg_last_error                                       prod+dev: nextcloud-db = 1
 #
+# AGGREGATION SCOPE — WHICH RULES MAY KEY ON `pod`, AND WHY (incident, 2026-09-12). The first cut of
+# this file grouped EVERY rule `by (namespace, pod)`. That is correct only for metrics every instance
+# reports independently, and WRONG for the two backup-catalogue metrics, which ONLY THE PRIMARY
+# populates. Verified live on prod 2026-09-12 (`cnpg_collector_first_recoverability_point`, raw):
+#     keycloak-db-2  = 1788764922   nextcloud-db-2 = 1788764955   semaphore-db-2 = 1788764954  <- PRIMARIES
+#     keycloak-db-1/-3, nextcloud-db-1/-3, semaphore-db-1/-3 = 0                               <- REPLICAS
+# Since the Clusters went to `instances: 3`, `CNPGNoRecoverabilityPoint` was therefore PERMANENTLY
+# FIRING on all six replicas while all three Clusters reported firstRecoverabilityPoint=2026-09-07,
+# lastSuccessfulBackup within hours, ContinuousArchiving=True and LastBackupSucceeded=True. Backups
+# were fine; the alert was noise, and noise on a DR alarm is worse than no alarm at all — a REAL loss
+# of recoverability would have arrived as a seventh identical email nobody reads.
+#
+# THE RULE OF THUMB, applied as an audit to every rule below:
+#   * `== 0` / staleness shapes ("nothing reports a value") MUST aggregate `pod` AWAY, or every pod
+#     that structurally never reports the value fires forever.
+#   * `> 0` shapes ("some pod reports a bad value") are SAFE per-pod: a pod sitting at 0 simply does
+#     not fire, and keeping `pod` names the instance to go look at.
+#
+# THE GROUPING LABEL IS `job`, NOT `namespace`. The metric carries NO `cnpg.io/cluster` label — its
+# full label set on prod is {container, endpoint, instance, job, namespace, pod} — but the CNPG
+# operator creates one PodMonitor per Cluster, named after the Cluster, in the Cluster's namespace,
+# so prometheus-operator derives `job="<namespace>/<cluster>"`: `keycloak/keycloak-db`,
+# `nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`. That identifies the CNPG CLUSTER
+# uniquely. Grouping by `namespace` alone happens to work today (one Cluster per namespace, verified)
+# but would let two Clusters in one namespace MASK each other — the healthy one's non-zero value would
+# satisfy `max` for both. `job` cannot do that, so `job` is what the DR rules group by.
+#
 # SEVERITIES: every rule here is `warning` (-> the email-warning receiver). This is deliberate.
 # A database heading for disk-full deserves a warning with weeks of lead time, and this repo already
 # has alert fatigue from a permanently-firing `critical` (see the VPNTunnelDown post-mortem in
@@ -82,6 +109,12 @@ spec:
         # 24h; prod has never been below 3 (min_over_time 24h: keycloak 9, nextcloud 5, semaphore 3).
         # A count threshold like `> 5` would have delayed semaphore-db's alert indefinitely — its
         # backlog sits at 3 because it writes almost nothing.
+        #
+        # SCOPE AUDIT (2026-09-12): keeps `pod`, deliberately. Every instance has its own pg_wal on its
+        # own PVC and every instance reports this metric (verified: all 9 prod pods publish
+        # `{value="ready"}` = 0 today), and this is a `> 0` shape — a replica sitting at 0 never fires,
+        # so there is no permanent-firing hazard and no masking (`> 0` fires if ANY pod is bad). `pod`
+        # earns its place here: it names the instance whose archiver is stuck.
         - alert: CNPGWALArchiveBacklog
           expr: |
             max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0
@@ -102,15 +135,37 @@ spec:
         # `for: 6h` because a legitimately fresh Cluster reads 0 until its first base backup lands
         # (ScheduledBackup immediate:true, but MinIO/bucket setup can lag) — 6h keeps a normal cold
         # start quiet while still catching a bootstrap that never produced a recoverability point.
+        #
+        # SCOPE — `by (namespace, job)`, NOT `by (namespace, pod)`. THIS IS THE FIX FOR THE 2026-09-12
+        # permanently-firing incident (see the AGGREGATION SCOPE block in the header). The recoverability
+        # point is a property of the CLUSTER's backup catalogue, which only the PRIMARY reads; a replica
+        # reports a hard 0 forever. Grouped by `pod`, each replica was evaluated on its own and this
+        # `== 0` fired permanently on all six of them. Aggregating `pod` away makes `max` take the
+        # PRIMARY's non-zero value as the cluster's answer, so the alert says what its own summary always
+        # claimed to say — "this Postgres CLUSTER has no recoverability point" — and fires only when NO
+        # instance of that Cluster reports one.
+        #
+        # PROVEN LIVE, read-only, prod, 2026-09-12 (promtool query instant):
+        #   OLD  max by (namespace, pod) (...) == 0  -> 6 series: keycloak-db-1/-3, nextcloud-db-1/-3,
+        #                                              semaphore-db-1/-3   (and 6 ALERTS firing)
+        #   NEW  max by (namespace, job) (...) == 0  -> EMPTY
+        # and it is not merely vacuous — forcing the metric to 0 for one cluster's primary only
+        # (`* on(pod) group_left() ...{pod="keycloak-db-2"}`) yields exactly one series,
+        # {job="keycloak/keycloak-db"}, so a genuine single-cluster loss of DR still alerts, per cluster.
+        #
+        # A switchover cannot flap this: while the primary restarts, `max` briefly sees only replicas' 0,
+        # but `for: 6h` requires the condition at EVERY evaluation for six hours and a promotion takes
+        # seconds. Annotations key on `job` (= "<namespace>/<cluster>") because `pod` no longer exists on
+        # these series — a `{{ $labels.pod }}` left behind here would render as an empty string.
         - alert: CNPGNoRecoverabilityPoint
           expr: |
-            max by (namespace, pod) (cnpg_collector_first_recoverability_point) == 0
+            max by (namespace, job) (cnpg_collector_first_recoverability_point) == 0
           for: 6h
           labels:
             severity: warning
           annotations:
-            summary: "CNPG cluster has NO recoverability point: {{ $labels.namespace }}/{{ $labels.pod }}"
-            description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} reports firstRecoverabilityPoint = null for 6h — there is NO point-in-time this database can be restored to. Off-cluster DR for this database does not exist (R-IDP-05 / R-STATE-03). Usually the barmanObjectStore credentials/endpoint/path are wrong; check `kubectl get cluster -n {{ $labels.namespace }} -o yaml` for the ContinuousArchiving condition."
+            summary: "CNPG cluster has NO recoverability point: {{ $labels.job }}"
+            description: "PostgreSQL cluster {{ $labels.job }} (namespace {{ $labels.namespace }}) reports firstRecoverabilityPoint = null on EVERY instance for 6h — there is NO point-in-time this database can be restored to. Off-cluster DR for this database does not exist (R-IDP-05 / R-STATE-03). This is a per-CLUSTER statement: the primary owns the backup catalogue, so a healthy cluster clears it even though its replicas report 0. Usually the barmanObjectStore credentials/endpoint/path are wrong; check `kubectl get cluster -n {{ $labels.namespace }} -o yaml` for the ContinuousArchiving condition."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
         # ---------------------------------------------------------------------------------------
         # The daily ScheduledBackup silently stopped. Nothing else covers this: the Velero rules in
@@ -119,16 +174,31 @@ spec:
         # backup). Guarded with `> 0` so a cluster that has NEVER had a backup is reported by
         # CNPGNoRecoverabilityPoint above and does not double-notify here (time()-0 would otherwise be
         # ~57 years and always true). 36h = one missed daily run plus slack.
+        #
+        # SCOPE — `by (namespace, job)`, fixed alongside CNPGNoRecoverabilityPoint (2026-09-12).
+        # `cnpg_collector_last_available_backup_timestamp` has the SAME primary-only defect, measured on
+        # prod the same day: keycloak-db-2 = 3.1h old, nextcloud-db-2 = 2.1h, semaphore-db-2 = 0.1h, and
+        # ALL SIX REPLICAS = 0, i.e. `time() - 0` = 497020h ≈ 56.7 YEARS. Unlike the rule above this one
+        # was not firing, because the `> 0` guard was load-bearing — it was the single thing standing
+        # between this rule and six more permanent false positives, which is far too thin a margin for a
+        # DR alarm. Aggregating `pod` away removes the dependence: `max` picks the PRIMARY's timestamp as
+        # the cluster's answer (a demoted primary drops to 0 and can never win the `max`, and a fresher
+        # backup always beats a staler one), the `> 0` guard then means exactly "this cluster has had at
+        # least one backup", and the result is a real per-cluster staleness statement.
+        #
+        # PROVEN LIVE, prod, 2026-09-12: OLD -> empty, NEW -> empty (so the fix changes no live verdict),
+        # while the per-CLUSTER ages the new aggregation actually evaluates are keycloak 3.09h /
+        # nextcloud 2.09h / semaphore 0.09h — the primaries' real values, all far inside 36h.
         - alert: CNPGNoRecentBaseBackup
           expr: |
-            (time() - max by (namespace, pod) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
-            and max by (namespace, pod) (cnpg_collector_last_available_backup_timestamp) > 0
+            (time() - max by (namespace, job) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
+            and max by (namespace, job) (cnpg_collector_last_available_backup_timestamp) > 0
           for: 2h
           labels:
             severity: warning
           annotations:
-            summary: "No recent CNPG base backup: {{ $labels.namespace }}/{{ $labels.pod }}"
-            description: "The most recent completed base backup for {{ $labels.namespace }}/{{ $labels.pod }} is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Check `kubectl get backup -n {{ $labels.namespace }}`."
+            summary: "No recent CNPG base backup: {{ $labels.job }}"
+            description: "The most recent completed base backup for PostgreSQL cluster {{ $labels.job }} (namespace {{ $labels.namespace }}) is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Per-CLUSTER: the newest timestamp across the cluster's instances is used, which is the primary's. Check `kubectl get backups.postgresql.cnpg.io -n {{ $labels.namespace }}` — the FULLY-QUALIFIED name is required: longhorn.io, postgresql.cnpg.io and velero.io all register kind Backup, and bare `kubectl get backup` resolves to Longhorn's (empty) set and reports 'No resources found' while CNPG backups exist."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-backup-missed"
         # ---------------------------------------------------------------------------------------
         # WAL-VOLUME-SPECIFIC ESCALATION. pg_wal holds more segments than max_wal_size allows for,
@@ -146,6 +216,12 @@ spec:
         # any Cluster), which is why the PVC-fill rules cover it at all; the CNPG-native
         # `cnpg_collector_pg_wal{value="volume_size"}` is NaN on both clusters for that reason and is
         # therefore NOT usable as a signal.
+        #
+        # SCOPE AUDIT (2026-09-12): keeps `pod`, deliberately — and here it is REQUIRED. Every instance
+        # reports its own count/max (verified on all 9 prod pods: count 16-37 against max 64), pg_wal is
+        # per-instance on a per-instance PVC, and the comparison only means anything pod-against-itself:
+        # aggregating `pod` away would compare one instance's count with another's max. `>` is also a
+        # per-pod shape, so no instance can sit permanently true.
         - alert: CNPGWALVolumeNotRecycling
           expr: |
             max by (namespace, pod) (cnpg_collector_pg_wal{value="count"})
@@ -170,6 +246,15 @@ spec:
         # Remedy: grant the exporter role CONNECT on the application database. `for: 6h` keeps a pod
         # restart or a brief connection blip quiet; severity stays `warning` so it never becomes the
         # permanently-firing `critical` this repo has been burned by.
+        #
+        # SCOPE AUDIT (2026-09-12): keeps `pod`, UNCHANGED — this rule's meaning and firing set are
+        # deliberately left exactly as they were. `cnpg_last_error` is genuinely per-instance: each
+        # instance's own exporter reports its own SQL-collection health, and prod proves the metric
+        # discriminates rather than tracking role — all THREE nextcloud-db pods (primary -2 included)
+        # report 1 while all six keycloak/semaphore pods report 0. So there is no primary-only defect to
+        # fix, and per-pod is the right granularity: the remedy is a GRANT applied per database, and you
+        # want to see it clear on every instance. It is a TRUE POSITIVE right now (nextcloud-db, the
+        # 42501 CONNECT grant) and is owned by separate work; nothing here silences or rescopes it.
         - alert: CNPGMetricsCollectorErroring
           expr: |
             max by (namespace, pod) (cnpg_last_error) == 1