From 7880d84bce168134ec2af0809b9aafaff98adca4 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 23:22:27 -0500 Subject: [PATCH] fix(cnpg): grant cnpg_metrics_exporter CONNECT on the nextcloud database MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CNPG metrics exporter could not CONNECT to the `nextcloud` application database, so every user-defined collector query against that target failed on all three instances: FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) DETAIL: User does not have CONNECT privilege. The whole cnpg_pg_stat_* / pg_settings / pg_postmaster family was therefore MISSING for nextcloud-db while the scrape target still reported up=1 — the blind spot platform/cnpg/alerts.yaml's CNPGMetricsCollectorErroring was written to catch. It had been firing on all three pods; measured before this change, cnpg_pg_settings_setting had 765 series for keycloak, 765 for semaphore-db and ZERO for nextcloud-db. Nextcloud's database had never been monitored. Root cause is the live ACL, not the spec — all three Cluster specs bootstrap identically: nextcloud : datacl = {nextcloud=CTc/nextcloud} <- explicit, no PUBLIC entry keycloak : datacl = NULL semaphore : datacl = NULL A NULL datacl means PostgreSQL's defaults, which include CONNECT for PUBLIC, so keycloak and semaphore admit the exporter only by inheriting PUBLIC's implicit CONNECT. On `nextcloud` something ran the equivalent of REVOKE ALL ON DATABASE nextcloud FROM PUBLIC, which materialises datacl and drops that implicit entry. That REVOKE is in neither this repo nor DECISIONS-LOG.md — it was applied out of band, which is why the remedy has to be declarative rather than another hand-run psql. CNPG 1.30.0 cannot express a database-level GRANT (verified against the live CRDs): `Database` has no ACL field (`allowConnections` is the datallowconn flag, not a per-role grant), and managed.roles/DatabaseRole cover role attributes plus `inRoles` membership only — and membership would hand the exporter the owner's CREATE/TEMP and all table access to buy one CONNECT. postInitApplicationSQL runs once at bootstrap, so it cannot repair the already-bootstrapped instances that are actually broken. So this follows the pattern the repo already uses for state Flux cannot reach: the level-triggered CronJob reconciler alongside platform/keycloak/client-secret-reconciler.yaml. A one-shot Job is deliberately avoided — a completed Job never re-runs, so a rebuild or PITR restore would silently reopen the blind spot. Exactly one privilege, and nothing else: GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter; giving `cnpg_metrics_exporter=c/nextcloud` — CONNECT only, no CREATE, no TEMPORARY, no ownership, no superuser, no table or schema grants. Nothing more is needed because CNPG already creates the role as a member of the built-in pg_monitor role (verified on all three clusters), which carries the read access the collector's pg_catalog queries want. It runs as the database OWNER, not a superuser: enableSuperuserAccess stays off, and a database owner holds the grant option on its own database. Verified live on prod: * the grant issued under SET ROLE nextcloud, proving no superuser is needed; * permission-denied and "Error collecting user query" both went to 0 on nextcloud-db-1/-2/-3, i.e. every collector query now succeeds; * cnpg_pg_settings_setting now 765 series for nextcloud-db, matching the other two, and cnpg_pg_postmaster_start_time present on all three instances; * cnpg_last_error = 0 on all NINE instances, so keycloak-db and semaphore-db are unaffected and their datacl is still NULL; * convergence proven by revoking the grant again and letting the committed CronJob restore it, then re-running it against a converged database to show idempotence; * CNPGMetricsCollectorErroring, firing on all three pods beforehand, has cleared (`for: 6h` delays firing, not resolution). Co-Authored-By: Claude Opus 5 --- apps/wave-0/nextcloud-db.yaml | 231 ++++++++++++++++++++++++++++++++++ 1 file changed, 231 insertions(+) diff --git a/apps/wave-0/nextcloud-db.yaml b/apps/wave-0/nextcloud-db.yaml index 41d3b6a..8bfe33a 100644 --- a/apps/wave-0/nextcloud-db.yaml +++ b/apps/wave-0/nextcloud-db.yaml @@ -158,3 +158,234 @@ spec: cluster: name: nextcloud-db method: barmanObjectStore +--- +# ============================================================================= +# CNPG METRICS-EXPORTER CONNECT RECONCILER (R-OBS-01) +# +# THE BUG THIS CLOSES (true positive, verified live on prod 2026-09-12, and the +# condition platform/cnpg/alerts.yaml already recorded on BOTH clusters 2026-09-06): +# the `nextcloud` database's ACL had been narrowed out-of-band so that PUBLIC held +# nothing, and CNPG's metrics exporter role therefore could not even CONNECT: +# +# nextcloud-db-1/-2/-3, every scrape: +# FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) +# DETAIL: User does not have CONNECT privilege. +# msg="Error collecting user query" targetDatabase=nextcloud +# +# CONSEQUENCE: `cnpg_last_error` = 1 on all three instances and the ENTIRE +# cnpg_pg_stat_* / pg_settings / pg_postmaster family MISSING for this database, +# while the scrape target still reported up=1 — a blind spot that looks like health. +# Measured before the fix: cnpg_pg_settings_setting had 765 series for keycloak and +# 765 for semaphore-db and ZERO for nextcloud-db. Nextcloud's database had never +# been monitored. +# +# ROOT CAUSE, AND WHY ONLY THIS DATABASE. All three Cluster specs are identical in +# their bootstrap; the difference is entirely in the live ACL: +# +# nextcloud : datacl = {nextcloud=CTc/nextcloud} <-- explicit, no PUBLIC entry +# keycloak : datacl = NULL +# semaphore : datacl = NULL +# +# A NULL datacl means "PostgreSQL defaults", which include CONNECT and TEMP for +# PUBLIC — so keycloak and semaphore let the exporter in only by inheriting PUBLIC's +# implicit CONNECT. On `nextcloud` something ran the equivalent of +# `REVOKE ALL ON DATABASE nextcloud FROM PUBLIC`, which MATERIALISES datacl and +# drops the implicit PUBLIC entry with it. That REVOKE is not in this repo and not +# in DECISIONS-LOG.md — it was applied out of band, which is exactly why a +# git-declared fix is needed rather than another hand-run psql. +# +# WHY THE GRANT IS EXPLICIT RATHER THAN "RE-GRANT CONNECT TO PUBLIC": restoring +# PUBLIC's CONNECT would undo whatever hardening that REVOKE intended and would +# leave the exporter depending on an implicit default again — the precise fragility +# that broke here and that keycloak/semaphore are still relying on. Naming the +# exporter role makes the dependency explicit and survives a future REVOKE FROM +# PUBLIC on any of the three databases. +# +# WHY A CronJob AND NOT A CNPG FIELD — CNPG 1.30.0's API GENUINELY CANNOT EXPRESS +# THIS. Checked against the live CRDs on prod: +# * `Database` (databases.postgresql.cnpg.io) has no database-ACL field at all. +# `allowConnections` is the pg_database.datallowconn FLAG (open/closed to +# everyone), not a per-role GRANT; its only grant/revoke verbs are the `usage` +# lists on `fdws` and `servers`. +# * `spec.managed.roles` and `DatabaseRole` manage ROLE ATTRIBUTES (login, +# superuser, connectionLimit, ...) plus `inRoles` MEMBERSHIP. Membership is the +# wrong tool: the only local role holding CONNECT here is the OWNER, so +# `inRoles: [nextcloud]` would hand the exporter the owner's CREATE and TEMP on +# the database and read/write on every application table — the opposite of +# least privilege, to buy one CONNECT. +# * `bootstrap.initdb.postInitApplicationSQL` runs ONCE, at bootstrap. It cannot +# repair these three already-bootstrapped instances (the actual bug), and on a +# FRESH cluster the grant is unnecessary anyway (datacl starts NULL). Adding it +# would be a second mechanism that fixes nothing this one does not. +# So the SQL has to be applied by something, and this repo already has the pattern +# for state Flux cannot reach: the level-triggered CronJob reconciler in +# platform/keycloak/client-secret-reconciler.yaml, whose reasoning transfers +# verbatim — the drifted state lives inside PostgreSQL, not in a Kubernetes object, +# so no admission or mutation policy can see it. A one-shot Job is deliberately NOT +# used: a completed Job is never re-run, so a DB rebuild or a PITR restore would +# silently reopen the blind spot. A loop converges instead. +# +# PRIVILEGE GRANTED — EXACTLY ONE, AND NOTHING ELSE: +# GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter; +# Resulting ACL entry is `cnpg_metrics_exporter=c/nextcloud` — the `c` is CONNECT; +# there is no `C` (CREATE) and no `T` (TEMPORARY). No ownership, no superuser, no +# write, no table or schema grants. Nothing further is needed because CNPG already +# creates this role as a member of the built-in `pg_monitor` role (verified on all +# three prod clusters), and pg_monitor already carries the read access every +# collector query wants — they read pg_catalog views (pg_stat_database, +# pg_stat_archiver, pg_settings, pg_postmaster) and no application tables. Proof +# that CONNECT alone is sufficient: after this grant, `Error collecting user query` +# went to ZERO on all three pods, i.e. every collector query now succeeds. +# +# IT RUNS AS THE DATABASE OWNER, NOT AS A SUPERUSER. `enableSuperuserAccess` is off +# on this Cluster (there is no nextcloud-db-superuser Secret), and it stays off: the +# owner of a database holds the grant option on that database's privileges, so +# `nextcloud` can issue this GRANT itself. Verified live before committing, by +# running the statement under `SET ROLE nextcloud`. Credentials come from the +# in-namespace SOPS Secret nextcloud-db-app that already exists for the app. +# +# NETWORKING: nextcloud-db is intentionally NOT an app-namespace (see this file's +# header), so Kyverno generates no default-deny baseline here and this same-namespace +# pod -> nextcloud-db-rw:5432 connection needs no rendered allow. +# +# THE POD LABEL IS DELIBERATELY *NOT* app.kubernetes.io/name=nextcloud-db. That +# label is what `inheritedMetadata` stamps on the CNPG instance pods and what the +# rendered nextcloud egress CiliumNetworkPolicy selects; reusing it here would quietly +# make this reconciler a permitted destination for the Nextcloud app. It gets its own +# name so that selector keeps meaning "the database". +apiVersion: batch/v1 +kind: CronJob +metadata: + name: nextcloud-db-exporter-grant + namespace: nextcloud-db + labels: + app.kubernetes.io/name: nextcloud-db-exporter-grant + app.kubernetes.io/part-of: nextcloud-db +spec: + # Every 15 minutes, matching the keycloak reconciler's cadence. A converged + # database makes this one idempotent GRANT and exits; the cost is negligible, and + # after a rebuild or PITR restore monitoring is blind for at most one interval + # (well inside the alert's `for: 6h`). + schedule: "*/15 * * * *" + # A run still waiting on the primary must not be lapped by the next tick. + concurrencyPolicy: Forbid + startingDeadlineSeconds: 300 + successfulJobsHistoryLimit: 1 + failedJobsHistoryLimit: 3 + jobTemplate: + spec: + backoffLimit: 2 + activeDeadlineSeconds: 300 + ttlSecondsAfterFinished: 3600 + template: + metadata: + labels: + app.kubernetes.io/name: nextcloud-db-exporter-grant + app.kubernetes.io/part-of: nextcloud-db + spec: + restartPolicy: Never + # Talks only to PostgreSQL; its one input is a mounted Secret, so it needs + # no Kubernetes API access at all (R-SEC-02, least privilege). + automountServiceAccountToken: false + securityContext: + runAsNonRoot: true + # uid/gid 26 is `postgres` in the CNPG PostgreSQL image. + runAsUser: 26 + runAsGroup: 26 + seccompProfile: + type: RuntimeDefault + containers: + - name: grant + # The SAME image the instances above run, pinned to the same tag. psql + # is already in it and it is already present on every node, so this + # adds no new image and no new supply-chain surface. Keep the tag in + # step with spec.imageName above when Renovate bumps it. + image: ghcr.io/cloudnative-pg/postgresql:16.4 + imagePullPolicy: IfNotPresent + command: + - psql + # -w never prompts; -X skips ~/.psqlrc; ON_ERROR_STOP makes the Job + # FAIL LOUDLY rather than reporting success on a refused grant. + - "-w" + - "-X" + - "-v" + - "ON_ERROR_STOP=1" + - "-f" + - /etc/grant/grant.sql + env: + # libpq connection, assembled from the owner Secret. Target the -rw + # Service: GRANT is a write and must reach the primary. + - name: PGHOST + value: nextcloud-db-rw.nextcloud-db.svc.cluster.local + - name: PGPORT + value: "5432" + # Connecting to the application database itself is deliberate: the + # owner has CONNECT on it, and a successful connection is the same + # check the exporter was failing. + - name: PGDATABASE + value: nextcloud + - name: PGUSER + valueFrom: + secretKeyRef: + name: nextcloud-db-app + key: username + # Read by libpq from the environment and never logged; psql echoes + # no connection parameters. + - name: PGPASSWORD + valueFrom: + secretKeyRef: + name: nextcloud-db-app + key: password + - name: PGCONNECT_TIMEOUT + value: "10" + volumeMounts: + - name: sql + mountPath: /etc/grant + readOnly: true + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + memory: 128Mi + volumes: + - name: sql + configMap: + name: nextcloud-db-exporter-grant + defaultMode: 0444 +--- +# The desired state, as SQL. Idempotent: re-granting an existing privilege is a +# no-op, so a converged run changes nothing. The DO block is the ASSERTION — it +# turns "the grant did not take" into a failed Job instead of a silent success. +apiVersion: v1 +kind: ConfigMap +metadata: + name: nextcloud-db-exporter-grant + namespace: nextcloud-db + labels: + app.kubernetes.io/name: nextcloud-db-exporter-grant + app.kubernetes.io/part-of: nextcloud-db +data: + grant.sql: | + -- The ONE privilege the CNPG metrics exporter is missing on this database. + -- CONNECT only: no CREATE, no TEMPORARY, no ownership, no table/schema grants. + -- Read access for the collector's pg_catalog queries already comes from the + -- built-in pg_monitor role, which CNPG grants this role when it creates it. + GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter; + + -- Fail the Job if the privilege is somehow still absent, so a broken + -- reconciler surfaces as a failed Job rather than as a quiet blind spot. + DO $$ + BEGIN + IF NOT has_database_privilege('cnpg_metrics_exporter', 'nextcloud', 'CONNECT') THEN + RAISE EXCEPTION 'cnpg_metrics_exporter still lacks CONNECT on database nextcloud'; + END IF; + RAISE NOTICE 'converged: cnpg_metrics_exporter has CONNECT on database nextcloud'; + END + $$;