jakman85andClaude Opus 5
21491f901b
CNPGNoRecoverabilityPoint has been firing continuously on prod against six
pods — keycloak-db-1/-3, nextcloud-db-1/-3, semaphore-db-1/-3 — i.e. exactly
the REPLICAS of all three Clusters, while the databases themselves are fine:
every Cluster reports firstRecoverabilityPoint=2026-09-07T07:0x, a
lastSuccessfulBackup within the last few hours, ContinuousArchiving=True and
LastBackupSucceeded=True.
Root cause is aggregation scope, not archiving. `cnpg_collector_first_recoverability_point`
reads the backup CATALOGUE, which only the primary owns; a replica reports a
hard 0 for as long as it is a replica. The metric carries no cnpg.io/cluster
label (its label set on prod is container/endpoint/instance/job/namespace/pod),
so `max by (namespace, pod)` evaluated each replica as its own subject and an
`== 0` comparison then held forever. The rule went from correct to permanently
wrong the moment the Clusters went to `instances: 3`; nothing about the
expression changed.
WHY THIS IS WORSE THAN HAVING NO ALERT. An alert that is always firing has
zero information content: its value is identical in the healthy state and in
the failure state, so it cannot distinguish them. It is worse than silence in
two specific ways. First, it trains the operator to delete the notification on
sight, so the day a base backup genuinely stops the seventh identical email is
discarded with the other six — the alert actively MASKS the failure it was
written to catch, which for a DR alarm means discovering the loss at restore
time. Second, it is indistinguishable from a broken alert, so nobody can tell
whether the absence of new alerts means health or means the rule rotted. This
file exists because a real 38-hour archiving failure told nobody; a rule that
cries wolf about six healthy replicas rebuilds exactly the blindness it was
supposed to remove. This repo has been here before — see the permanently-firing
critical VPNTunnelDown post-mortem in platform/monitoring/alerts.yaml.
Fix: group by (namespace, job) and aggregate `pod` away. The CNPG operator
creates one PodMonitor per Cluster, named after the Cluster, in the Cluster's
namespace, so prometheus-operator derives job="<namespace>/<cluster>" —
keycloak/keycloak-db, nextcloud-db/nextcloud-db, semaphore-db/semaphore-db —
which identifies the CNPG Cluster uniquely. `job` rather than bare `namespace`
deliberately: namespace alone works today (one Cluster per namespace, verified
live) but would let two Clusters in one namespace mask each other, since the
healthy one's non-zero value would satisfy `max` for both. The alert now
states what its summary always claimed — this CLUSTER has no recoverability
point — and fires only when no instance of that Cluster reports one.
Proven live against prod Prometheus, read-only (promtool query instant):
OLD max by (namespace, pod) (...) == 0 -> 6 series, and 6 ALERTS firing
NEW max by (namespace, job) (...) == 0 -> empty
The new expression is not vacuous: forcing the metric to 0 for one cluster's
primary only yields exactly one series, {job="keycloak/keycloak-db"}, so a
genuine single-cluster loss of DR still alerts, per cluster. `promtool check
rules` passes on all 5 rules.
CNPGNoRecentBaseBackup carried the same defect and is fixed the same way.
`cnpg_collector_last_available_backup_timestamp` is also primary-only: on prod
the primaries read 3.1h/2.1h/0.1h old while all six replicas read 0, i.e.
time()-0 = ~56.7 YEARS. It was not firing only because the `> 0` guard was
load-bearing — a single clause was all that stood between this rule and six
more permanent false positives, too thin a margin for a DR alarm. Aggregating
`pod` away removes the dependence: `max` takes the primary's timestamp as the
cluster's answer, and the guard now means "this cluster has had at least one
backup". Verdict unchanged live (old empty, new empty).
The other three rules were audited and deliberately left on `by (namespace, pod)`:
* CNPGWALArchiveBacklog — every instance reports archive_status and the shape
is `> 0`, so a replica at 0 never fires and no pod can mask another; `pod`
names the instance whose archiver is stuck.
* CNPGWALVolumeNotRecycling — per-pod is REQUIRED, not merely safe:
aggregating it away would compare one instance's WAL count against
another's max_wal_size.
* CNPGMetricsCollectorErroring — unchanged in meaning AND grouping. Its
grouping is already correct: cnpg_last_error is genuinely per-instance, and
prod proves it discriminates rather than tracking role — all three
nextcloud-db pods report 1 while all six keycloak/semaphore pods report 0.
It is a true positive (the nextcloud 42501 CONNECT grant) owned by separate
work; nothing here silences or rescopes it.
The general rule, recorded in the file: `== 0`/staleness shapes must aggregate
`pod` away or every structurally-silent pod fires forever; `> 0` shapes are
safe per-pod and keep the instance name.
Co-Authored-By: Claude Opus 5 <[email protected]>
platform/cnpg
CloudNativePG — the PostgreSQL operator that runs Keycloak's DR-critical database.
- Governing ADR: ADR-0014 — IdP: Keycloak
- Serves: R-IDP-05 (DR-critical datastore), supports R-IDP-01
- Owner: WS-SEC
Why CNPG (not the Bitnami-bundled Postgres)
The original Keycloak stub used the Bitnami Keycloak chart with a bundled Bitnami PostgreSQL. Bitnami's free Docker Hub images were purged/relocated (DECISIONS-LOG F-KY1) — a deploy on stock Bitnami tags ImagePullBackOffs. CloudNativePG:
- ships from ghcr.io/cloudnative-pg (unaffected by the Bitnami purge),
- runs Postgres as a replicated
Clusteron Longhorn (3 instances → survives node loss), - gives declarative backup/restore (
ScheduledBackup+ Barman object store) — the clean DR hook for R-IDP-05 (off-cluster backup) once the backup target lands (OQ-BKP, Phase 4).
Controllers/configs split (F-MON1)
This directory installs only the operator + CRDs (controllers). The keycloak database Cluster
CR lives in ../../platform-config
(dependsOn: platform, wait) so the postgresql.cnpg.io CRDs exist before the CR applies.