# cnpg-archiving — WAL archiving is stuck, or a Postgres cluster has no recoverability point **Alerts:** `CNPGWALArchiveBacklog` (warning, 1h) and `CNPGNoRecoverabilityPoint` (warning, 6h) **File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg` **Expressions:** ``` max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0 max by (namespace, job) (cnpg_collector_first_recoverability_point) == 0 ``` ## What this means here — and why it is an AVAILABILITY incident, not just a backup one `pg_wal/archive_status` holds one marker per WAL segment: `.ready` = written but **not yet archived**, `.done` = archived and safe to recycle. A healthy archiver drains `.ready` within seconds, so a persistently non-zero ready count means `barman-cloud-wal-archive` is failing. **Two consequences, and the second one is the emergency:** 1. There is no continuous archive, so **PITR is broken** (R-IDP-05 / R-STATE-03). 2. **`archive_mode=on` means Postgres will NOT recycle an unarchived WAL segment.** `pg_wal` grows without bound until the data PVC fills and the postmaster **PANICs**. `pg_wal` shares the **data** PVC on every Cluster here — no `walStorage:` is configured on any of them — so there is no separate volume to absorb it. This is measured, not theoretical. Prod ran **from bootstrap to 2026-09-07** with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`: all three Clusters at `ContinuousArchiving=False` with `barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null`, **for 38+ hours, with no alert of any kind** — found by hand. On `keycloak-db` that was 17 unarchived segments / ~268 MiB on a 5 GiB PVC, growing **~6.8 MiB/h ≈ 29 days to disk-full** (about 579 MiB already burned by 2026-09-07). The rule recorded verbatim: *"Never leave a placeholder here, and treat `ContinuousArchiving=False` on this Cluster as an availability incident, not a backup one."* `CNPGWALArchiveBacklog` is *"the one that would have caught the 2026-09-06 incident within the hour instead of 38 hours later."* `> 0` with `for: 1h` — not a count threshold — because `for` requires the condition at **every** evaluation for the full hour, so a write burst resets the timer while a genuinely stuck archiver never returns to zero. A `> 5` threshold would have delayed `semaphore-db` indefinitely: its backlog sits at 3 because it writes almost nothing. ## The two alerts are NOT the same statement `CNPGWALArchiveBacklog` is about the **mechanism** and is per-**pod** (each instance has its own `pg_wal` on its own PVC; a `> 0` shape is safe per-pod, and `pod` names the instance to go look at). `CNPGNoRecoverabilityPoint` is about **DR coverage** and is per-**CLUSTER**, grouped `by (namespace, job)`. That grouping is a fix, dated 2026-09-12: only the **primary** reads the backup catalogue, so a replica reports a hard `0` forever. Grouped by `pod`, this `== 0` fired **permanently on all six replicas** while all three Clusters were healthy. Verified live: old `max by (namespace, pod) (...) == 0` → 6 series (`keycloak-db-1/-3`, `nextcloud-db-1/-3`, `semaphore-db-1/-3`); new `max by (namespace, job) (...) == 0` → **empty** — and not vacuously so: forcing the metric to 0 for one primary yields exactly one series. `job` is the grouping label because the metric carries **no `cnpg.io/cluster` label**; the operator creates one PodMonitor per Cluster, so prometheus-operator derives `job="/"`: `keycloak/keycloak-db`, `nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`. `namespace` alone happens to work today but would let two Clusters in one namespace mask each other. **Practical note:** if you see this alert firing on *pods* rather than per cluster, the per-cluster aggregation is not deployed on this cluster yet — check the deployed PrometheusRule before treating it as a real loss of DR. **Severity is `warning` on purpose.** A database heading for disk-full deserves weeks of lead time, and this repo has been burned by a permanently-firing `critical` (the `VPNTunnelDown` post-mortem). The `critical` tier for this failure mode is `PersistentVolumeAlmostFull`, which fires only when the outage is genuinely imminent. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands The three Clusters' archiving and recoverability status — the single most informative command: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint' ``` The archiver's actual error text — `exit status 4` and its cause live here: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -i archive | tail -30' ``` The backlog on disk, per instance: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c \ "ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | grep -c ready; du -sh /var/lib/postgresql/data/pgdata/pg_wal; df -h /var/lib/postgresql/data"' ``` The backups that exist — **fully-qualified, or you will be told there are none**: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \ kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A' ``` `kubectl get backup -A` prints "No resources found" on this cluster even with 27 CNPG backups present: three CRDs register the kind `Backup` (`longhorn.io`, `postgresql.cnpg.io`, `velero.io`) and the short name resolves to Longhorn's, which holds zero. Verified on prod 2026-09-13. **Never decide a recovery action from the short name.** The object-store configuration, which is where the fault usually is: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak keycloak-db -o yaml | sed -n "/backup:/,/^ [a-z]/p"' ``` ``` grep -nE "CNPG_BACKUP_PATH|S3_BACKUP_ENDPOINT|S3_BACKUP_REGION|BACKUP_CREDENTIALS_SUFFIX" clusters/base/cluster-vars.yaml clusters/prod/kustomization.yaml ``` ## Likely causes, most likely first ### 1. A wrong or placeholder object-store endpoint / path / credential The recorded cause of the 38-hour incident: prod ran with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`. The placeholder was **deliberately loud** because *"base defaults are the DEV values, so an OMITTED op means prod silently inherits a dev address — far worse than a failed apply"*. Live prod values to expect: `CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`, `S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`, `S3_BACKUP_REGION = auto`, `BACKUP_CREDENTIALS_SUFFIX = -prod`, retention `30d`, WAL compression gzip, `maxParallel: 2`. Remediation: fix it in git and push; Flux applies and the archiver drains on its own. ### 2. R2-specific S3 incompatibilities Cloudflare R2 rejects things MinIO tolerates. The two that are mandatory elsewhere in this platform and worth checking here: `region` must be exactly `auto`, and AWS-SDK checksum headers are rejected with **HTTP 400 InvalidArgument**. See `/runbooks/backup-failure` for the full R2 story, including the Velero plugin's `x-amz-tagging` `NotImplemented` failure. ### 3. Credentials expired or mis-scoped A SOPS-encrypted Secret selected by `BACKUP_CREDENTIALS_SUFFIX`. Standing rule from the OIDC incident: **non-secret, environment-specific values must NOT live inside an encrypted Secret** — `${BASE_DOMAIN}` substitution can never reach inside one. Keep endpoints and paths in cluster-vars, tokens in the Secret. ### 4. Egress is blocked Default-deny is the baseline. The database namespaces need outbound HTTPS to R2. Cross-read `/runbooks/denied-flows`. ### 5. A fresh Cluster that has not made its first base backup `firstRecoverabilityPoint` reads null until the first base backup lands; the `for: 6h` exists exactly to keep a legitimate cold start quiet. Base backups are staggered: keycloak 01:00, nextcloud 02:00, semaphore 04:00 (Velero at 03:00). If a Cluster is hours old, wait. ### 6. Scaling out made it worse Noted honestly: scaling a Cluster while `archive_mode=on` with a failing `archive_command` **multiplies the disk-full clock**. Do not add instances as a remedy here. ## When to escalate or stop - **Treat `ContinuousArchiving=False` as an availability incident with a countdown.** Compute the runway: free bytes on the data PVC ÷ WAL growth rate. At ~6.8 MiB/h on a 5 GiB PVC that is weeks; on a busier database it is days. - **NEVER delete WAL segments by hand.** It destroys the database. This is the one irreversible mistake available in this runbook. - **Do not delete the `.ready` marker files.** Same outcome. - **Do not expand the PVC and stop there.** It buys time and hides the fault. - **Do not scale the Cluster** while archiving is failing. - **Verify before any restore** that `firstRecoverabilityPoint` is non-null and `backups.postgresql.cnpg.io` shows completed backups. Prod's healthy state on 2026-09-13 was: all three Clusters `ContinuousArchiving=True`, `LastBackupSucceeded=True`, `firstRecoverabilityPoint` 2026-09-07, 27 backups (9 per cluster) completing daily, with one historical failure (`semaphore-db-daily-20260905130918`). That is what "good" looks like. - Escalate to the operator for credential or endpoint changes, and for any restore. --- Source of truth: `runbooks/cnpg-archiving.md`. Index: `/runbooks/` Records: `platform/cnpg/alerts.yaml` header (the 2026-09-06 incident, the measured fill rate, and the 2026-09-12 aggregation-scope fix with its live promtool proof); `platform/keycloak/postgres-cluster.yaml`; commit `32d6766` (the R2 decision).