# cnpg-wal-growth — pg_wal is not being recycled **Alert:** `CNPGWALVolumeNotRecycling` (warning, 2h) **File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg` **Expression:** ``` max by (namespace, pod) (cnpg_collector_pg_wal{value="count"}) > max by (namespace, pod) (cnpg_collector_pg_wal{value="max"}) ``` ## What this means here `pg_wal` is holding more WAL segments than `max_wal_size` allows for, which means **checkpoints are no longer reclaiming space**. On CNPG the units are **segments of 16 MiB**: `max` is `max_wal_size` (**64 = 1 GiB** on this platform) and `count` is what is currently sitting in `pg_wal`. This is the **storage escalation** of `/runbooks/cnpg-archiving`, not an independent fault. `CNPGWALArchiveBacklog` is the day-one signal — it fires within an hour of the archiver getting stuck. This one fires when the resulting backlog has grown large enough to be a *disk* problem. With `archive_mode=on` a failing `archive_command` is exactly what this looks like after a while, and because **`pg_wal` shares the data PVC on every Cluster here** (no `walStorage:` is configured on any of them) it ends in the volume filling and the postmaster **PANICking**. It does **not** fire today and is not meant to: measured, prod `keycloak-db` holds 17 segments against max 64 (dev held 36 against 64), and across all nine prod instances counts ran 16-37 against 64. At prod's observed ~6.8 MiB/h, `keycloak-db` would reach 64 segments roughly **40 days before** its 5 GiB PVC actually ran out — so this alert leads the PVC-fill alerts by weeks. **Why `pod` is kept here, and is required.** Every instance reports its own `count` and `max`, `pg_wal` is per-instance on a per-instance PVC, and the comparison only means anything pod-against-itself — aggregating `pod` away would compare one instance's `count` with another's `max`. A `>` shape is also safe per-pod: no instance can sit permanently true. (Contrast the `== 0` and staleness rules in the same file, which **must** aggregate `pod` away — that is the 2026-09-12 fix.) **One metric that looks right and is not:** the CNPG-native `cnpg_collector_pg_wal{value="volume_size"}` is **NaN on both clusters**, precisely because `pg_wal` shares the data PVC rather than having its own. Do not build anything on it. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git (`git push origin main` **is** the deploy). Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands Which instance, and how far past `max_wal_size`: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ curl -s -G --data-urlencode "query=max by (namespace,pod) (cnpg_collector_pg_wal{value=\"count\"})" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "pod|\"1\""' ``` The same thing from inside the instance, which also shows you the archive backlog and the free space in one view: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c \ "du -sh /var/lib/postgresql/data/pgdata/pg_wal; \ ls /var/lib/postgresql/data/pgdata/pg_wal | grep -c \"^0\"; \ ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | grep -c ready; \ df -h /var/lib/postgresql/data; \ psql -tAc \"show max_wal_size; show archive_mode; select pg_walfile_name(pg_current_wal_lsn());\""' ``` Is archiving the cause? (It almost always is.) ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \ -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint; echo; \ kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -i archive | tail -20' ``` Is a replication slot or a stale standby holding WAL instead? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- \ psql -x -c "select slot_name, active, wal_status, safe_wal_size, restart_lsn from pg_replication_slots;" \ -c "select application_name, state, sync_state, replay_lag from pg_stat_replication;"' ``` Compute the runway — free bytes on the PVC divided by the observed growth rate: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \ curl -s -G --data-urlencode "query=predict_linear(kubelet_volume_stats_available_bytes{job=\"kubelet\",namespace=\"keycloak\"}[24h], 7*24*3600)" localhost:19090/api/v1/query' ``` ## Likely causes, most likely first ### 1. WAL archiving is failing (dominant cause) Go to `/runbooks/cnpg-archiving` — that is the actual fix. Fix the `archive_command` and Postgres recycles the backlog on its own, without any intervention in `pg_wal`. Historical precedent: prod ran 38+ hours from bootstrap with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, all three Clusters at `ContinuousArchiving=False` with `barman-cloud-wal-archive: exit status 4`. ### 2. An inactive replication slot is pinning WAL A standby that has been gone long enough leaves a slot holding `restart_lsn`, and Postgres keeps every segment after it. `pg_replication_slots.wal_status` will read `extended` or `lost`. The healthy fix is to bring the standby back (CNPG manages the slots), not to drop the slot by hand. Note `podAntiAffinityType: required` means a lost instance stays **Pending** until a node returns — expected, and accepted. ### 3. Genuinely heavy write load Then `count` exceeding `max_wal_size` is checkpointing lagging, not a leak, and it self-corrects. Confirm that `archive_status/*.ready` is near zero: a large `pg_wal` with an empty ready queue is a busy database, not a stuck archiver. ### 4. `max_wal_size` was lowered 64 segments = 1 GiB is the configured value on both clusters. A reduction makes this rule fire without anything actually being wrong. Check the Cluster's postgresql parameters — and note that under `primaryUpdateMethod: switchover` the operator **rejects an update that changes `imageName` and a postgresql parameter in the same reconcile**. ### 5. The instance is a replica that fell behind Replay lag holds WAL on the primary. `pg_stat_replication.replay_lag` shows it. ## When to escalate or stop - **NEVER delete WAL segments or `archive_status` marker files by hand.** It destroys the database. This is the single irreversible mistake available here, and the alert's own description says so: *"Fix archiving — do NOT delete WAL by hand."* - **Do not drop a replication slot** to free space without the operator: you may make a lagging standby unrecoverable and force a full re-seed. - **Do not expand the PVC and stop there.** It buys days and hides the fault. - **Do not scale the Cluster out** while archiving is failing — that **multiplies** the disk-full clock. - Escalate when the computed runway is under a week, or when `PersistentVolumeAlmostFull` (`/runbooks/pvc-capacity`) is also firing for the same PVC — at that point an outage is imminent and a Postgres data volume under 5% free will PANIC the postmaster. --- Source of truth: `runbooks/cnpg-wal-growth.md`. Index: `/runbooks/` Records: `platform/cnpg/alerts.yaml` (the measured segment counts, the NaN `volume_size` finding, and the 2026-09-12 scope audit that deliberately KEPT `pod` here).