home-network-k3s-impl/runbooks/cnpg-wal-growth.md

cnpg-wal-growth — pg_wal is not being recycled

Alert: CNPGWALVolumeNotRecycling (warning, 2h) File: platform/cnpg/alerts.yaml, group platform-health.cnpg Expression:

max by (namespace, pod) (cnpg_collector_pg_wal{value="count"})
  > max by (namespace, pod) (cnpg_collector_pg_wal{value="max"})

What this means here

pg_wal is holding more WAL segments than max_wal_size allows for, which means checkpoints are no longer reclaiming space. On CNPG the units are segments of 16 MiB: max is max_wal_size (64 = 1 GiB on this platform) and count is what is currently sitting in pg_wal.

This is the storage escalation of /runbooks/cnpg-archiving, not an independent fault. CNPGWALArchiveBacklog is the day-one signal — it fires within an hour of the archiver getting stuck. This one fires when the resulting backlog has grown large enough to be a disk problem. With archive_mode=on a failing archive_command is exactly what this looks like after a while, and because pg_wal shares the data PVC on every Cluster here (no walStorage: is configured on any of them) it ends in the volume filling and the postmaster PANICking.

It does not fire today and is not meant to: measured, prod keycloak-db holds 17 segments against max 64 (dev held 36 against 64), and across all nine prod instances counts ran 16-37 against 64. At prod's observed ~6.8 MiB/h, keycloak-db would reach 64 segments roughly 40 days before its 5 GiB PVC actually ran out — so this alert leads the PVC-fill alerts by weeks.

Why pod is kept here, and is required. Every instance reports its own count and max, pg_wal is per-instance on a per-instance PVC, and the comparison only means anything pod-against-itself — aggregating pod away would compare one instance's count with another's max. A > shape is also safe per-pod: no instance can sit permanently true. (Contrast the == 0 and staleness rules in the same file, which must aggregate pod away — that is the 2026-09-12 fix.)

One metric that looks right and is not: the CNPG-native cnpg_collector_pg_wal{value="volume_size"} is NaN on both clusters, precisely because pg_wal shares the data PVC rather than having its own. Do not build anything on it.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

Which instance, and how far past max_wal_size:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s -G --data-urlencode "query=max by (namespace,pod) (cnpg_collector_pg_wal{value=\"count\"})" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "pod|\"1\""'

The same thing from inside the instance, which also shows you the archive backlog and the free space in one view:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c \
     "du -sh /var/lib/postgresql/data/pgdata/pg_wal; \
      ls /var/lib/postgresql/data/pgdata/pg_wal | grep -c \"^0\"; \
      ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | grep -c ready; \
      df -h /var/lib/postgresql/data; \
      psql -tAc \"show max_wal_size; show archive_mode; select pg_walfile_name(pg_current_wal_lsn());\""'

Is archiving the cause? (It almost always is.)

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint; echo; \
   kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -i archive | tail -20'

Is a replication slot or a stale standby holding WAL instead?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- \
     psql -x -c "select slot_name, active, wal_status, safe_wal_size, restart_lsn from pg_replication_slots;" \
     -c "select application_name, state, sync_state, replay_lag from pg_stat_replication;"'

Compute the runway — free bytes on the PVC divided by the observed growth rate:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s -G --data-urlencode "query=predict_linear(kubelet_volume_stats_available_bytes{job=\"kubelet\",namespace=\"keycloak\"}[24h], 7*24*3600)" localhost:19090/api/v1/query'

Likely causes, most likely first

1. WAL archiving is failing (dominant cause)

Go to /runbooks/cnpg-archiving — that is the actual fix. Fix the archive_command and Postgres recycles the backlog on its own, without any intervention in pg_wal. Historical precedent: prod ran 38+ hours from bootstrap with endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT, all three Clusters at ContinuousArchiving=False with barman-cloud-wal-archive: exit status 4.

2. An inactive replication slot is pinning WAL

A standby that has been gone long enough leaves a slot holding restart_lsn, and Postgres keeps every segment after it. pg_replication_slots.wal_status will read extended or lost. The healthy fix is to bring the standby back (CNPG manages the slots), not to drop the slot by hand. Note podAntiAffinityType: required means a lost instance stays Pending until a node returns — expected, and accepted.

3. Genuinely heavy write load

Then count exceeding max_wal_size is checkpointing lagging, not a leak, and it self-corrects. Confirm that archive_status/*.ready is near zero: a large pg_wal with an empty ready queue is a busy database, not a stuck archiver.

4. max_wal_size was lowered

64 segments = 1 GiB is the configured value on both clusters. A reduction makes this rule fire without anything actually being wrong. Check the Cluster's postgresql parameters — and note that under primaryUpdateMethod: switchover the operator rejects an update that changes imageName and a postgresql parameter in the same reconcile.

5. The instance is a replica that fell behind

Replay lag holds WAL on the primary. pg_stat_replication.replay_lag shows it.

When to escalate or stop

• NEVER delete WAL segments or archive_status marker files by hand. It destroys the database. This is the single irreversible mistake available here, and the alert's own description says so: "Fix archiving — do NOT delete WAL by hand."
• Do not drop a replication slot to free space without the operator: you may make a lagging standby unrecoverable and force a full re-seed.
• Do not expand the PVC and stop there. It buys days and hides the fault.
• Do not scale the Cluster out while archiving is failing — that multiplies the disk-full clock.
• Escalate when the computed runway is under a week, or when PersistentVolumeAlmostFull (/runbooks/pvc-capacity) is also firing for the same PVC — at that point an outage is imminent and a Postgres data volume under 5% free will PANIC the postmaster.

---

Source of truth: runbooks/cnpg-wal-growth.md. Index: /runbooks/ Records: platform/cnpg/alerts.yaml (the measured segment counts, the NaN volume_size finding, and the 2026-09-12 scope audit that deliberately KEPT pod here).