home-network-k3s-impl/runbooks/cnpg-wal-growth.md

# cnpg-wal-growth — pg_wal is not being recycled

**Alert:** `CNPGWALVolumeNotRecycling` (warning, 2h)
**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
**Expression:**
```
max by (namespace, pod) (cnpg_collector_pg_wal{value="count"})
  > max by (namespace, pod) (cnpg_collector_pg_wal{value="max"})
```

## What this means here

`pg_wal` is holding more WAL segments than `max_wal_size` allows for, which means **checkpoints are no
longer reclaiming space**. On CNPG the units are **segments of 16 MiB**: `max` is `max_wal_size`
(**64 = 1 GiB** on this platform) and `count` is what is currently sitting in `pg_wal`.

This is the **storage escalation** of `/runbooks/cnpg-archiving`, not an independent fault.
`CNPGWALArchiveBacklog` is the day-one signal — it fires within an hour of the archiver getting stuck.
This one fires when the resulting backlog has grown large enough to be a *disk* problem. With
`archive_mode=on` a failing `archive_command` is exactly what this looks like after a while, and because
**`pg_wal` shares the data PVC on every Cluster here** (no `walStorage:` is configured on any of them)
it ends in the volume filling and the postmaster **PANICking**.

It does **not** fire today and is not meant to: measured, prod `keycloak-db` holds 17 segments against
max 64 (dev held 36 against 64), and across all nine prod instances counts ran 16-37 against 64. At
prod's observed ~6.8 MiB/h, `keycloak-db` would reach 64 segments roughly **40 days before** its 5 GiB
PVC actually ran out — so this alert leads the PVC-fill alerts by weeks.

**Why `pod` is kept here, and is required.** Every instance reports its own `count` and `max`, `pg_wal`
is per-instance on a per-instance PVC, and the comparison only means anything pod-against-itself —
aggregating `pod` away would compare one instance's `count` with another's `max`. A `>` shape is also
safe per-pod: no instance can sit permanently true. (Contrast the `== 0` and staleness rules in the same
file, which **must** aggregate `pod` away — that is the 2026-09-12 fix.)

**One metric that looks right and is not:** the CNPG-native
`cnpg_collector_pg_wal{value="volume_size"}` is **NaN on both clusters**, precisely because `pg_wal`
shares the data PVC rather than having its own. Do not build anything on it.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

Which instance, and how far past `max_wal_size`:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s -G --data-urlencode "query=max by (namespace,pod) (cnpg_collector_pg_wal{value=\"count\"})" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "pod|\"1\""'
```

The same thing from inside the instance, which also shows you the archive backlog and the free space in
one view:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c \
     "du -sh /var/lib/postgresql/data/pgdata/pg_wal; \
      ls /var/lib/postgresql/data/pgdata/pg_wal | grep -c \"^0\"; \
      ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | grep -c ready; \
      df -h /var/lib/postgresql/data; \
      psql -tAc \"show max_wal_size; show archive_mode; select pg_walfile_name(pg_current_wal_lsn());\""'
```

Is archiving the cause? (It almost always is.)

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint; echo; \
   kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -i archive | tail -20'
```

Is a replication slot or a stale standby holding WAL instead?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- \
     psql -x -c "select slot_name, active, wal_status, safe_wal_size, restart_lsn from pg_replication_slots;" \
     -c "select application_name, state, sync_state, replay_lag from pg_stat_replication;"'
```

Compute the runway — free bytes on the PVC divided by the observed growth rate:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
   curl -s -G --data-urlencode "query=predict_linear(kubelet_volume_stats_available_bytes{job=\"kubelet\",namespace=\"keycloak\"}[24h], 7*24*3600)" localhost:19090/api/v1/query'
```

## Likely causes, most likely first

### 1. WAL archiving is failing (dominant cause)
Go to `/runbooks/cnpg-archiving` — that is the actual fix. Fix the `archive_command` and Postgres recycles
the backlog on its own, without any intervention in `pg_wal`. Historical precedent: prod ran 38+ hours
from bootstrap with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, all three Clusters at
`ContinuousArchiving=False` with `barman-cloud-wal-archive: exit status 4`.

### 2. An inactive replication slot is pinning WAL
A standby that has been gone long enough leaves a slot holding `restart_lsn`, and Postgres keeps every
segment after it. `pg_replication_slots.wal_status` will read `extended` or `lost`. The healthy fix is to
bring the standby back (CNPG manages the slots), not to drop the slot by hand. Note
`podAntiAffinityType: required` means a lost instance stays **Pending** until a node returns — expected,
and accepted.

### 3. Genuinely heavy write load
Then `count` exceeding `max_wal_size` is checkpointing lagging, not a leak, and it self-corrects. Confirm
that `archive_status/*.ready` is near zero: a large `pg_wal` with an empty ready queue is a busy
database, not a stuck archiver.

### 4. `max_wal_size` was lowered
64 segments = 1 GiB is the configured value on both clusters. A reduction makes this rule fire without
anything actually being wrong. Check the Cluster's postgresql parameters — and note that under
`primaryUpdateMethod: switchover` the operator **rejects an update that changes `imageName` and a
postgresql parameter in the same reconcile**.

### 5. The instance is a replica that fell behind
Replay lag holds WAL on the primary. `pg_stat_replication.replay_lag` shows it.

## When to escalate or stop

- **NEVER delete WAL segments or `archive_status` marker files by hand.** It destroys the database. This
  is the single irreversible mistake available here, and the alert's own description says so: *"Fix
  archiving — do NOT delete WAL by hand."*
- **Do not drop a replication slot** to free space without the operator: you may make a lagging standby
  unrecoverable and force a full re-seed.
- **Do not expand the PVC and stop there.** It buys days and hides the fault.
- **Do not scale the Cluster out** while archiving is failing — that **multiplies** the disk-full clock.
- Escalate when the computed runway is under a week, or when `PersistentVolumeAlmostFull`
  (`/runbooks/pvc-capacity`) is also firing for the same PVC — at that point an outage is imminent and a
  Postgres data volume under 5% free will PANIC the postmaster.

---
Source of truth: `runbooks/cnpg-wal-growth.md`. Index: `/runbooks/`
Records: `platform/cnpg/alerts.yaml` (the measured segment counts, the NaN `volume_size` finding, and the
2026-09-12 scope audit that deliberately KEPT `pod` here).