jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
164 lines
9.4 KiB
Markdown
164 lines
9.4 KiB
Markdown
# cnpg-wal-growth — pg_wal is not being recycled
|
|
|
|
**Alert:** `CNPGWALVolumeNotRecycling` (warning, 2h)
|
|
**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
|
|
**Expression:**
|
|
```
|
|
max by (namespace, pod) (cnpg_collector_pg_wal{value="count"})
|
|
> max by (namespace, pod) (cnpg_collector_pg_wal{value="max"})
|
|
```
|
|
|
|
## What this means here
|
|
|
|
`pg_wal` is holding more WAL segments than `max_wal_size` allows for, which means **checkpoints are no
|
|
longer reclaiming space**. On CNPG the units are **segments of 16 MiB**: `max` is `max_wal_size`
|
|
(**64 = 1 GiB** on this platform) and `count` is what is currently sitting in `pg_wal`.
|
|
|
|
This is the **storage escalation** of `/runbooks/cnpg-archiving`, not an independent fault.
|
|
`CNPGWALArchiveBacklog` is the day-one signal — it fires within an hour of the archiver getting stuck.
|
|
This one fires when the resulting backlog has grown large enough to be a *disk* problem. With
|
|
`archive_mode=on` a failing `archive_command` is exactly what this looks like after a while, and because
|
|
**`pg_wal` shares the data PVC on every Cluster here** (no `walStorage:` is configured on any of them)
|
|
it ends in the volume filling and the postmaster **PANICking**.
|
|
|
|
It does **not** fire today and is not meant to: measured, prod `keycloak-db` holds 17 segments against
|
|
max 64 (dev held 36 against 64), and across all nine prod instances counts ran 16-37 against 64. At
|
|
prod's observed ~6.8 MiB/h, `keycloak-db` would reach 64 segments roughly **40 days before** its 5 GiB
|
|
PVC actually ran out — so this alert leads the PVC-fill alerts by weeks.
|
|
|
|
**Why `pod` is kept here, and is required.** Every instance reports its own `count` and `max`, `pg_wal`
|
|
is per-instance on a per-instance PVC, and the comparison only means anything pod-against-itself —
|
|
aggregating `pod` away would compare one instance's `count` with another's `max`. A `>` shape is also
|
|
safe per-pod: no instance can sit permanently true. (Contrast the `== 0` and staleness rules in the same
|
|
file, which **must** aggregate `pod` away — that is the 2026-09-12 fix.)
|
|
|
|
**One metric that looks right and is not:** the CNPG-native
|
|
`cnpg_collector_pg_wal{value="volume_size"}` is **NaN on both clusters**, precisely because `pg_wal`
|
|
shares the data PVC rather than having its own. Do not build anything on it.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Which instance, and how far past `max_wal_size`:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
|
|
curl -s -G --data-urlencode "query=max by (namespace,pod) (cnpg_collector_pg_wal{value=\"count\"})" localhost:19090/api/v1/query | python3 -m json.tool | grep -E "pod|\"1\""'
|
|
```
|
|
|
|
The same thing from inside the instance, which also shows you the archive backlog and the free space in
|
|
one view:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c \
|
|
"du -sh /var/lib/postgresql/data/pgdata/pg_wal; \
|
|
ls /var/lib/postgresql/data/pgdata/pg_wal | grep -c \"^0\"; \
|
|
ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | grep -c ready; \
|
|
df -h /var/lib/postgresql/data; \
|
|
psql -tAc \"show max_wal_size; show archive_mode; select pg_walfile_name(pg_current_wal_lsn());\""'
|
|
```
|
|
|
|
Is archiving the cause? (It almost always is.)
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
|
|
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint; echo; \
|
|
kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -i archive | tail -20'
|
|
```
|
|
|
|
Is a replication slot or a stale standby holding WAL instead?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- \
|
|
psql -x -c "select slot_name, active, wal_status, safe_wal_size, restart_lsn from pg_replication_slots;" \
|
|
-c "select application_name, state, sync_state, replay_lag from pg_stat_replication;"'
|
|
```
|
|
|
|
Compute the runway — free bytes on the PVC divided by the observed growth rate:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n monitoring port-forward svc/kube-prometheus-stack-prometheus 19090:9090 & sleep 5; \
|
|
curl -s -G --data-urlencode "query=predict_linear(kubelet_volume_stats_available_bytes{job=\"kubelet\",namespace=\"keycloak\"}[24h], 7*24*3600)" localhost:19090/api/v1/query'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. WAL archiving is failing (dominant cause)
|
|
Go to `/runbooks/cnpg-archiving` — that is the actual fix. Fix the `archive_command` and Postgres recycles
|
|
the backlog on its own, without any intervention in `pg_wal`. Historical precedent: prod ran 38+ hours
|
|
from bootstrap with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, all three Clusters at
|
|
`ContinuousArchiving=False` with `barman-cloud-wal-archive: exit status 4`.
|
|
|
|
### 2. An inactive replication slot is pinning WAL
|
|
A standby that has been gone long enough leaves a slot holding `restart_lsn`, and Postgres keeps every
|
|
segment after it. `pg_replication_slots.wal_status` will read `extended` or `lost`. The healthy fix is to
|
|
bring the standby back (CNPG manages the slots), not to drop the slot by hand. Note
|
|
`podAntiAffinityType: required` means a lost instance stays **Pending** until a node returns — expected,
|
|
and accepted.
|
|
|
|
### 3. Genuinely heavy write load
|
|
Then `count` exceeding `max_wal_size` is checkpointing lagging, not a leak, and it self-corrects. Confirm
|
|
that `archive_status/*.ready` is near zero: a large `pg_wal` with an empty ready queue is a busy
|
|
database, not a stuck archiver.
|
|
|
|
### 4. `max_wal_size` was lowered
|
|
64 segments = 1 GiB is the configured value on both clusters. A reduction makes this rule fire without
|
|
anything actually being wrong. Check the Cluster's postgresql parameters — and note that under
|
|
`primaryUpdateMethod: switchover` the operator **rejects an update that changes `imageName` and a
|
|
postgresql parameter in the same reconcile**.
|
|
|
|
### 5. The instance is a replica that fell behind
|
|
Replay lag holds WAL on the primary. `pg_stat_replication.replay_lag` shows it.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **NEVER delete WAL segments or `archive_status` marker files by hand.** It destroys the database. This
|
|
is the single irreversible mistake available here, and the alert's own description says so: *"Fix
|
|
archiving — do NOT delete WAL by hand."*
|
|
- **Do not drop a replication slot** to free space without the operator: you may make a lagging standby
|
|
unrecoverable and force a full re-seed.
|
|
- **Do not expand the PVC and stop there.** It buys days and hides the fault.
|
|
- **Do not scale the Cluster out** while archiving is failing — that **multiplies** the disk-full clock.
|
|
- Escalate when the computed runway is under a week, or when `PersistentVolumeAlmostFull`
|
|
(`/runbooks/pvc-capacity`) is also firing for the same PVC — at that point an outage is imminent and a
|
|
Postgres data volume under 5% free will PANIC the postmaster.
|
|
|
|
---
|
|
Source of truth: `runbooks/cnpg-wal-growth.md`. Index: `/runbooks/`
|
|
Records: `platform/cnpg/alerts.yaml` (the measured segment counts, the NaN `volume_size` finding, and the
|
|
2026-09-12 scope audit that deliberately KEPT `pod` here).
|