home-network-k3s-impl/runbooks/cnpg-archiving.md

# cnpg-archiving — WAL archiving is stuck, or a Postgres cluster has no recoverability point

**Alerts:** `CNPGWALArchiveBacklog` (warning, 1h) and `CNPGNoRecoverabilityPoint` (warning, 6h)
**File:** `platform/cnpg/alerts.yaml`, group `platform-health.cnpg`
**Expressions:**
```
max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0
max by (namespace, job)  (cnpg_collector_first_recoverability_point) == 0
```

## What this means here — and why it is an AVAILABILITY incident, not just a backup one

`pg_wal/archive_status` holds one marker per WAL segment: `.ready` = written but **not yet archived**,
`.done` = archived and safe to recycle. A healthy archiver drains `.ready` within seconds, so a
persistently non-zero ready count means `barman-cloud-wal-archive` is failing.

**Two consequences, and the second one is the emergency:**

1. There is no continuous archive, so **PITR is broken** (R-IDP-05 / R-STATE-03).
2. **`archive_mode=on` means Postgres will NOT recycle an unarchived WAL segment.** `pg_wal` grows
   without bound until the data PVC fills and the postmaster **PANICs**. `pg_wal` shares the **data**
   PVC on every Cluster here — no `walStorage:` is configured on any of them — so there is no separate
   volume to absorb it.

This is measured, not theoretical. Prod ran **from bootstrap to 2026-09-07** with
`endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`: all three Clusters at `ContinuousArchiving=False` with
`barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null`, **for 38+ hours, with no
alert of any kind** — found by hand. On `keycloak-db` that was 17 unarchived segments / ~268 MiB on a
5 GiB PVC, growing **~6.8 MiB/h ≈ 29 days to disk-full** (about 579 MiB already burned by 2026-09-07).
The rule recorded verbatim: *"Never leave a placeholder here, and treat `ContinuousArchiving=False` on
this Cluster as an availability incident, not a backup one."*

`CNPGWALArchiveBacklog` is *"the one that would have caught the 2026-09-06 incident within the hour
instead of 38 hours later."* `> 0` with `for: 1h` — not a count threshold — because `for` requires the
condition at **every** evaluation for the full hour, so a write burst resets the timer while a genuinely
stuck archiver never returns to zero. A `> 5` threshold would have delayed `semaphore-db` indefinitely:
its backlog sits at 3 because it writes almost nothing.

## The two alerts are NOT the same statement

`CNPGWALArchiveBacklog` is about the **mechanism** and is per-**pod** (each instance has its own `pg_wal`
on its own PVC; a `> 0` shape is safe per-pod, and `pod` names the instance to go look at).

`CNPGNoRecoverabilityPoint` is about **DR coverage** and is per-**CLUSTER**, grouped
`by (namespace, job)`. That grouping is a fix, dated 2026-09-12: only the **primary** reads the backup
catalogue, so a replica reports a hard `0` forever. Grouped by `pod`, this `== 0` fired **permanently on
all six replicas** while all three Clusters were healthy. Verified live:
old `max by (namespace, pod) (...) == 0` → 6 series (`keycloak-db-1/-3`, `nextcloud-db-1/-3`,
`semaphore-db-1/-3`); new `max by (namespace, job) (...) == 0` → **empty** — and not vacuously so:
forcing the metric to 0 for one primary yields exactly one series. `job` is the grouping label because
the metric carries **no `cnpg.io/cluster` label**; the operator creates one PodMonitor per Cluster, so
prometheus-operator derives `job="<namespace>/<cluster>"`: `keycloak/keycloak-db`,
`nextcloud-db/nextcloud-db`, `semaphore-db/semaphore-db`. `namespace` alone happens to work today but
would let two Clusters in one namespace mask each other.

**Practical note:** if you see this alert firing on *pods* rather than per cluster, the per-cluster
aggregation is not deployed on this cluster yet — check the deployed PrometheusRule before treating it
as a real loss of DR.

**Severity is `warning` on purpose.** A database heading for disk-full deserves weeks of lead time, and
this repo has been burned by a permanently-firing `critical` (the `VPNTunnelDown` post-mortem). The
`critical` tier for this failure mode is `PersistentVolumeAlmostFull`, which fires only when the outage
is genuinely imminent.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
and it says so.

## First commands

The three Clusters' archiving and recoverability status — the single most informative command:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -A \
     -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,READY:.status.readyInstances,PRIMARY:.status.currentPrimary,ARCHIVING:.status.conditions[?\(@.type==\"ContinuousArchiving\"\)].status,LASTBACKUP:.status.conditions[?\(@.type==\"LastBackupSucceeded\"\)].status,FIRSTPITR:.status.firstRecoverabilityPoint'
```

The archiver's actual error text — `exit status 4` and its cause live here:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak logs keycloak-db-2 -c postgres --tail=200 2>&1 | grep -i archive | tail -30'
```

The backlog on disk, per instance:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n keycloak exec keycloak-db-2 -c postgres -- bash -c \
     "ls /var/lib/postgresql/data/pgdata/pg_wal/archive_status | grep -c ready; du -sh /var/lib/postgresql/data/pgdata/pg_wal; df -h /var/lib/postgresql/data"'
```

The backups that exist — **fully-qualified, or you will be told there are none**:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get backups.postgresql.cnpg.io -A; echo; \
   kubectl --context k3s-prod get scheduledbackups.postgresql.cnpg.io -A'
```

`kubectl get backup -A` prints "No resources found" on this cluster even with 27 CNPG backups present:
three CRDs register the kind `Backup` (`longhorn.io`, `postgresql.cnpg.io`, `velero.io`) and the short
name resolves to Longhorn's, which holds zero. Verified on prod 2026-09-13. **Never decide a recovery
action from the short name.**

The object-store configuration, which is where the fault usually is:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusters.postgresql.cnpg.io -n keycloak keycloak-db -o yaml | sed -n "/backup:/,/^  [a-z]/p"'
```

```
grep -nE "CNPG_BACKUP_PATH|S3_BACKUP_ENDPOINT|S3_BACKUP_REGION|BACKUP_CREDENTIALS_SUFFIX" clusters/base/cluster-vars.yaml clusters/prod/kustomization.yaml
```

## Likely causes, most likely first

### 1. A wrong or placeholder object-store endpoint / path / credential
The recorded cause of the 38-hour incident: prod ran with `endpointURL:
TBD-PROD-S3-BACKUP-ENDPOINT`. The placeholder was **deliberately loud** because *"base defaults are the
DEV values, so an OMITTED op means prod silently inherits a dev address — far worse than a failed
apply"*. Live prod values to expect: `CNPG_BACKUP_PATH = s3://k3s-prod-backup/barman`,
`S3_BACKUP_ENDPOINT = https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`,
`S3_BACKUP_REGION = auto`, `BACKUP_CREDENTIALS_SUFFIX = -prod`, retention `30d`, WAL compression gzip,
`maxParallel: 2`. Remediation: fix it in git and push; Flux applies and the archiver drains on its own.

### 2. R2-specific S3 incompatibilities
Cloudflare R2 rejects things MinIO tolerates. The two that are mandatory elsewhere in this platform and
worth checking here: `region` must be exactly `auto`, and AWS-SDK checksum headers are rejected with
**HTTP 400 InvalidArgument**. See `/runbooks/backup-failure` for the full R2 story, including the Velero
plugin's `x-amz-tagging` `NotImplemented` failure.

### 3. Credentials expired or mis-scoped
A SOPS-encrypted Secret selected by `BACKUP_CREDENTIALS_SUFFIX`. Standing rule from the OIDC incident:
**non-secret, environment-specific values must NOT live inside an encrypted Secret** — `${BASE_DOMAIN}`
substitution can never reach inside one. Keep endpoints and paths in cluster-vars, tokens in the Secret.

### 4. Egress is blocked
Default-deny is the baseline. The database namespaces need outbound HTTPS to R2. Cross-read
`/runbooks/denied-flows`.

### 5. A fresh Cluster that has not made its first base backup
`firstRecoverabilityPoint` reads null until the first base backup lands; the `for: 6h` exists exactly to
keep a legitimate cold start quiet. Base backups are staggered: keycloak 01:00, nextcloud 02:00,
semaphore 04:00 (Velero at 03:00). If a Cluster is hours old, wait.

### 6. Scaling out made it worse
Noted honestly: scaling a Cluster while `archive_mode=on` with a failing `archive_command`
**multiplies the disk-full clock**. Do not add instances as a remedy here.

## When to escalate or stop

- **Treat `ContinuousArchiving=False` as an availability incident with a countdown.** Compute the runway:
  free bytes on the data PVC ÷ WAL growth rate. At ~6.8 MiB/h on a 5 GiB PVC that is weeks; on a busier
  database it is days.
- **NEVER delete WAL segments by hand.** It destroys the database. This is the one irreversible mistake
  available in this runbook.
- **Do not delete the `.ready` marker files.** Same outcome.
- **Do not expand the PVC and stop there.** It buys time and hides the fault.
- **Do not scale the Cluster** while archiving is failing.
- **Verify before any restore** that `firstRecoverabilityPoint` is non-null and
  `backups.postgresql.cnpg.io` shows completed backups. Prod's healthy state on 2026-09-13 was:
  all three Clusters `ContinuousArchiving=True`, `LastBackupSucceeded=True`, `firstRecoverabilityPoint`
  2026-09-07, 27 backups (9 per cluster) completing daily, with one historical failure
  (`semaphore-db-daily-20260905130918`). That is what "good" looks like.
- Escalate to the operator for credential or endpoint changes, and for any restore.

---
Source of truth: `runbooks/cnpg-archiving.md`. Index: `/runbooks/`
Records: `platform/cnpg/alerts.yaml` header (the 2026-09-06 incident, the measured fill rate, and the
2026-09-12 aggregation-scope fix with its live promtool proof);
`platform/keycloak/postgres-cluster.yaml`; commit `32d6766` (the R2 decision).