jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
176 lines
9.8 KiB
Markdown
176 lines
9.8 KiB
Markdown
# backup-failure — A Velero backup failed
|
|
|
|
**Alert:** `VeleroBackupFailure` (critical, 5m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups`
|
|
**Expression:** `increase(velero_backup_failure_total[1h]) > 0`
|
|
|
|
## What this means here
|
|
|
|
A Velero backup in the last hour ended in `phase=Failed`. Off-cluster DR (R-STATE-03) is at risk. Velero
|
|
backs up the **cluster's Kubernetes objects and its Longhorn PVC data**; it does **not** provide the DR
|
|
path for the three CloudNativePG databases — CNPG owns those via its own Barman archive, because a
|
|
Velero snapshot of a live Postgres is not a backup. So a Velero failure and a CNPG failure are separate
|
|
incidents with separate runbooks (`/runbooks/cnpg-archiving`).
|
|
|
|
## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE
|
|
|
|
`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**.
|
|
Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and
|
|
`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified
|
|
on prod 2026-09-13, where the fully-qualified queries returned **6 Velero backups** and **27 CNPG
|
|
backups** while the short name returned nothing.
|
|
|
|
This has already caused two false "the backups are missing" conclusions in one night. **A destructive
|
|
recovery action taken on that false negative is how a monitoring quirk becomes real data loss.** Use:
|
|
|
|
```
|
|
backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io
|
|
schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io
|
|
```
|
|
|
|
## Where the backups go
|
|
|
|
Cloudflare **R2**, since the operator's 2026-09-05 decision (commit `32d6766`). Prod cannot reach the
|
|
on-Proxmox MinIO at all — measured from a prod pod, *connection timed out after 15s* — and that MinIO
|
|
ran on the same physical host as dev's VMs, so dev's "off-cluster" backup shared a failure domain.
|
|
Live config: endpoint `https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com`, region
|
|
**`auto`**, bucket `k3s-prod-backup`, prefix `velero`, schedule `0 3 * * *`, retention `720h`.
|
|
`backupstoragelocation.velero.io/default` should read **Available**.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Which backup failed, and why — fully-qualified, always:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get backups.velero.io -A \
|
|
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,START:.status.startTimestamp,ERRORS:.status.errors,WARN:.status.warnings'
|
|
```
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n velero get backups.velero.io <name> -o yaml | sed -n "/status:/,\$p"'
|
|
```
|
|
|
|
The storage location and the controller's log (the R2 error text is in the log, not the CR):
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n velero get backupstoragelocations.velero.io,schedules.velero.io; echo; \
|
|
kubectl --context k3s-prod -n velero logs deploy/velero --tail=150 2>&1 | grep -iE "error|NotImplemented|InvalidArgument|tagging|checksum" | tail -25'
|
|
```
|
|
|
|
Did the **volume data** upload even though the backup failed? (This distinction is the whole point —
|
|
see cause 1.)
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n velero get podvolumebackups.velero.io \
|
|
-o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,PHASE:.status.phase,BYTES:.status.progress.bytesDone'
|
|
```
|
|
|
|
The plugin pin (cause 1 depends entirely on this):
|
|
|
|
```
|
|
grep -n "velero-plugin-for-aws\|checksumAlgorithm\|region" platform/velero/helmrelease.yaml
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. The R2 object-tagging incompatibility (the known, specific, repo-documented one)
|
|
`velero-plugin-for-aws` **v1.14.2** sets PutObject's Tagging unconditionally —
|
|
`Tagging: aws.String(o.tagging)` with `o.tagging == ""` — so the SDK emits `x-amz-tagging: ` (empty) on
|
|
**every** PutObject. **R2 does not implement object tagging at all** and answers:
|
|
|
|
```
|
|
NotImplemented: Header 'x-amz-tagging' with value '' not implemented
|
|
```
|
|
|
|
Observed live on prod **twice**, on the write of `velero/backups/<name>/velero-backup.json`. The shape
|
|
of the failure is the dangerous part: **kopia uploads the volume DATA successfully (its own S3 client),
|
|
then the Backup goes `phase=Failed` with NO metadata object — an unrestorable backup.**
|
|
**No configuration avoids it** (`config.tagging` with a real value sends a tag set R2 also rejects). The
|
|
fix is upstream PR #309, which landed on release-1.14 *after* v1.14.2 was cut, so the image is pinned to
|
|
**`velero/velero-plugin-for-aws:v1.14.3-rc.2`**.
|
|
**EXIT CONDITION: move to v1.14.3 the moment it ships GA. Do NOT "clean this up" back to v1.14.2 — that
|
|
silently breaks every prod backup's metadata.** If this alert fires and the pin has drifted, that is
|
|
your answer.
|
|
|
|
Two more mandatory R2 settings in the same file: `config.checksumAlgorithm: ""` (R2 rejects the AWS
|
|
SDK's `x-amz-checksum-*` headers with **HTTP 400 InvalidArgument** — harmless against MinIO, fatal
|
|
against R2) and `region` exactly **`auto`** (it was once hardcoded `us-east-1`, correct-by-accident for
|
|
MinIO).
|
|
|
|
### 2. A placeholder or wrong endpoint
|
|
`clusters/prod` deliberately carried the **loud** placeholder
|
|
`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` rather than inheriting the base (dev) value,
|
|
because *"an omitted op means prod silently inherits a dev address — far worse than a failed apply"*.
|
|
Check `S3_BACKUP_ENDPOINT`, `S3_BACKUP_REGION`, `VELERO_BACKUP_BUCKET`, `VELERO_BACKUP_PREFIX` and
|
|
`BACKUP_CREDENTIALS_SUFFIX` in `clusters/base/cluster-vars.yaml` and the prod overlay. A
|
|
`backupstoragelocation` that is not `Available` points straight here.
|
|
|
|
### 3. Credentials
|
|
The R2 token lives in a SOPS-encrypted Secret (`velero-cloud-prod.sops.yaml`), and the `velero` tier
|
|
carries its own `decryption` stanza. An expired or mis-scoped token gives 403s in the log.
|
|
|
|
### 4. A volume could not be read
|
|
Check `podvolumebackups.velero.io` for a specific volume. Note that NFS-sourced volumes are
|
|
**deliberately skipped** by the `velero-volume-policies` ConfigMap — see `/runbooks/backup-partial`,
|
|
which is where that belongs.
|
|
|
|
### 5. The node or storage was unhealthy during the run
|
|
Velero's daily runs at 03:00 and reads Longhorn PVCs. If a node was under I/O pressure or a volume was
|
|
degraded, the read can fail. Cross-read `/runbooks/node-io` and `/runbooks/longhorn-degraded`.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Escalate promptly.** This is a `critical` because it is a DR alarm: the cluster may currently have no
|
|
recent restorable backup.
|
|
- **Do not delete a Failed backup to "clean up".** A Failed Velero backup can leave **orphaned kopia
|
|
blobs** in the bucket — prod has two such backups from 2026-09-12 (`prod-proof-20260912`,
|
|
`prod-vp-probe-20260912`) whose orphaned blobs are **pending owner consent to delete**. Deleting
|
|
objects in R2 is an operator decision, not a tidy-up.
|
|
- **Do not assume a Completed backup is restorable** if the plugin pin has drifted — the failure mode is
|
|
precisely "data uploaded, metadata missing".
|
|
- **Be honest about restore**: `dr-restore` is **authored and has never been executed**
|
|
(`conformance.home.k3s/never-executed`), so the R2 restore path is unrehearsed. R2 egress is free,
|
|
which is the whole reason restore rehearsals are affordable — but nobody has done one yet.
|
|
- The three CNPG databases are **not** covered by Velero. If the concern is a database, go to
|
|
`/runbooks/cnpg-archiving`.
|
|
|
|
---
|
|
Source of truth: `runbooks/backup-failure.md`. Index: `/runbooks/`
|
|
Records: commit `2e259a8` and `platform/velero/helmrelease.yaml` lines ~68-95 (the R2 tagging
|
|
incompatibility, `checksumAlgorithm`, `region: auto`); commit `32d6766` (the move to R2); commit
|
|
`2d5171e` (Velero shipped with no ServiceMonitor, so these alerts could never fire).
|