jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
167 lines
9.1 KiB
Markdown
167 lines
9.1 KiB
Markdown
# backup-partial — A Velero backup completed with partial failures
|
|
|
|
**Alert:** `VeleroBackupPartialFailure` (warning, 5m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups`
|
|
**Expression:** `increase(velero_backup_partial_failure_total[1h]) > 0`
|
|
|
|
## What this means here
|
|
|
|
The backup finished, but some resources or volumes were skipped or errored. It is a warning rather than a
|
|
critical because *most of it worked* — and that is exactly what makes it dangerous: a
|
|
`PartiallyFailed` backup looks like a backup in every listing, and the thing it silently omitted may be
|
|
the thing you need.
|
|
|
|
**The first question is always: was the skip deliberate?** On this cluster a great deal of skipping is
|
|
intentional and correct.
|
|
## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE
|
|
|
|
`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**.
|
|
Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and
|
|
`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified
|
|
on prod 2026-09-13: the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups**
|
|
while the short name returned nothing. This false negative has already produced two wrong "the backups
|
|
are missing" conclusions in one night. **A destructive recovery action taken on it is how a CLI quirk
|
|
becomes real data loss.** Always:
|
|
|
|
```
|
|
backups.velero.io backups.postgresql.cnpg.io backups.longhorn.io
|
|
schedules.velero.io scheduledbackups.postgresql.cnpg.io clusters.postgresql.cnpg.io
|
|
```
|
|
|
|
## The deliberate exclusions (know these before calling anything a failure)
|
|
|
|
**NFS-sourced volumes are skipped on purpose, and the reason is a near-miss worth remembering.** The
|
|
opt-out file-system-backup schedule (`defaultVolumesToFsBackup: true`, excluding only `monitoring` and
|
|
`unifi`) would have read the NFS media library **three times** — `plex-media`, `sonarr-media` and
|
|
`radarr-media` are all bound to the same `192.168.2.102:/mnt/Storage/plex` export, **8.32 TiB
|
|
measured** — plus `nextcloud-legacy-archive` (270 GiB), pushing roughly **8.6 TiB to R2 nightly with
|
|
720h retention**. Namespace exclusion could not express the fix, because those namespaces also hold real
|
|
Longhorn PVCs that must be backed up.
|
|
|
|
The fix is a **per-volume opt-out by SOURCE TYPE**: the ConfigMap `velero-volume-policies` skips any
|
|
volume whose source is NFS, and the schedule references it. The per-pod
|
|
`backup-volumes-excludes` annotation was **rejected for failing OPEN** (forget the annotation and you
|
|
back up 8 TiB). Proven on prod: a Backup of namespace `nextcloud` produced exactly **one**
|
|
PodVolumeBackup (`data`, Longhorn, 875,417,102 bytes, Completed) and the NFS volume produced none.
|
|
**Residual gap, stated honestly: the rule matches NFS only.** A future non-NFS bulk source would not be
|
|
covered.
|
|
|
|
**The three CNPG databases are also out of scope by design.** `keycloak-db`, `nextcloud-db` and
|
|
`semaphore-db` have their own DR path (Barman WAL archive + base backups to R2) because a Velero snapshot
|
|
of a live Postgres is not a backup. If a `*-db-N` volume is skipped, that is correct — see
|
|
`/runbooks/cnpg-archiving` for the path that does cover them.
|
|
|
|
**`monitoring` and `unifi` are excluded** from file-system backup as well.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
Which backup, and what exactly it warned about:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get backups.velero.io -A \
|
|
-o custom-columns=NAME:.metadata.name,PHASE:.status.phase,ERRORS:.status.errors,WARN:.status.warnings,START:.status.startTimestamp'
|
|
```
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n velero get backups.velero.io <name> -o yaml | sed -n "/status:/,\$p"'
|
|
```
|
|
|
|
Per-volume results — this tells you whether a skip was the NFS policy or a real failure:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n velero get podvolumebackups.velero.io \
|
|
-o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,POD:.spec.pod.name,VOL:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone'
|
|
```
|
|
|
|
The volume policy that does the deliberate skipping:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n velero get cm velero-volume-policies -o yaml'
|
|
```
|
|
|
|
The controller's own account of what it skipped:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n velero logs deploy/velero --tail=200 2>&1 | grep -iE "skip|excluded|warn|error" | tail -30'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. The deliberate NFS / CNPG / monitoring exclusions
|
|
Most common by far, and correct. Confirm against the `velero-volume-policies` ConfigMap and the
|
|
schedule's excluded namespaces. **No action.** If you are tempted to "fix" it, re-read the 8.6 TiB
|
|
near-miss above.
|
|
|
|
### 2. A resource Velero could not serialise
|
|
Usually a CRD whose API was unavailable at the moment of the backup, or an admission webhook refusing a
|
|
read. The log names the group/kind. Often transient; if it recurs for the same kind, that kind is
|
|
genuinely not backed up.
|
|
|
|
### 3. A volume was busy, degraded, or on a node under pressure
|
|
Velero's daily runs at 03:00 against Longhorn PVCs. A degraded volume or a node with high I/O-PSI can
|
|
fail one volume's read while the rest succeed. Cross-read `/runbooks/longhorn-degraded` and
|
|
`/runbooks/node-io`.
|
|
|
|
### 4. A partial R2 write
|
|
Related to the tagging/checksum incompatibilities in `/runbooks/backup-failure`: those normally produce a
|
|
hard `Failed`, but a partially-written backup is worth checking the plugin pin for
|
|
(`velero/velero-plugin-for-aws:v1.14.3-rc.2` — do **not** revert to v1.14.2).
|
|
|
|
### 5. A namespace was created or deleted mid-backup
|
|
Benign race. Leaked test namespaces have existed on this cluster and been deleted; a namespace
|
|
disappearing under Velero produces exactly this warning.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Decide, and record, whether the skipped items matter.** The honest failure mode here is shrugging at
|
|
a `PartiallyFailed` and discovering later that something was never covered. If the skip was not one of
|
|
the documented exclusions, treat it as an uncovered resource.
|
|
- **Do not add namespaces to the exclusion list to quiet the alert.** Those namespaces hold real
|
|
Longhorn PVCs; namespace-level exclusion is the mistake the source-type policy exists to avoid.
|
|
- **Do not switch to the per-pod `backup-volumes-excludes` annotation** — it was rejected for failing
|
|
OPEN, and the consequence of forgetting it is 8 TiB per night.
|
|
- **Do not delete a PartiallyFailed backup** — like a Failed one it can leave orphaned kopia blobs, and
|
|
deleting R2 objects is an operator decision.
|
|
- Escalate if the same resource or volume is skipped on consecutive nights: that is a coverage gap, which
|
|
is an R-STATE-03 matter and not a transient.
|
|
|
|
---
|
|
Source of truth: `runbooks/backup-partial.md`. Index: `/runbooks/`
|
|
Records: commit `2e259a8` (the 8.6 TiB near-miss, the source-type volume policy, and the rejection of the
|
|
fail-open annotation); `docs/BULK-NFS-STORAGE.md`.
|