home-network-k3s-impl/runbooks/backup-partial.md

# backup-partial — A Velero backup completed with partial failures

**Alert:** `VeleroBackupPartialFailure` (warning, 5m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.backups`
**Expression:** `increase(velero_backup_partial_failure_total[1h]) > 0`

## What this means here

The backup finished, but some resources or volumes were skipped or errored. It is a warning rather than a
critical because *most of it worked* — and that is exactly what makes it dangerous: a
`PartiallyFailed` backup looks like a backup in every listing, and the thing it silently omitted may be
the thing you need.

**The first question is always: was the skip deliberate?** On this cluster a great deal of skipping is
intentional and correct.
## READ THIS BEFORE YOU CONCLUDE THE BACKUPS ARE GONE

`kubectl get backup -A` prints **"No resources found"** on this cluster **even when backups exist**.
Three CRDs register the kind `Backup` — `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and
`velero.io/v1` — and the bare short name resolves to **Longhorn's**, which holds zero objects. Verified
on prod 2026-09-13: the fully-qualified queries returned **6 Velero backups** and **27 CNPG backups**
while the short name returned nothing. This false negative has already produced two wrong "the backups
are missing" conclusions in one night. **A destructive recovery action taken on it is how a CLI quirk
becomes real data loss.** Always:

```
backups.velero.io           backups.postgresql.cnpg.io           backups.longhorn.io
schedules.velero.io         scheduledbackups.postgresql.cnpg.io  clusters.postgresql.cnpg.io
```

## The deliberate exclusions (know these before calling anything a failure)

**NFS-sourced volumes are skipped on purpose, and the reason is a near-miss worth remembering.** The
opt-out file-system-backup schedule (`defaultVolumesToFsBackup: true`, excluding only `monitoring` and
`unifi`) would have read the NFS media library **three times** — `plex-media`, `sonarr-media` and
`radarr-media` are all bound to the same `192.168.2.102:/mnt/Storage/plex` export, **8.32 TiB
measured** — plus `nextcloud-legacy-archive` (270 GiB), pushing roughly **8.6 TiB to R2 nightly with
720h retention**. Namespace exclusion could not express the fix, because those namespaces also hold real
Longhorn PVCs that must be backed up.

The fix is a **per-volume opt-out by SOURCE TYPE**: the ConfigMap `velero-volume-policies` skips any
volume whose source is NFS, and the schedule references it. The per-pod
`backup-volumes-excludes` annotation was **rejected for failing OPEN** (forget the annotation and you
back up 8 TiB). Proven on prod: a Backup of namespace `nextcloud` produced exactly **one**
PodVolumeBackup (`data`, Longhorn, 875,417,102 bytes, Completed) and the NFS volume produced none.
**Residual gap, stated honestly: the rule matches NFS only.** A future non-NFS bulk source would not be
covered.

**The three CNPG databases are also out of scope by design.** `keycloak-db`, `nextcloud-db` and
`semaphore-db` have their own DR path (Barman WAL archive + base backups to R2) because a Velero snapshot
of a live Postgres is not a backup. If a `*-db-N` volume is skipped, that is correct — see
`/runbooks/cnpg-archiving` for the path that does cover them.

**`monitoring` and `unifi` are excluded** from file-system backup as well.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

Which backup, and what exactly it warned about:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get backups.velero.io -A \
     -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,ERRORS:.status.errors,WARN:.status.warnings,START:.status.startTimestamp'
```

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n velero get backups.velero.io <name> -o yaml | sed -n "/status:/,\$p"'
```

Per-volume results — this tells you whether a skip was the NFS policy or a real failure:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n velero get podvolumebackups.velero.io \
     -o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,POD:.spec.pod.name,VOL:.spec.volume,PHASE:.status.phase,BYTES:.status.progress.bytesDone'
```

The volume policy that does the deliberate skipping:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n velero get cm velero-volume-policies -o yaml'
```

The controller's own account of what it skipped:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n velero logs deploy/velero --tail=200 2>&1 | grep -iE "skip|excluded|warn|error" | tail -30'
```

## Likely causes, most likely first

### 1. The deliberate NFS / CNPG / monitoring exclusions
Most common by far, and correct. Confirm against the `velero-volume-policies` ConfigMap and the
schedule's excluded namespaces. **No action.** If you are tempted to "fix" it, re-read the 8.6 TiB
near-miss above.

### 2. A resource Velero could not serialise
Usually a CRD whose API was unavailable at the moment of the backup, or an admission webhook refusing a
read. The log names the group/kind. Often transient; if it recurs for the same kind, that kind is
genuinely not backed up.

### 3. A volume was busy, degraded, or on a node under pressure
Velero's daily runs at 03:00 against Longhorn PVCs. A degraded volume or a node with high I/O-PSI can
fail one volume's read while the rest succeed. Cross-read `/runbooks/longhorn-degraded` and
`/runbooks/node-io`.

### 4. A partial R2 write
Related to the tagging/checksum incompatibilities in `/runbooks/backup-failure`: those normally produce a
hard `Failed`, but a partially-written backup is worth checking the plugin pin for
(`velero/velero-plugin-for-aws:v1.14.3-rc.2` — do **not** revert to v1.14.2).

### 5. A namespace was created or deleted mid-backup
Benign race. Leaked test namespaces have existed on this cluster and been deleted; a namespace
disappearing under Velero produces exactly this warning.

## When to escalate or stop

- **Decide, and record, whether the skipped items matter.** The honest failure mode here is shrugging at
  a `PartiallyFailed` and discovering later that something was never covered. If the skip was not one of
  the documented exclusions, treat it as an uncovered resource.
- **Do not add namespaces to the exclusion list to quiet the alert.** Those namespaces hold real
  Longhorn PVCs; namespace-level exclusion is the mistake the source-type policy exists to avoid.
- **Do not switch to the per-pod `backup-volumes-excludes` annotation** — it was rejected for failing
  OPEN, and the consequence of forgetting it is 8 TiB per night.
- **Do not delete a PartiallyFailed backup** — like a Failed one it can leave orphaned kopia blobs, and
  deleting R2 objects is an operator decision.
- Escalate if the same resource or volume is skipped on consecutive nights: that is a coverage gap, which
  is an R-STATE-03 matter and not a transient.

---
Source of truth: `runbooks/backup-partial.md`. Index: `/runbooks/`
Records: commit `2e259a8` (the 8.6 TiB near-miss, the source-type volume policy, and the rejection of the
fail-open annotation); `docs/BULK-NFS-STORAGE.md`.