diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index b5f1327..90af052 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -2518,3 +2518,46 @@ source. Confirmed live: managedFields moved from `kubectl-client-side-apply` to not, and there is no urgency. - Testing HTTPS from inside a pod is misleading: nextcloud's Cilium egress policy returned envoy `403 Access denied` for other apps' FQDNs. Test from outside with `--resolve`. + +--- + +## 2026-09-13 — `kubectl get backup` LIES on this cluster (ambiguous short name) + +**Finding.** `kubectl get backup` / `get backup -A` prints **"No resources found"** on prod while +backups demonstrably exist. Three API groups each register a `Backup` kind: + +| group | kind | populated? | +|---|---|---| +| `longhorn.io/v1beta2` | Backup (`lhb`) | **no** — 0 objects | +| `postgresql.cnpg.io/v1` | Backup | yes — 27 (9 per CNPG cluster) | +| `velero.io/v1` | Backup | yes | + +The unqualified name resolves to Longhorn's, which is empty, so kubectl reports absence and exits 0. +Nothing warns about the ambiguity. + +**Why it matters more than a papercut.** This is a **false negative on the one question you ask +during a disaster**: "do I have backups?" The honest-looking answer is "No resources found". An +operator who trusts it concludes backups never ran and may act destructively — re-bootstrapping a +database, or re-seeding a volume — while good backups sit in R2 the whole time. It is the inverse of +this repo's recurring "a thing that looks applied and isn't": here, a thing that looks ABSENT and isn't. + +I hit it twice in one session and both times my first conclusion was wrong: +- On CNPG I wrote "(none — no base backup has ever been taken)" and started reasoning about an + unrecoverable platform. The Cluster `.status` contradicted me within a minute + (`lastSuccessfulBackup` hours old, `firstRecoverabilityPoint` 2026-09-07, `LastBackupSucceeded=True`), + and `get backups.postgresql.cnpg.io -A` then showed 27 completed backups. +- On Velero I read `get backup` as empty while `velero-daily-20260913040106` was *executing*, its + PodVolumeBackups logging "Async fs backup completed". + +**Rule adopted.** Always use fully-qualified resource names for any kind this cluster multiply +registers — `backups.velero.io`, `backups.postgresql.cnpg.io`, `backups.longhorn.io`. Never the bare +short name in a runbook, an alert annotation, a script, or an incident check. The same hazard applies +to `Cluster` (CNPG vs others) — prefer fully-qualified names generally. + +**Corollary for status-checking in general.** `.status` on the owning object was the reliable source +both times, and the object list was the misleading one. When the two disagree, trust the controller's +own status conditions and find out why the list is lying before acting. + +**Verified same session:** BSL `default` Available (validated 04:13Z); `velero-daily` unpaused and a +backup kicked off immediately; two Failed backups from 2026-09-12 (`prod-proof-20260912`, +`prod-vp-probe-20260912`) with orphaned kopia blobs still awaiting owner consent to delete.