docs: kubectl get backup lies on this cluster — ambiguous Backup short name

Three API groups register kind Backup (longhorn, cnpg, velero). The bare
short name resolves to Longhorn's empty set, so kubectl reports absence
while 27 CNPG backups and live Velero backups exist. A false negative on
'do I have backups' is the worst possible time to be wrong, so record the
rule: fully-qualified resource names only.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-12 23:15:06 -05:00
1 parent ebce6dfabb
commit 5e7ab65c65
1 file changed
+43
+43
View File
@@ -2518,3 +2518,46 @@ source. Confirmed live: managedFields moved from `kubectl-client-side-apply` to
not, and there is no urgency.
- Testing HTTPS from inside a pod is misleading: nextcloud's Cilium egress policy returned envoy
`403 Access denied` for other apps' FQDNs. Test from outside with `--resolve`.
---
## 2026-09-13 — `kubectl get backup` LIES on this cluster (ambiguous short name)
**Finding.** `kubectl get backup` / `get backup -A` prints **"No resources found"** on prod while
backups demonstrably exist. Three API groups each register a `Backup` kind:
| group | kind | populated? |
|---|---|---|
| `longhorn.io/v1beta2` | Backup (`lhb`) | **no** — 0 objects |
| `postgresql.cnpg.io/v1` | Backup | yes — 27 (9 per CNPG cluster) |
| `velero.io/v1` | Backup | yes |
The unqualified name resolves to Longhorn's, which is empty, so kubectl reports absence and exits 0.
Nothing warns about the ambiguity.
**Why it matters more than a papercut.** This is a **false negative on the one question you ask
during a disaster**: "do I have backups?" The honest-looking answer is "No resources found". An
operator who trusts it concludes backups never ran and may act destructively — re-bootstrapping a
database, or re-seeding a volume — while good backups sit in R2 the whole time. It is the inverse of
this repo's recurring "a thing that looks applied and isn't": here, a thing that looks ABSENT and isn't.
I hit it twice in one session and both times my first conclusion was wrong:
- On CNPG I wrote "(none — no base backup has ever been taken)" and started reasoning about an
unrecoverable platform. The Cluster `.status` contradicted me within a minute
(`lastSuccessfulBackup` hours old, `firstRecoverabilityPoint` 2026-09-07, `LastBackupSucceeded=True`),
and `get backups.postgresql.cnpg.io -A` then showed 27 completed backups.
- On Velero I read `get backup` as empty while `velero-daily-20260913040106` was *executing*, its
PodVolumeBackups logging "Async fs backup completed".
**Rule adopted.** Always use fully-qualified resource names for any kind this cluster multiply
registers — `backups.velero.io`, `backups.postgresql.cnpg.io`, `backups.longhorn.io`. Never the bare
short name in a runbook, an alert annotation, a script, or an incident check. The same hazard applies
to `Cluster` (CNPG vs others) — prefer fully-qualified names generally.
**Corollary for status-checking in general.** `.status` on the owning object was the reliable source
both times, and the object list was the misleading one. When the two disagree, trust the controller's
own status conditions and find out why the list is lying before acting.
**Verified same session:** BSL `default` Available (validated 04:13Z); `velero-daily` unpaused and a
backup kicked off immediately; two Failed backups from 2026-09-12 (`prod-proof-20260912`,
`prod-vp-probe-20260912`) with orphaned kopia blobs still awaiting owner consent to delete.