From 5e7ab65c656f659dadf303d606faa6a2e6bd0fd9 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 23:15:06 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20kubectl=20get=20backup=20lies=20on=20th?= =?UTF-8?q?is=20cluster=20=E2=80=94=20ambiguous=20Backup=20short=20name?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three API groups register kind Backup (longhorn, cnpg, velero). The bare short name resolves to Longhorn's empty set, so kubectl reports absence while 27 CNPG backups and live Velero backups exist. A false negative on 'do I have backups' is the worst possible time to be wrong, so record the rule: fully-qualified resource names only. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index b5f1327..90af052 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -2518,3 +2518,46 @@ source. Confirmed live: managedFields moved from `kubectl-client-side-apply` to not, and there is no urgency. - Testing HTTPS from inside a pod is misleading: nextcloud's Cilium egress policy returned envoy `403 Access denied` for other apps' FQDNs. Test from outside with `--resolve`. + +--- + +## 2026-09-13 — `kubectl get backup` LIES on this cluster (ambiguous short name) + +**Finding.** `kubectl get backup` / `get backup -A` prints **"No resources found"** on prod while +backups demonstrably exist. Three API groups each register a `Backup` kind: + +| group | kind | populated? | +|---|---|---| +| `longhorn.io/v1beta2` | Backup (`lhb`) | **no** — 0 objects | +| `postgresql.cnpg.io/v1` | Backup | yes — 27 (9 per CNPG cluster) | +| `velero.io/v1` | Backup | yes | + +The unqualified name resolves to Longhorn's, which is empty, so kubectl reports absence and exits 0. +Nothing warns about the ambiguity. + +**Why it matters more than a papercut.** This is a **false negative on the one question you ask +during a disaster**: "do I have backups?" The honest-looking answer is "No resources found". An +operator who trusts it concludes backups never ran and may act destructively — re-bootstrapping a +database, or re-seeding a volume — while good backups sit in R2 the whole time. It is the inverse of +this repo's recurring "a thing that looks applied and isn't": here, a thing that looks ABSENT and isn't. + +I hit it twice in one session and both times my first conclusion was wrong: +- On CNPG I wrote "(none — no base backup has ever been taken)" and started reasoning about an + unrecoverable platform. The Cluster `.status` contradicted me within a minute + (`lastSuccessfulBackup` hours old, `firstRecoverabilityPoint` 2026-09-07, `LastBackupSucceeded=True`), + and `get backups.postgresql.cnpg.io -A` then showed 27 completed backups. +- On Velero I read `get backup` as empty while `velero-daily-20260913040106` was *executing*, its + PodVolumeBackups logging "Async fs backup completed". + +**Rule adopted.** Always use fully-qualified resource names for any kind this cluster multiply +registers — `backups.velero.io`, `backups.postgresql.cnpg.io`, `backups.longhorn.io`. Never the bare +short name in a runbook, an alert annotation, a script, or an incident check. The same hazard applies +to `Cluster` (CNPG vs others) — prefer fully-qualified names generally. + +**Corollary for status-checking in general.** `.status` on the owning object was the reliable source +both times, and the object list was the misleading one. When the two disagree, trust the controller's +own status conditions and find out why the list is lying before acting. + +**Verified same session:** BSL `default` Available (validated 04:13Z); `velero-daily` unpaused and a +backup kicked off immediately; two Failed backups from 2026-09-12 (`prod-proof-20260912`, +`prod-vp-probe-20260912`) with orphaned kopia blobs still awaiting owner consent to delete.