From 178d130dc3c00c6d856161f3769e15d193bfe089 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 29 Aug 2026 11:09:45 -0500 Subject: [PATCH] fix(velero): disable CRD-upgrade hook instead of pinning bitnami/kubectl MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Supersedes the previous kubectl-image repoint. The velero chart's pre-install `velero-upgrade-crds` HOOK job runs docker.io/bitnami/kubectl, whose public Docker Hub catalog Bitnami removed (Aug 2025); `bitnami/kubectl:1.31` 404s, leaving the hook in Init:ImagePullBackOff so Velero never installs. Repointing the hook to the exact-version rancher/kubectl:v1.31.5 (prior commit) does not work: that image is distroless, and the hook's init runs `/bin/sh -c "cp $(which sh)/... $(which kubectl) /tmp"` — with no /bin/sh it crashes StartError (verified on-cluster). The hook exists ONLY to UPDATE CRDs on chart upgrades — it is NOT how CRDs are first installed. The chart ships all 13 Velero CRDs in its native `crds/` directory, which Helm applies unconditionally on install (before hooks), independent of this flag; the CRDs are already present on-cluster from Helm's crds/ handling. Setting `upgradeCRDs: false` drops the fragile external kubectl-image dependency entirely while CRDs still install. Trade-off: Helm won't manage crds/ after install, so CRD schema upgrades become a manual/Renovate step; acceptable given Velero's CRDs are stable. All other Velero values (BSL -> versitygw s3://velero, kopia FSB node-agent, the two Schedules, the velero-cloud-credentials secret) are unchanged. Co-Authored-By: Claude Opus 4.8 --- platform/velero/helmrelease.yaml | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/platform/velero/helmrelease.yaml b/platform/velero/helmrelease.yaml index 31d56f1..6cfd8d6 100644 --- a/platform/velero/helmrelease.yaml +++ b/platform/velero/helmrelease.yaml @@ -40,15 +40,19 @@ spec: remediation: retries: 3 values: - # --- CRD upgrade/cleanup hook image. The chart default is docker.io/bitnami/kubectl, whose public - # Docker Hub catalog Bitnami REMOVED (Aug 2025) — `docker.io/bitnami/kubectl:1.31` 404s, leaving - # the pre-install `velero-upgrade-crds` hook in Init:ImagePullBackOff and Velero never installing. - # Repoint to the Rancher-maintained kubectl mirror, pinned to the cluster's exact k8s minor - # (v1.31.5, multi-arch amd64/arm64). Verified pullable via the Docker Hub registry API. --- - kubectl: - image: - repository: docker.io/rancher/kubectl - tag: v1.31.5 + # --- CRD install strategy. The chart's pre-install `velero-upgrade-crds` HOOK job runs + # docker.io/bitnami/kubectl, whose public Docker Hub catalog Bitnami REMOVED (Aug 2025) — + # `docker.io/bitnami/kubectl:1.31` now 404s, leaving the hook in Init:ImagePullBackOff so Velero + # never installs. The hook exists ONLY to UPDATE CRDs on chart upgrades; it is NOT how CRDs are + # first installed. The chart ships all 13 Velero CRDs (backups/restores/backupstoragelocations/ + # schedules/podvolumebackups/…) in its native `crds/` directory, which Helm applies + # UNCONDITIONALLY on install — before hooks — regardless of this flag. + # Disabling the hook therefore drops the fragile external-kubectl-image dependency entirely while + # the CRDs still install via Helm's crds/ handling. Trade-off: Helm never manages crds/ after the + # initial install, so CRD *schema upgrades* on future chart bumps become a manual/Renovate step — + # acceptable: Velero's CRDs are stable, and this is far less fragile than pinning a shell-bearing + # kubectl mirror (rancher/kubectl is distroless → no /bin/sh for the hook's copy-binary step). --- + upgradeCRDs: false # --- AWS object-store plugin (BSL provider=aws → MinIO S3-compatible), loaded as an init container. # Image minor MUST track the Velero minor (chart 8.x = Velero 1.15 → plugin v1.11.x). --- initContainers: