From 60e3a246523506c728e60f0bd3374fcbc3f03ee8 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 7 Nov 2026 23:18:21 -0600 Subject: [PATCH] feat(devcontainer): bake velero CLI into the toolbox, pinned to prod server v1.18.2 The velero CLI and server are version-coupled, so VELERO_VERSION follows the server running on prod (docker.io/velero/velero:v1.18.2, chart 12.2.3, verified 2026-11-07). It has no renovate marker on purpose: bump it together with the chart's appVersion. The verify stage asserts that `velero version --client-only` reports the pin. The build arg reaches CI through build-image.sh --print-build-args, so no .gitea/ or ci/ change is needed. Co-Authored-By: Claude Opus 5.5 --- .../2026-11-07-devcontainer-velero-cli.md | 54 +++++++++++++++++++ infra/devcontainer/Containerfile | 15 ++++-- infra/devcontainer/README.md | 1 + infra/devcontainer/build-image.sh | 1 + infra/devcontainer/versions.env | 15 ++++++ runbooks/backup-failure.md | 10 ++++ 6 files changed, 93 insertions(+), 3 deletions(-) create mode 100644 decisions/2026-11-07-devcontainer-velero-cli.md diff --git a/decisions/2026-11-07-devcontainer-velero-cli.md b/decisions/2026-11-07-devcontainer-velero-cli.md new file mode 100644 index 0000000..147dd5e --- /dev/null +++ b/decisions/2026-11-07-devcontainer-velero-cli.md @@ -0,0 +1,54 @@ +--- +date: 2026-11-07 +branch: feat/devcontainer-velero-cli +title: Bake the velero CLI into the dev toolbox, pinned to the prod server version +--- + +## What changed + +- `infra/devcontainer/versions.env`: new `VELERO_VERSION=1.18.2`, with a comment explaining that + the CLI and server are version-coupled, so this pin follows the server and is not bumped on + its own. It has **no** `# renovate:` marker on purpose. When a `velero` chart bump in + `platform/velero/helmrelease.yaml` changes the appVersion, bump this pin in the same PR. +- `infra/devcontainer/Containerfile`: `ARG VELERO_VERSION`; the binary comes from the + vmware-tanzu GitHub release tarball and is installed in the same RUN as flux/helm. There is + no checksum check, which matches the other tools in that file (none of them verify one). + `velero=` is recorded in `/etc/dev-toolchain-versions`. The verify stage now checks that + `velero` is on PATH, and `check velero v${VELERO_VERSION}` asserts the output of + `velero version --client-only`. +- `infra/devcontainer/build-image.sh`: `VELERO_VERSION` added to `BUILD_ARGS`. CI reads that + list generically through `--print-build-args` (`.gitea/workflows/scripts/images.sh`), so + **no sensitive path (`.gitea/**`, `ci/**`) was touched**. +- `infra/devcontainer/README.md`: added a Backups row to the tool table. +- `runbooks/backup-failure.md`: under "First commands", a read-only + `velero backup describe --details` / `backup logs` example run through dev.sh. + +## What I verified + +- Live server version, read-only: `kubectl --context k3s-prod -n velero get deploy velero` + shows image `docker.io/velero/velero:v1.18.2`, and HelmRelease `velero` has + lastAttemptedRevision `12.2.3`. +- Local build: `IMAGE=localhost/home-k3s-dev:velero-wt ./build-image.sh` returned rc 0. The + verify stage printed `ok velero` (on PATH), `ok velero v1.18.2` (pin check) and + `TOOLBOX VERIFIED`. The image was built under a separate local tag and **not pushed**. +- Smoke test with that image against prod (read-only): `velero version` reports Client v1.18.2 + and Server v1.18.2. `velero backup get` lists the daily backups, all `Completed` with 0 errors. +- Recurring warning (reported only, not fixed): every daily backup since + `velero-daily-20261031030050` has 1 warning. Backups up to 20261030 have 0. The warning text is: + `gitea-actions: resource: /pods name: /act-runner-5944dfdfb8-69bvq message: /Skip pod volume + data error: /pod is not in the expected status, ... phase=Failed: pod is not running`. + It is the same pod in 20261031, 20261104 and 20261108. `gitea-actions` holds 11 `Failed` + pods with reason `Evicted` from ReplicaSet `act-runner-5944dfdfb8`, ten of them started + 2026-10-30T08:26Z. The running runner is `act-runner-fc597f6b5-vsvkl`. So the warning comes from a + leftover evicted (`Failed`) pod that Velero still tries to back up volumes for. Cleaning up the evicted pods is an + operator action. +- Gate: see the PR description for the toolbox `ci/gate.sh` rc and the skipped stages. + +## What I did NOT verify + +- The CI image build (`.gitea/workflows/images.yaml`) on this branch. It runs on push/PR and I + have not seen its result while writing this. +- An arm64 build. Only amd64 was built (`TARGETARCH` defaults to amd64). +- Any mutating velero command (`backup create`, `restore`, `delete`). None were run. +- Why only `-69bvq` of the 11 evicted act-runner pods shows up in the warning. A likely guess is + that only that pod's volumes match the backup's pod-volume selection, but I did not check. diff --git a/infra/devcontainer/Containerfile b/infra/devcontainer/Containerfile index 469d199..a1a3c65 100644 --- a/infra/devcontainer/Containerfile +++ b/infra/devcontainer/Containerfile @@ -30,6 +30,9 @@ ARG CHECK_JSONSCHEMA_VERSION # Pinned to the Prometheus version prod actually runs — see versions.env for why that lockstep is # the point, and for the live-cluster command that establishes it. ARG PROMTOOL_VERSION +# velero: operator CLI for the backup server. Pinned to the server version prod runs — the CLI +# and server are version-coupled; see versions.env. +ARG VELERO_VERSION ARG TARGETARCH=amd64 # Distro packages: libvirt client libs (harmless disk-image utilities), ansible, ssh, @@ -66,6 +69,10 @@ RUN set -eux; \ curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl" -o /usr/local/bin/kubectl; chmod +x /usr/local/bin/kubectl; \ curl -fsSL "https://get.helm.sh/helm-v${HELM_VERSION}-linux-${TARGETARCH}.tar.gz" | tar -xz -C /tmp; \ install -m0755 "/tmp/linux-${TARGETARCH}/helm" /usr/local/bin/helm; \ + VELERO_TARBALL="velero-v${VELERO_VERSION}-linux-${TARGETARCH}"; \ + curl -fsSL "https://github.com/vmware-tanzu/velero/releases/download/v${VELERO_VERSION}/${VELERO_TARBALL}.tar.gz" \ + | tar -xz -C /usr/local/bin --strip-components=1 "${VELERO_TARBALL}/velero"; \ + chmod +x /usr/local/bin/velero; \ rm -rf /tmp/* # Pinned conformance test toolchain (L0/L1 static gate + the L2/L3 Chainsaw runner). @@ -93,7 +100,7 @@ RUN { echo "opentofu=${OPENTOFU_VERSION}"; echo "age=${AGE_VERSION}"; echo "sops echo "kyverno=${KYVERNO_VERSION}"; echo "kyverno-json=${KYVERNO_JSON_VERSION}"; \ echo "chainsaw=${CHAINSAW_VERSION}"; echo "yamllint=${YAMLLINT_VERSION}"; \ echo "yq=${YQ_VERSION}"; echo "check-jsonschema=${CHECK_JSONSCHEMA_VERSION}"; \ - echo "promtool=${PROMTOOL_VERSION}"; \ + echo "promtool=${PROMTOOL_VERSION}"; echo "velero=${VELERO_VERSION}"; \ } > /etc/dev-toolchain-versions WORKDIR /work @@ -112,6 +119,7 @@ ARG KUBECTL_VERSION ARG CHAINSAW_VERSION ARG KYVERNO_VERSION ARG SOPS_VERSION +ARG VELERO_VERSION RUN --network=none /bin/bash -euo pipefail <<'VERIFY' echo "--- no network (the build-time stand-in for run --network none) ---" if getent hosts registry.fedoraproject.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then @@ -122,13 +130,13 @@ echo "--- baked versions ---"; cat /etc/dev-toolchain-versions echo "--- every tool is on PATH ---" for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \ yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \ - wg qrencode; do + wg qrencode velero; do if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t"; else printf " MISSING %s\n" "$t"; missing=1; fi done [ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; } echo "--- version-sensitive tools report their pins ---" -: "${PROMTOOL_VERSION:?}" "${KUBECTL_VERSION:?}" "${CHAINSAW_VERSION:?}" "${KYVERNO_VERSION:?}" "${SOPS_VERSION:?}" +: "${PROMTOOL_VERSION:?}" "${KUBECTL_VERSION:?}" "${CHAINSAW_VERSION:?}" "${KYVERNO_VERSION:?}" "${SOPS_VERSION:?}" "${VELERO_VERSION:?}" check() { # name pin actual-output if printf "%s" "$3" | grep -qF "$2"; then printf " ok %-10s %s\n" "$1" "$2"; else printf " DRIFT %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi @@ -138,6 +146,7 @@ check kubectl "${KUBECTL_VERSION}" "$(kubectl version --client 2>&1)" check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)" check kyverno "${KYVERNO_VERSION}" "$(kyverno version 2>&1)" check sops "${SOPS_VERSION}" "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)" +check velero "v${VELERO_VERSION}" "$(velero version --client-only 2>&1)" [ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; } echo "--- python can import what the renderer and the gate need ---" python3 -c "import yaml; print(' PyYAML', yaml.__version__)" diff --git a/infra/devcontainer/README.md b/infra/devcontainer/README.md index 6ff9d40..46478bb 100644 --- a/infra/devcontainer/README.md +++ b/infra/devcontainer/README.md @@ -14,6 +14,7 @@ baseline, and the *same* binaries CI uses. | Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` | | GitOps / cluster | `flux`, `kubectl`, `helm` | | Secrets | `sops`, `age` + `age-keygen` | +| Backups (`platform/velero`) | `velero` CLI — pinned to the **server** version prod runs (version-coupled; see `versions.env`) | | VPN (`platform/vpn-ingress`) | `wg` (wireguard-tools) — mint the server key, derive client public keys; `qrencode` — render a client profile as a phone-scannable QR | | **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` | | Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) | diff --git a/infra/devcontainer/build-image.sh b/infra/devcontainer/build-image.sh index 07c9cc4..203cb22 100755 --- a/infra/devcontainer/build-image.sh +++ b/infra/devcontainer/build-image.sh @@ -54,6 +54,7 @@ BUILD_ARGS=( "KYVERNO_JSON_VERSION=$KYVERNO_JSON_VERSION" "CHAINSAW_VERSION=$CHAINSAW_VERSION" "YAMLLINT_VERSION=$YAMLLINT_VERSION" "YQ_VERSION=$YQ_VERSION" "CHECK_JSONSCHEMA_VERSION=$CHECK_JSONSCHEMA_VERSION" "PROMTOOL_VERSION=$PROMTOOL_VERSION" + "VELERO_VERSION=$VELERO_VERSION" ) if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi diff --git a/infra/devcontainer/versions.env b/infra/devcontainer/versions.env index 2c8acea..0207644 100644 --- a/infra/devcontainer/versions.env +++ b/infra/devcontainer/versions.env @@ -44,4 +44,19 @@ CHECK_JSONSCHEMA_VERSION=0.29.4 # promtool ships INSIDE the prometheus release tarball (prometheus-.linux-.tar.gz), so # the version string here is a Prometheus release, not a separate promtool release. PROMTOOL_VERSION=3.14.0 +# velero — the operator CLI for the Velero backup server (platform/velero): `velero backup +# describe --details`, `velero backup logs`, `velero backup create --from-schedule`. +# +# THIS PIN IS NOT FREE-FLOATING: the CLI and server are VERSION-COUPLED (the CLI talks to the +# server's CRDs and its download-request protocol; upstream supports only a matching minor), so +# it MUST equal the Velero server version actually running on prod. +# Verified against the live cluster on 2026-11-07: +# kubectl --context k3s-prod -n velero get deploy velero \ +# -o jsonpath='{.spec.template.spec.containers[0].image}' +# -> docker.io/velero/velero:v1.18.2 (HelmRelease velero, chart velero-12.2.3) +# Deliberately carries NO `# renovate:` marker: it must move WITH the server, i.e. when a +# vmware-tanzu `velero` chart bump (platform/velero/helmrelease.yaml, range >=12 <13) changes the +# chart's appVersion, bump this in the same PR. Never bump it on its own. The verify stage +# asserts `velero version --client-only` reports exactly this version. +VELERO_VERSION=1.18.2 # Proxmox provider (bpg/proxmox) is pinned in the tofu module (infra/tofu/versions.tf), not here. diff --git a/runbooks/backup-failure.md b/runbooks/backup-failure.md index f4422ed..6d80790 100644 --- a/runbooks/backup-failure.md +++ b/runbooks/backup-failure.md @@ -99,6 +99,16 @@ infra/devcontainer/dev.sh run bash -c \ -o custom-columns=NAME:.metadata.name,BACKUP:.metadata.labels.velero\\.io/backup-name,PHASE:.status.phase,BYTES:.status.progress.bytesDone' ``` +The `velero` CLI is in the toolbox (pinned to the server version) and gives the per-item errors and +warnings the CR only counts. Read-only; `velero backup create/delete` and `velero restore` are +operator actions: + +``` +infra/devcontainer/dev.sh run bash -c \ + 'velero --kubecontext k3s-prod -n velero backup describe --details; \ + velero --kubecontext k3s-prod -n velero backup logs | grep -iE "level=(error|warn)" | tail -25' +``` + The plugin pin (cause 1 depends entirely on this): ```