feat(backup): s3check proves a real round trip — and revives a guard I had killed
*** THE BUG, AND IT WAS MINE. *** The placeholder guards were `case "$s3" in
""|TBD-*)` — a PREFIX glob. Two commits earlier (32d6766) I changed the
placeholder from `TBD-PROD-S3-BACKUP-ENDPOINT` to
`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`, which does NOT start with
`TBD-`. Verified: fnmatch('https://TBD-R2-...', 'TBD-*') is False. All three
guards silently stopped matching, so **step_resume would have resumed velero on
prod straight onto a nonexistent hostname** — velero installs, reports healthy,
and no backup exists. That is precisely the failure the guard was written to
prevent, disarmed by a value change in a DIFFERENT file with nothing tying the
two together. Replaced with substring-matching s3_is_placeholder in preflight,
s3check and resume.
s3check rewritten from a reachability ping into a real proof. Per credential, from
a throwaway in-cluster pod (pod egress is the only egress that matters — the
workstation sits on a different network side and a host-side reading has already
misled this project once):
1. the credential LOADS AND AUTHENTICATES in the exact shape its consumer reads
— Velero's INI mounted from the `cloud` key, CNPG's discrete env keys;
2. the bucket EXISTS and is LISTABLE under this environment's own prefix, via
ListObjectsV2 — the op Velero's BSL sync and Barman's WAL scan actually use.
ListBuckets deliberately avoided: a bucket-scoped R2 token may not grant it;
3. PUT -> GET -> sha256+length compare -> DELETE -> confirm-gone. A HEAD would
pass with a read-only token and with every write being rejected;
4. the R2 checksum trap probed ON PURPOSE by repeating the PUT with checksums
enabled, and named either way.
Buckets and prefixes are DERIVED (velero's from the authored HelmRelease via yq,
cnpg's parsed from CNPG_BACKUP_PATH), not restated.
Fails closed, with no "unknown, carrying on" arm left: no pod, unfinished pod,
empty log, missing/duplicate/malformed verdict, any failed assertion, a claimed
PASS with failures>0, an assertion count below the caller-computed minimum, an
OK-line count disagreeing with the claimed count, or a PASS from a pod that ended
Failed. Failures are classified by cause — no egress, NoSuchBucket,
InvalidAccessKeyId, AccessDenied (token not scoped to BOTH buckets),
InvalidArgument -> the checksum trap.
`amazon/aws-cli` pinned rather than `minio/mc`: mc lacks sed/awk (which already
broke one probe) AND cannot send checksum headers, so it cannot test for the trap.
TIER_ORDER now includes platform-config-cdi, between platform-config and the leaf
tiers. Without it `resume` returned success without ever waiting on the tier that
guarantees the CDI StorageProfiles — and apps-vms depends on them: unifi
provisions on longhorn-vm, where an unapplied profile means CDI infers RWO instead
of the authored RWX, permanently non-live-migratable.
CREDENTIALS: one key, four committed Secrets, two shapes — and NO manifest change
is needed, only encrypted values. platform/backup/set-r2-credentials.sh writes all
four with `sops set`, which encrypts IN PLACE, so no decrypted Secret ever exists
in the working tree to be committed by accident — strictly better than staging a
plaintext template. Prompts with `read -rs`, rejects anything outside the S3
credential charset (a paste that swallowed a newline authenticates nowhere), never
prints a value, and verifies by decrypting back and comparing fingerprints.
`--check` is read-only. The MinIO key is RETAINED — it is what reads the
pre-migration backups.
THE ACCOUNT ID IS ALREADY IN THE REPO: infra/tofu/backend.r2.hcl commits
https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com for the tofu
state bucket. If the backup buckets go in the same account, only the two token
values are needed. The placeholder is KEPT anyway — fail-closed beats convenience,
and the guard above just demonstrated why.
Corroborating the checksum fix independently: infra/tofu/backend.tf already carries
`region = "auto"` and `skip_s3_checksum = true # R2 rejects AWS trailing
(streaming) checksums`. This repo had learned both R2 facts for tofu state already.
VERSION FINDINGS, documented not measured against R2: plugin v1.14.2 documents
checksumAlgorithm defaulting to CRC32, so the authored "" is load-bearing;
Velero's kopia backend has NO checksum knob, so BSL metadata and PVC-data
compatibility are SEPARATE questions; and in the live CNPG 16.4 image,
boto3/botocore 1.35.60 is below the 1.36 line that turned on default request
checksums — so CNPG should be clear today, but barman-cloud exposes no escape
hatch, making a CNPG image bump an R2 re-verification event.
VERIFIED BY ME, independently of the agent: the dead-guard glob (fnmatch False);
the placeholder now refused on prod before any pod is created; a full round trip
against dev's MinIO — 7/7 assertions on BOTH credential paths, byte-identical,
pod Succeeded, nothing left behind, checksum trap correctly reported absent; all
four Secrets and the committed account id present. Full CI gate PASS.
STILL UNPROVEN AND SAID SO IN THE DOC: Velero-to-R2 (metadata or kopia data),
CNPG-barman-to-R2, and any restore from R2. R-STATE-03 is PROVEN MinIO-ONLY and
its ledger row predates this move. infra/BACKUP-R2.md §5 is the re-proof sequence
and it ends in a RESTORE, not an upload.
Refs R-STATE-03, R-IDP-05, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
59ffbaf11a
commit
8f9b2e4270
7 files changed
+1088
-63
No files matched your search
+505
-48
@@ -26,6 +26,11 @@
|
||||
#
|
||||
# --confirm-destructive is required by `unwedge` only. Everything else is additive or reversible.
|
||||
#
|
||||
# --s3-endpoint <url> makes `s3check` probe a CANDIDATE backup endpoint instead of the live
|
||||
# cluster-vars value, so a proposed target can be PROVEN before it is authored into cluster-vars.
|
||||
# `s3check` writes and then deletes ONE throwaway object under `<prefix>/.s3check/`; it changes
|
||||
# nothing else and applies nothing. See infra/BACKUP-R2.md.
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
# ---------------------------------------------------------------------------- args
|
||||
@@ -85,10 +90,18 @@ say "target confirmed: ${live_server}"
|
||||
|
||||
# Dependency order. Flux enforces the edges itself via dependsOn; resuming in this order simply means
|
||||
# each wait is meaningful instead of blocking on a dependency that is still suspended.
|
||||
# flux-system -> cilium -> platform -> platform-config -> {leaf tiers, apps-wave-0} -> wave-1 -> wave-2
|
||||
TIER_ORDER=(platform platform-config oauth2-proxy headlamp homepage crafty apps-wave-0 apps-wave-1 apps-wave-2 apps-vms)
|
||||
# velero is NOT in that list on purpose — see step `resume`: it consumes S3_BACKUP_ENDPOINT, which on
|
||||
# prod is an intentionally-invalid TBD placeholder pending an operator decision.
|
||||
# flux-system -> cilium -> platform -> platform-config -> platform-config-cdi
|
||||
# -> {leaf tiers, apps-wave-0} -> wave-1 -> wave-2 -> apps-vms
|
||||
#
|
||||
# `platform-config-cdi` MUST stay in this list and MUST stay before apps-vms. It carries the CDI
|
||||
# StorageProfiles, which were split out of platform-config because StorageProfile's CRD does not exist
|
||||
# until cdi-operator has reconciled the CDI CR (clusters/base/platform-config-cdi.yaml). apps-vms
|
||||
# dependsOn it: apps/vms/unifi provisions on `longhorn-vm`, and without the profile applied CDI infers
|
||||
# ReadWriteOnce instead of the authored ReadWriteMany — permanently non-live-migratable for that disk.
|
||||
# Omitting it here would make `resume` return success without ever waiting on the tier that guarantees it.
|
||||
TIER_ORDER=(platform platform-config platform-config-cdi oauth2-proxy headlamp homepage crafty apps-wave-0 apps-wave-1 apps-wave-2 apps-vms)
|
||||
# velero is NOT in that list on purpose — see step `resume`: it consumes S3_BACKUP_ENDPOINT, which is
|
||||
# an intentionally-invalid placeholder until the operator supplies the real R2 endpoint.
|
||||
|
||||
cvar() { kubectl -n flux-system get cm cluster-vars -o jsonpath="{.data.$1}" 2>/dev/null || true; }
|
||||
|
||||
@@ -150,6 +163,354 @@ dns_diff() { # <before> <after> — returns non-zero if anything changed
|
||||
return 1
|
||||
}
|
||||
|
||||
# ======================================================================= s3check plumbing
|
||||
# WHY THIS IS THIS BIG. The previous version of `s3check` curl'd /minio/health/live and reported
|
||||
# REACHABLE. That probe could not pass against Cloudflare R2 (no such path), and reachability is not
|
||||
# the property anyone cares about: a reachable endpoint with a wrong key, a missing bucket, or R2's
|
||||
# checksum rejection still yields a velero tier that reports Ready while NO backup exists. What must
|
||||
# be proven is the write path Velero and CNPG-barman actually use, with the credentials they actually
|
||||
# hold, from the network position they actually occupy.
|
||||
|
||||
# Pinned. The AWS CLI is a deliberate choice over `minio/mc`:
|
||||
# - it prints the S3 ERROR CODE (NoSuchBucket / AccessDenied / InvalidAccessKeyId / InvalidArgument),
|
||||
# so a failure can be NAMED and turned into a fix instead of guessed at from an HTTP status;
|
||||
# - it exposes the FLEXIBLE-CHECKSUM behaviour that is the R2 trap
|
||||
# (AWS_REQUEST_CHECKSUM_CALCULATION), so the trap can be reproduced on purpose here instead of
|
||||
# discovered by a silently-failing backup at 03:00;
|
||||
# - its image carries bash + coreutils. `minio/mc` ships no sed/awk/diff — that minimalism has
|
||||
# already broken one probe in this repo — and it cannot be made to send the checksum headers, so
|
||||
# it cannot test for the trap at all.
|
||||
S3CHECK_IMAGE="amazon/aws-cli:2.36.40"
|
||||
S3CHECK_RUN_ID="$(date +%s)"
|
||||
|
||||
# Throwaway probe pods, tracked so they are removed on EVERY exit path (including `die`). A probe
|
||||
# must never leave a pod behind on a live cluster.
|
||||
S3CHECK_PODS=()
|
||||
s3_cleanup_pods() {
|
||||
local p
|
||||
for p in ${S3CHECK_PODS[@]+"${S3CHECK_PODS[@]}"}; do
|
||||
kubectl -n "${p%%/*}" delete pod "${p#*/}" --ignore-not-found --wait=false >/dev/null 2>&1 || true
|
||||
done
|
||||
S3CHECK_PODS=()
|
||||
}
|
||||
trap s3_cleanup_pods EXIT
|
||||
|
||||
# TRUE when the endpoint is not a usable target. `TBD-*` alone was WRONG once the placeholder became
|
||||
# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`: that does not START with TBD-, so the old
|
||||
# prefix pattern matched nothing and `resume` would have resumed velero straight onto a placeholder
|
||||
# hostname — the exact silent-no-backups failure the guard exists to prevent. Match anywhere.
|
||||
s3_is_placeholder() { # <endpoint>
|
||||
case "$1" in
|
||||
""|*TBD*|*tbd-*|*REPLACE-ME*|*replace-me*|*CHANGE-ME*|*change-me*) return 0 ;;
|
||||
esac
|
||||
return 1
|
||||
}
|
||||
|
||||
# Refuse any value that is not a plain URL/name fragment. This is BOTH a safety guard — these values
|
||||
# are interpolated into the probe pod's JSON overrides AND into a shell script that runs inside it, so
|
||||
# a quote, a backslash or a newline could rewrite either — and a correctness guard: a cluster-vars
|
||||
# value with a trailing space or a stray quote is a defect, not something to paper over.
|
||||
s3_safe_token() { # <label> <value>
|
||||
case "$2" in
|
||||
*[!A-Za-z0-9:/._-]*) die "refusing to probe: $1 contains a character outside [A-Za-z0-9:/._-].
|
||||
That is either a corrupt cluster-vars value or an injection attempt. Value (quoted): $(printf '%q' "$2")" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- the in-pod probe
|
||||
# Emitted to the pod base64-encoded, so nothing in it has to survive JSON or shell quoting.
|
||||
# CONTRACT WITH THE CALLER (this is what makes a silent no-op impossible):
|
||||
# - every proven property prints exactly one `S3CHECK-OK[n] ...` line and increments n;
|
||||
# - every failure prints `S3CHECK-FAIL ...` and is counted;
|
||||
# - the LAST line is `S3CHECK-RESULT PASS|FAIL assertions=<n> failures=<n>`.
|
||||
# The caller independently knows how many assertions this environment must produce and re-counts the
|
||||
# OK lines itself, so neither a truncated log, a skipped bucket, an empty log, nor a lying result line
|
||||
# can be read as success. (A CRD guard in this repo once counted a blank line as "1 CRD" and sailed
|
||||
# past; that class of bug is designed out here rather than commented about.)
|
||||
s3_probe_script() {
|
||||
cat <<'PROBE'
|
||||
set -u
|
||||
umask 077
|
||||
A=0; F=0
|
||||
ok() { A=$((A+1)); printf 'S3CHECK-OK[%d] %s\n' "$A" "$*"; }
|
||||
bad() { F=$((F+1)); printf 'S3CHECK-FAIL %s\n' "$*"; }
|
||||
note() { printf 'S3CHECK-NOTE %s\n' "$*"; }
|
||||
verdict() {
|
||||
if [ "${F}" -eq 0 ] && [ "${A}" -gt 0 ]; then
|
||||
printf 'S3CHECK-RESULT PASS assertions=%d failures=0\n' "${A}"; exit 0
|
||||
fi
|
||||
printf 'S3CHECK-RESULT FAIL assertions=%d failures=%d\n' "${A}" "${F}"; exit 1
|
||||
}
|
||||
|
||||
for v in S3EP S3REGION S3TARGETS S3KEY CRED_MODE; do
|
||||
eval "x=\${${v}-}"
|
||||
[ -n "${x}" ] || { bad "required env ${v} is empty — the probe was launched wrong"; verdict; }
|
||||
done
|
||||
|
||||
export AWS_EC2_METADATA_DISABLED=true
|
||||
export AWS_PAGER=""
|
||||
export AWS_DEFAULT_REGION="${S3REGION}"
|
||||
export AWS_REGION="${S3REGION}"
|
||||
export AWS_MAX_ATTEMPTS=2
|
||||
export AWS_RETRY_MODE=standard
|
||||
export AWS_CONFIG_FILE=/tmp/s3check-aws-config
|
||||
# Path-style addressing, because both targets need it: MinIO on a bare IP has no virtual-host names,
|
||||
# and R2's S3 API is documented path-style. Checksums OFF, mirroring `checksumAlgorithm: ""` on the
|
||||
# Velero BSL — this run must exercise the same wire shape the real backup will use. Written to the
|
||||
# CONFIG FILE as well as the environment so the setting holds even if one channel is ignored by a
|
||||
# future CLI version.
|
||||
{
|
||||
echo '[default]'
|
||||
echo 'request_checksum_calculation = when_required'
|
||||
echo 'response_checksum_validation = when_required'
|
||||
echo 's3 ='
|
||||
echo ' addressing_style = path'
|
||||
} > "${AWS_CONFIG_FILE}"
|
||||
|
||||
# ---- (1) the credential loads at all -------------------------------------------------
|
||||
# NEVER print a credential value, in either `key: value` or `key=value` shape. Only booleans,
|
||||
# byte counts and lengths leave this pod.
|
||||
case "${CRED_MODE}" in
|
||||
file)
|
||||
cf="${AWS_SHARED_CREDENTIALS_FILE-}"
|
||||
if [ -z "${cf}" ] || [ ! -s "${cf}" ]; then
|
||||
bad "credential file '${cf:-<unset>}' is missing or empty — the Secret key did not mount"
|
||||
elif ! grep -Eq '^[[:space:]]*\[default\][[:space:]]*$' "${cf}"; then
|
||||
bad "credential file has no [default] profile. velero-plugin-for-aws reads an AWS INI file from
|
||||
the 'cloud' key of velero-cloud-credentials; without [default] every backup fails at RUNTIME"
|
||||
elif ! grep -Eq '^[[:space:]]*aws_access_key_id[[:space:]]*=[[:space:]]*[^[:space:]]' "${cf}" \
|
||||
|| ! grep -Eq '^[[:space:]]*aws_secret_access_key[[:space:]]*=[[:space:]]*[^[:space:]]' "${cf}"; then
|
||||
bad "credential file [default] lacks a non-empty aws_access_key_id / aws_secret_access_key"
|
||||
else
|
||||
ok "credential LOADS: AWS INI with [default] + both keys ($(wc -c < "${cf}") bytes; values never printed)"
|
||||
fi
|
||||
;;
|
||||
env)
|
||||
if [ -z "${AWS_ACCESS_KEY_ID-}" ] || [ -z "${AWS_SECRET_ACCESS_KEY-}" ]; then
|
||||
bad "AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY did not arrive from the Secret (missing ACCESS_KEY_ID / ACCESS_SECRET_KEY key?)"
|
||||
else
|
||||
ok "credential LOADS: access key id present (${#AWS_ACCESS_KEY_ID} chars; values never printed)"
|
||||
fi
|
||||
;;
|
||||
*) bad "unknown CRED_MODE '${CRED_MODE}'" ;;
|
||||
esac
|
||||
[ "${F}" -eq 0 ] || verdict # no point talking to S3 with no credential
|
||||
|
||||
OUT=""; RC=0
|
||||
awsx() { OUT="$(aws --endpoint-url "${S3EP}" "$@" 2>&1)"; RC=$?; return 0; }
|
||||
|
||||
detail() { printf '%s\n' "${OUT}" | head -n 12 | while IFS= read -r l; do printf 'S3CHECK-DETAIL %s\n' "${l}"; done; }
|
||||
|
||||
# Every failure is NAMED and given the fix. An unclassified failure is still a failure.
|
||||
diagnose() { # <operation label>
|
||||
local op="$1"
|
||||
case "${OUT}" in
|
||||
*InvalidArgument*|*checksum*|*Checksum*)
|
||||
bad "${op}: 400 InvalidArgument — THE R2 CHECKSUM TRAP. velero-plugin-for-aws v1.14+ bundles an
|
||||
AWS SDK that sends x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT, and
|
||||
Cloudflare R2 REJECTS those headers with HTTP 400 InvalidArgument, so every Velero backup and
|
||||
every kopia blob upload fails. FIX: checksumAlgorithm: \"\" must be present in the
|
||||
backupStorageLocation config in platform/velero/helmrelease.yaml. This probe already disables
|
||||
checksums, so seeing this here means the client sent one anyway — treat it as a hard blocker." ;;
|
||||
*"Could not connect"*|*EndpointConnectionError*|*ConnectTimeoutError*|*"Failed to establish"*|\
|
||||
*"Name or service not known"*|*"nodename nor servname"*|*"Temporary failure in name resolution"*|\
|
||||
*"Connection reset"*|*"Read timeout"*)
|
||||
bad "${op}: this POD cannot reach ${S3EP} (connect / DNS / timeout). Velero and CNPG would
|
||||
install perfectly clean and then fail every backup at RUNTIME — a tier that looks healthy
|
||||
while no backup exists. This is the reading that matters: the workstation's own reachability
|
||||
is irrelevant (prod sits on the SOHO net, pfSense's WAN; the dev LAN's MinIO sits INSIDE
|
||||
pfSense) and a host-side probe has already misled this project once. Fix pod egress/routing,
|
||||
or give this environment its own endpoint." ;;
|
||||
*NoSuchBucket*|*"(404)"*|*"Not Found"*)
|
||||
bad "${op}: the bucket does not exist at this endpoint. Create it before enabling velero/CNPG
|
||||
(R2: dashboard -> R2 -> Create bucket; see infra/BACKUP-R2.md)." ;;
|
||||
*InvalidAccessKeyId*|*SignatureDoesNotMatch*|*"(401)"*|*Unauthorized*)
|
||||
bad "${op}: the endpoint REJECTED these credentials. Wrong key, or a key belonging to a
|
||||
different R2 account / MinIO. Re-encrypt the credential Secrets — infra/BACKUP-R2.md." ;;
|
||||
*AccessDenied*|*"(403)"*|*Forbidden*)
|
||||
bad "${op}: authenticated but FORBIDDEN. The token is not scoped to this bucket. An R2 API token
|
||||
needs Object Read & Write on BOTH the velero and cnpg buckets." ;;
|
||||
*) bad "${op}: unclassified failure (rc=${RC}). Treated as a FAILURE, never as unknown-so-fine." ;;
|
||||
esac
|
||||
detail
|
||||
}
|
||||
|
||||
PAY=/tmp/s3check.payload
|
||||
GOT=/tmp/s3check.got
|
||||
head -c 4096 /dev/urandom | base64 > "${PAY}"
|
||||
[ -s "${PAY}" ] || { bad "could not generate a probe payload inside the pod"; verdict; }
|
||||
|
||||
TRAP_DONE=0
|
||||
for t in ${S3TARGETS}; do
|
||||
B="${t%%:*}"; P="${t#*:}"
|
||||
[ "${P}" = "${B}" ] && P=""
|
||||
if [ -n "${P}" ]; then K="${P}/.s3check/${S3KEY}"; else K=".s3check/${S3KEY}"; fi
|
||||
|
||||
# ---- (2) the bucket exists and is LISTABLE. This is the operation Velero's BSL sync and Barman's
|
||||
# WAL scan both perform; ListBuckets deliberately is NOT used, because neither consumer
|
||||
# needs it and a correctly bucket-scoped R2 token may not grant it.
|
||||
args=(s3api list-objects-v2 --bucket "${B}" --max-keys 1)
|
||||
[ -n "${P}" ] && args+=(--prefix "${P}/")
|
||||
awsx "${args[@]}"
|
||||
if [ "${RC}" -eq 0 ]; then
|
||||
ok "${B}: bucket EXISTS, AUTHENTICATES and is LISTABLE (prefix '${P:-<bucket root>}')"
|
||||
else
|
||||
diagnose "LIST ${B}"
|
||||
continue # a short assertion count is how the caller learns this target was abandoned
|
||||
fi
|
||||
|
||||
# ---- (3) PUT -> GET -> byte-compare -> DELETE. A real write, not a HEAD: HEAD on a bucket proves
|
||||
# nothing about whether an object can be STORED (wrong scope, checksum rejection and
|
||||
# read-only tokens all pass a HEAD and fail a backup).
|
||||
awsx s3api put-object --bucket "${B}" --key "${K}" --body "${PAY}"
|
||||
if [ "${RC}" -eq 0 ]; then ok "${B}: PUT ${K} accepted (genuine write path)"
|
||||
else diagnose "PUT ${B}/${K}"; continue; fi
|
||||
|
||||
rm -f "${GOT}"
|
||||
awsx s3api get-object --bucket "${B}" --key "${K}" "${GOT}"
|
||||
if [ "${RC}" -eq 0 ]; then ok "${B}: GET ${K} returned an object"
|
||||
else diagnose "GET ${B}/${K}"
|
||||
aws --endpoint-url "${S3EP}" s3api delete-object --bucket "${B}" --key "${K}" >/dev/null 2>&1 || true
|
||||
continue; fi
|
||||
|
||||
sp="$(wc -c < "${PAY}")"; sg="$(wc -c < "${GOT}" 2>/dev/null || echo 0)"
|
||||
hp="$(sha256sum < "${PAY}")"; hp="${hp%% *}"
|
||||
hg="$(sha256sum < "${GOT}")"; hg="${hg%% *}"
|
||||
if [ "${sp}" = "${sg}" ] && [ -n "${hp}" ] && [ "${hp}" = "${hg}" ]; then
|
||||
ok "${B}: round trip is BYTE-IDENTICAL (${sp} bytes, sha256 matches)"
|
||||
else
|
||||
bad "${B}: round trip CORRUPTED — wrote ${sp} bytes (sha256 ${hp}), read back ${sg} bytes (sha256 ${hg})"
|
||||
fi
|
||||
|
||||
awsx s3api delete-object --bucket "${B}" --key "${K}"
|
||||
if [ "${RC}" -eq 0 ]; then ok "${B}: DELETE ${K} accepted"
|
||||
else diagnose "DELETE ${B}/${K}"; fi
|
||||
|
||||
awsx s3api head-object --bucket "${B}" --key "${K}"
|
||||
if [ "${RC}" -ne 0 ]; then ok "${B}: scratch object is GONE — the probe left nothing behind"
|
||||
else bad "${B}: scratch object ${K} STILL EXISTS after DELETE. Remove it by hand; the probe leaked."; fi
|
||||
|
||||
# ---- (4) the R2 checksum trap, probed ON PURPOSE. Informational by design: the authored BSL sets
|
||||
# checksumAlgorithm: "", so a rejection here does NOT mean backups are broken — it means
|
||||
# that line is LOAD-BEARING. Reported either way so the operator learns which world this
|
||||
# endpoint lives in before, not after, enabling velero.
|
||||
if [ "${TRAP_DONE}" -eq 0 ]; then
|
||||
TRAP_DONE=1
|
||||
TK="${K}.checksum-trap-probe"
|
||||
OUT="$(AWS_REQUEST_CHECKSUM_CALCULATION=when_supported \
|
||||
aws --endpoint-url "${S3EP}" s3api put-object --bucket "${B}" --key "${TK}" --body "${PAY}" 2>&1)"; RC=$?
|
||||
if [ "${RC}" -eq 0 ]; then
|
||||
note "checksum probe: this endpoint ACCEPTS a checksum-bearing PUT — the R2 checksum trap is NOT
|
||||
present here (expected for MinIO). checksumAlgorithm: \"\" stays harmless."
|
||||
aws --endpoint-url "${S3EP}" s3api delete-object --bucket "${B}" --key "${TK}" >/dev/null 2>&1 \
|
||||
|| note "checksum probe: could NOT delete ${TK} — remove it by hand"
|
||||
else
|
||||
case "${OUT}" in
|
||||
*InvalidArgument*)
|
||||
note "checksum probe: R2 CHECKSUM TRAP CONFIRMED at this endpoint — a PUT carrying
|
||||
x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 is rejected with 400 InvalidArgument.
|
||||
checksumAlgorithm: \"\" in platform/velero/helmrelease.yaml is LOAD-BEARING: delete it, or bump
|
||||
velero-plugin-for-aws without re-checking it, and EVERY backup breaks." ;;
|
||||
*) note "checksum probe: inconclusive (rc=${RC}) — neither confirms nor rules out the trap." ;;
|
||||
esac
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
verdict
|
||||
PROBE
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- run one probe pod
|
||||
# <ns> <file|env> <secret> <space-separated bucket:prefix targets> <endpoint> <region>
|
||||
s3_probe() {
|
||||
local ns="$1" mode="$2" sec="$3" targets="$4" ep="$5" region="$6"
|
||||
# $$ alone is NOT unique here: inside the dev container this script is frequently PID 1, so two runs
|
||||
# would reuse the same pod name AND the same scratch object key. Mix in the epoch second.
|
||||
local pod="s3check-$$-${S3CHECK_RUN_ID}-${mode}"
|
||||
|
||||
# What this environment MUST produce: 1 credential assertion + 6 per bucket (list, put, get,
|
||||
# byte-compare, delete, delete-verified). Computed HERE, by the caller, from the target list — the
|
||||
# pod's own claim is never the only source of the expected count.
|
||||
local t n=0
|
||||
for t in ${targets}; do n=$((n+1)); done
|
||||
local min=$((1 + 6 * n))
|
||||
|
||||
local volsjson="" mountsjson="" creds=""
|
||||
case "${mode}" in
|
||||
file) # exactly how velero consumes it: the `cloud` key as an AWS INI credentials FILE
|
||||
volsjson='"volumes":[{"name":"s3creds","secret":{"secretName":"'"${sec}"'","items":[{"key":"cloud","path":"credentials"}],"defaultMode":256}}],'
|
||||
mountsjson='"volumeMounts":[{"name":"s3creds","mountPath":"/s3creds","readOnly":true}],'
|
||||
creds='{"name":"AWS_SHARED_CREDENTIALS_FILE","value":"/s3creds/credentials"},{"name":"CRED_MODE","value":"file"}' ;;
|
||||
env) # exactly how CNPG/barman consumes it: discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY keys
|
||||
creds='{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"name":"'"${sec}"'","key":"ACCESS_KEY_ID"}}},{"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"name":"'"${sec}"'","key":"ACCESS_SECRET_KEY"}}},{"name":"CRED_MODE","value":"env"}' ;;
|
||||
*) die "internal: unknown probe mode '${mode}'" ;;
|
||||
esac
|
||||
|
||||
local b64; b64="$(s3_probe_script | base64 | tr -d '\n')"
|
||||
[ -n "${b64}" ] || die "internal: probe script encoded to nothing"
|
||||
|
||||
# egress-class label: harmless here (the velero namespace is not an app namespace) but required by
|
||||
# the require-egress-class-label ClusterPolicy in any namespace labelled app-namespace=true, which
|
||||
# the CNPG credential namespaces are. Without it the probe pod would be REJECTED by admission and
|
||||
# the check would report "unknown" instead of a result.
|
||||
local overrides
|
||||
overrides='{"apiVersion":"v1","metadata":{"labels":{"app.kubernetes.io/name":"s3check","homelab.keithjackson.dev/egress-class":"internet"}},"spec":{"restartPolicy":"Never","automountServiceAccountToken":false,'"${volsjson}"'"containers":[{"name":"s3check","image":"'"${S3CHECK_IMAGE}"'","imagePullPolicy":"IfNotPresent","command":["/bin/bash","-c","printf %s \"$PROBE_B64\" | base64 -d > /tmp/probe.sh && exec bash /tmp/probe.sh"],'"${mountsjson}"'"env":['"${creds}"',{"name":"PROBE_B64","value":"'"${b64}"'"},{"name":"S3EP","value":"'"${ep}"'"},{"name":"S3REGION","value":"'"${region}"'"},{"name":"S3TARGETS","value":"'"${targets}"'"},{"name":"S3KEY","value":"'"${pod}"'.probe"},{"name":"AWS_REQUEST_CHECKSUM_CALCULATION","value":"when_required"},{"name":"AWS_RESPONSE_CHECKSUM_VALIDATION","value":"when_required"}],"resources":{"requests":{"cpu":"50m","memory":"64Mi"},"limits":{"memory":"256Mi"}}}]}}'
|
||||
|
||||
say "probe: ns=${ns} secret=${sec} (${mode}) targets='${targets}' → ${min} assertions required"
|
||||
kubectl -n "${ns}" delete pod "${pod}" --ignore-not-found >/dev/null 2>&1 || true
|
||||
S3CHECK_PODS+=("${ns}/${pod}")
|
||||
kubectl -n "${ns}" run "${pod}" --restart=Never --image="${S3CHECK_IMAGE}" \
|
||||
--overrides="${overrides}" >/dev/null 2>&1 \
|
||||
|| die "could NOT create the probe pod ${ns}/${pod} (admission policy? RBAC? quota?).
|
||||
That is a FAILURE, not an unknown: nothing about ${ep} has been proven."
|
||||
|
||||
local i=0 phase=""
|
||||
while [ "${i}" -lt 300 ]; do
|
||||
phase="$(kubectl -n "${ns}" get pod "${pod}" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
case "${phase}" in Succeeded|Failed) break ;; esac
|
||||
sleep 5; i=$((i+5))
|
||||
done
|
||||
case "${phase}" in
|
||||
Succeeded|Failed) ;;
|
||||
*) local reason
|
||||
reason="$(kubectl -n "${ns}" get pod "${pod}" \
|
||||
-o jsonpath='{.status.containerStatuses[0].state.waiting.reason}' 2>/dev/null || true)"
|
||||
die "probe pod ${ns}/${pod} did not finish in ${i}s (phase=${phase:-<none>}, waiting=${reason:-none}).
|
||||
NOT a pass. If waiting=ErrImagePull/ImagePullBackOff this cluster cannot pull ${S3CHECK_IMAGE}." ;;
|
||||
esac
|
||||
|
||||
local log; log="$(kubectl -n "${ns}" logs "${pod}" 2>/dev/null || true)"
|
||||
[ -n "${log}" ] || die "probe pod ${ns}/${pod} produced NO output (phase=${phase}). Failing closed —
|
||||
an empty log is the shape of a probe that never ran, not of a healthy backup target."
|
||||
printf '%s\n' "${log}" | while IFS= read -r l; do printf ' %s\n' "${l}"; done
|
||||
|
||||
# ---- the four independent accounting checks that make a silent no-op impossible ----
|
||||
local okcount failcount resline nres verd a f
|
||||
okcount="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-OK\[' || true)"
|
||||
failcount="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-FAIL' || true)"
|
||||
nres="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-RESULT ' || true)"
|
||||
[ "${nres}" = "1" ] || die "expected exactly ONE S3CHECK-RESULT line from ${ns}/${pod}, found ${nres}.
|
||||
A missing verdict line means the probe died mid-way; it is never a pass."
|
||||
resline="$(printf '%s\n' "${log}" | grep '^S3CHECK-RESULT ')"
|
||||
read -r _ verd a f <<<"${resline}"
|
||||
a="${a#assertions=}"; f="${f#failures=}"
|
||||
case "${a}" in ''|*[!0-9]*) die "malformed verdict from ${ns}/${pod}: '${resline}'" ;; esac
|
||||
case "${f}" in ''|*[!0-9]*) die "malformed verdict from ${ns}/${pod}: '${resline}'" ;; esac
|
||||
[ "${verd}" = "PASS" ] || die "S3 ROUND TRIP FAILED against ${ep} (${ns}/${pod}: ${f} failure(s)).
|
||||
Read the S3CHECK-FAIL lines above — each one names the cause and the fix. Do NOT enable velero
|
||||
or CNPG backups against this endpoint until they are all gone."
|
||||
[ "${f}" = "0" ] || die "${ns}/${pod} claimed PASS while reporting ${f} failures — refusing to believe it."
|
||||
[ "${failcount}" = "0" ] || die "${ns}/${pod} claimed PASS but the log carries ${failcount} S3CHECK-FAIL line(s)."
|
||||
[ "${a}" -ge "${min}" ] || die "${ns}/${pod} claimed PASS with only ${a} assertions; ${min} are required
|
||||
(1 credential + 6 per bucket). A short count means steps were SKIPPED — exactly the
|
||||
degrade-to-a-successful-no-op this check exists to prevent."
|
||||
[ "${okcount}" = "${a}" ] || die "assertion accounting mismatch for ${ns}/${pod}: ${okcount} S3CHECK-OK
|
||||
lines in the log vs a claimed ${a}. Refusing to pass on a self-report that does not add up."
|
||||
[ "${phase}" = "Succeeded" ] || die "${ns}/${pod} printed PASS but the pod ended ${phase}."
|
||||
info "PROVEN: ${a} assertions, 0 failures, pod ${phase}, no object left behind."
|
||||
}
|
||||
|
||||
# ============================================================================ steps
|
||||
step_preflight() {
|
||||
say "PREFLIGHT (read-only)"
|
||||
@@ -200,13 +561,14 @@ step_preflight() {
|
||||
info "owner id is distinct and non-default: OK"
|
||||
fi
|
||||
|
||||
case "${s3}" in
|
||||
""|TBD-*) warn "S3_BACKUP_ENDPOINT is unset/placeholder ('${s3}'). velero AND apps-wave-0 consume it.
|
||||
if s3_is_placeholder "${s3}"; then
|
||||
warn "S3_BACKUP_ENDPOINT is unset/placeholder ('${s3}'). velero AND apps-wave-0 consume it.
|
||||
velero will be SKIPPED by the resume step. apps-wave-0 will still apply (the token IS defined,
|
||||
so strict substitution succeeds) but anything writing to that endpoint fails at RUNTIME, not at
|
||||
apply time — so a green apps-wave-0 does NOT mean backups work. Operator decision required." ;;
|
||||
*) info "S3 backup endpoint set: OK" ;;
|
||||
esac
|
||||
apply time — so a green apps-wave-0 does NOT mean backups work. Operator decision required."
|
||||
else
|
||||
info "S3 backup endpoint set: OK"
|
||||
fi
|
||||
|
||||
# COLD-START CRD ORDERING. Flux server-side dry-runs EVERY object in a Kustomization before applying
|
||||
# any of them, so a single unknown `kind` aborts the whole tier. platform-config applies the CDI CR
|
||||
@@ -394,51 +756,139 @@ step_externaldns() {
|
||||
}
|
||||
|
||||
step_s3check() {
|
||||
say "S3 BACKUP ENDPOINT — reachability FROM INSIDE THE CLUSTER"
|
||||
# Velero runs in-cluster, so the only reachability that counts is the pods' own. Probing from the
|
||||
# workstation proves nothing either way: prod sits on the SOHO network (pfSense's WAN) while the
|
||||
# dev LAN's MinIO sits inside pfSense, so host and pod egress differ; a healthy reading there
|
||||
# network, and a healthy result there has already misled this project once.
|
||||
local ep live; live="$(cvar S3_BACKUP_ENDPOINT)"
|
||||
say "S3 BACKUP TARGET — PROVE A REAL ROUND TRIP FROM INSIDE THE CLUSTER"
|
||||
# Velero and CNPG run in-cluster, so pod egress is the only reachability that counts: prod sits on
|
||||
# the SOHO network (pfSense's WAN) while the dev LAN's MinIO sits INSIDE pfSense, so host and pod
|
||||
# egress differ — and a healthy host-side reading has already misled this project once.
|
||||
#
|
||||
# WHAT THIS PROVES (and it fails closed on every one of them):
|
||||
# 1. the committed credential loads and AUTHENTICATES, in the exact shape each consumer reads it
|
||||
# (velero: an AWS INI file from the `cloud` key; CNPG: discrete ACCESS_KEY_ID/ACCESS_SECRET_KEY);
|
||||
# 2. each required bucket EXISTS and is LISTABLE under this environment's own prefix;
|
||||
# 3. PUT -> GET -> byte-compare -> DELETE of a throwaway object under that prefix — the genuine
|
||||
# write path, not a HEAD (a HEAD passes with a read-only token and with R2 rejecting writes);
|
||||
# 4. whether the R2 CHECKSUM TRAP is present at this endpoint, named explicitly either way.
|
||||
#
|
||||
# WHAT IT STILL DOES NOT PROVE: that Velero's kopia uploader and CNPG's barman-cloud themselves work
|
||||
# against this target. Both are UNPROVEN against R2 in this repo. See infra/BACKUP-R2.md.
|
||||
local ep live region
|
||||
live="$(cvar S3_BACKUP_ENDPOINT)"
|
||||
info "live S3_BACKUP_ENDPOINT = ${live:-<unset>}"
|
||||
if [ -n "${S3_ENDPOINT_OVERRIDE}" ]; then
|
||||
# Probe a CANDIDATE endpoint before adopting it. The whole point is to find out whether a proposed
|
||||
# backup target is reachable from this cluster's pods BEFORE authoring it into cluster-vars —
|
||||
# otherwise the first evidence is silently-missing backups.
|
||||
# backup target works from this cluster's pods BEFORE authoring it into cluster-vars — otherwise
|
||||
# the first evidence is silently-missing backups.
|
||||
ep="${S3_ENDPOINT_OVERRIDE}"
|
||||
info "probing CANDIDATE endpoint from --s3-endpoint: ${ep}"
|
||||
else
|
||||
ep="${live}"
|
||||
fi
|
||||
if s3_is_placeholder "${ep}"; then
|
||||
die "no usable endpoint to probe (endpoint='${ep:-<unset>}').
|
||||
cluster-vars still carries the loud placeholder. Pass --s3-endpoint https://<account-id>.r2.cloudflarestorage.com
|
||||
to test a CANDIDATE before authoring it. See infra/BACKUP-R2.md."
|
||||
fi
|
||||
s3_safe_token S3_BACKUP_ENDPOINT "${ep}"
|
||||
case "${ep}" in
|
||||
""|TBD-*) die "no usable endpoint to probe (live='${live}'). Pass --s3-endpoint <url> to test a candidate." ;;
|
||||
http://*|https://*) ;;
|
||||
*) die "endpoint '${ep}' carries no http:// or https:// scheme — the AWS SDK needs one." ;;
|
||||
esac
|
||||
|
||||
local ns=velero pod=s3check-$$
|
||||
kubectl get ns "${ns}" >/dev/null 2>&1 || ns=default
|
||||
info "probing from a throwaway pod in namespace '${ns}'"
|
||||
# --restart=Never + explicit delete in a trap: a probe must never leave a pod behind on a cluster.
|
||||
trap 'kubectl -n "${ns}" delete pod "${pod}" --ignore-not-found --wait=false >/dev/null 2>&1 || true' RETURN
|
||||
if kubectl -n "${ns}" run "${pod}" --restart=Never --image=curlimages/curl:8.11.1 --command --quiet \
|
||||
-- curl -sS -o /dev/null -w '%{http_code}' --max-time 15 "${ep}/minio/health/live" >/dev/null 2>&1; then
|
||||
local i=0 phase=""
|
||||
while [ "$i" -lt 90 ]; do
|
||||
phase="$(kubectl -n "${ns}" get pod "${pod}" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
case "${phase}" in Succeeded|Failed) break ;; esac
|
||||
sleep 5; i=$((i+5))
|
||||
done
|
||||
local out; out="$(kubectl -n "${ns}" logs "${pod}" 2>/dev/null || true)"
|
||||
info "pod phase=${phase:-unknown} response=${out:-<none>}"
|
||||
case "${out}" in
|
||||
200|2??) info "REACHABLE: ${ep} answered from inside the cluster." ;;
|
||||
*) warn "NOT REACHABLE from inside the cluster (phase=${phase}, body='${out}').
|
||||
Velero would install cleanly and then fail every backup at RUNTIME — a tier that looks healthy
|
||||
while no backup exists. Fix routing, or give this environment its own endpoint, before
|
||||
resuming velero." ;;
|
||||
esac
|
||||
else
|
||||
warn "could not create the probe pod (admission policy? image pull?). Reachability UNKNOWN."
|
||||
# Region is a knob, not a constant: MinIO ignores it, R2 requires exactly `auto`, and an EMPTY value
|
||||
# in cluster-vars silently builds a BSL with no region.
|
||||
region="$(cvar S3_BACKUP_REGION)"
|
||||
case "${ep}" in
|
||||
*.r2.cloudflarestorage.com*)
|
||||
if [ "${region}" != "auto" ]; then
|
||||
warn "endpoint is Cloudflare R2 but cluster-vars S3_BACKUP_REGION='${region:-<unset>}'. R2 requires
|
||||
exactly 'auto'. Probing with 'auto' anyway; FIX clusters/base/cluster-vars.yaml before enabling
|
||||
velero, or the BackupStorageLocation is built with the wrong region."
|
||||
fi
|
||||
region=auto ;;
|
||||
*)
|
||||
if [ -z "${region}" ]; then
|
||||
region="us-east-1"
|
||||
warn "cluster-vars S3_BACKUP_REGION is EMPTY on this cluster. Probing with 'us-east-1' (MinIO
|
||||
ignores it) — but the BSL would be rendered with region: \"\". Set it explicitly."
|
||||
fi ;;
|
||||
esac
|
||||
s3_safe_token S3_BACKUP_REGION "${region}"
|
||||
|
||||
# Buckets come from the AUTHORED manifests + cluster-vars, never from a hardcoded name here: the
|
||||
# whole point is to probe what this environment will really write to (R-ENV-02, no dev/prod fork).
|
||||
local vbucket vprefix cpath cbucket cprefix rest
|
||||
vbucket="$(yq -r '.spec.values.configuration.backupStorageLocation[0].bucket' \
|
||||
platform/velero/helmrelease.yaml 2>/dev/null | tr -d '"')"
|
||||
[ -n "${vbucket}" ] && [ "${vbucket}" != "null" ] \
|
||||
|| die "could not read the Velero bucket name from platform/velero/helmrelease.yaml"
|
||||
vprefix="$(cvar VELERO_BACKUP_PREFIX)"
|
||||
|
||||
cpath="$(cvar CNPG_BACKUP_PATH)"
|
||||
if [ -z "${cpath}" ]; then
|
||||
warn "cluster-vars CNPG_BACKUP_PATH is EMPTY on this cluster. barmanObjectStore.destinationPath would
|
||||
substitute to the empty string and EVERY WAL archive would fail at runtime. Probing the historical
|
||||
default s3://cnpg so the round trip is still proven — but SET THIS VALUE."
|
||||
cpath="s3://cnpg"
|
||||
fi
|
||||
case "${cpath}" in s3://*) ;; *) die "CNPG_BACKUP_PATH='${cpath}' is not an s3:// URL." ;; esac
|
||||
rest="${cpath#s3://}"; cbucket="${rest%%/*}"
|
||||
if [ "${rest}" = "${cbucket}" ]; then cprefix=""; else cprefix="${rest#*/}"; fi
|
||||
cprefix="${cprefix%/}"
|
||||
[ -n "${cbucket}" ] || die "CNPG_BACKUP_PATH='${cpath}' names no bucket."
|
||||
|
||||
s3_safe_token velero-bucket "${vbucket}"; s3_safe_token VELERO_BACKUP_PREFIX "${vprefix}"
|
||||
s3_safe_token cnpg-bucket "${cbucket}"; s3_safe_token CNPG_BACKUP_PATH-prefix "${cprefix}"
|
||||
|
||||
say "what is about to be proven"
|
||||
info "endpoint ${ep}"
|
||||
info "region ${region}"
|
||||
info "velero bucket '${vbucket}' prefix '${vprefix:-<bucket root>}' (authored bucket + VELERO_BACKUP_PREFIX)"
|
||||
info "cnpg bucket '${cbucket}' prefix '${cprefix:-<bucket root>}' (from CNPG_BACKUP_PATH)"
|
||||
info "scratch key <prefix>/.s3check/<pod>.probe — written, read back, byte-compared, DELETED"
|
||||
|
||||
local ran=0 keys
|
||||
# (a) Velero's own credential, from its own namespace, mounted the way the aws plugin reads it.
|
||||
if kubectl get ns velero >/dev/null 2>&1 \
|
||||
&& kubectl -n velero get secret velero-cloud-credentials >/dev/null 2>&1; then
|
||||
# List KEY NAMES only. Never pipe or print a value (a careless redaction has leaked a password
|
||||
# into a transcript on this project already). go-template, not jsonpath: kubectl's jsonpath
|
||||
# cannot range over a map's keys (`range $k,$v :=` is rejected, U+002C) and returns EMPTY, which
|
||||
# silently looked like "the key is missing".
|
||||
keys="$(kubectl -n velero get secret velero-cloud-credentials \
|
||||
-o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}' 2>/dev/null || true)"
|
||||
printf '%s\n' "${keys}" | grep -qx 'cloud' \
|
||||
|| die "Secret velero/velero-cloud-credentials carries no 'cloud' key (keys: $(printf '%s' "${keys}" | tr '\n' ' ')).
|
||||
velero-plugin-for-aws reads exactly that key as an AWS INI file; without it every backup fails
|
||||
at RUNTIME while the HelmRelease reports Ready."
|
||||
s3_probe velero file velero-cloud-credentials "${vbucket}:${vprefix}" "${ep}" "${region}"
|
||||
ran=$((ran+1))
|
||||
else
|
||||
warn "no Secret velero/velero-cloud-credentials on this cluster — Velero's own credential could NOT
|
||||
be proven. That is expected before the velero tier has ever reconciled here; it is not a pass."
|
||||
fi
|
||||
|
||||
# (b) A CNPG/barman credential, discovered rather than hardcoded (R-ENV-02: no per-environment list
|
||||
# of app names in this script). Any one of them proves the key + the cnpg bucket + the discrete
|
||||
# ACCESS_KEY_ID/ACCESS_SECRET_KEY shape; they are all kept in sync by design.
|
||||
local pick cns csec
|
||||
pick="$(kubectl get secret -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\n"}{end}' 2>/dev/null \
|
||||
| grep -- '-db-backup-credentials$' | head -n 1 || true)"
|
||||
if [ -n "${pick}" ]; then
|
||||
cns="${pick%%/*}"; csec="${pick#*/}"
|
||||
s3_probe "${cns}" env "${csec}" "${cbucket}:${cprefix}" "${ep}" "${region}"
|
||||
ran=$((ran+1))
|
||||
else
|
||||
warn "no *-db-backup-credentials Secret found in any namespace — the CNPG/barman credential could
|
||||
NOT be proven."
|
||||
fi
|
||||
|
||||
[ "${ran}" -gt 0 ] || die "NO backup credential Secret exists on this cluster, so NOTHING could be
|
||||
proven about ${ep}. That is a FAILURE, not a skip: a green s3check must never mean 'I found
|
||||
nothing to check'."
|
||||
|
||||
say "S3 ROUND TRIP PROVEN from inside the cluster: ${ep} (${ran} credential path(s))"
|
||||
info "STILL UNPROVEN by this step: Velero's kopia uploader and CNPG barman-cloud against this target,"
|
||||
info "and a RESTORE. A backup that has never been restored is not a backup — see infra/BACKUP-R2.md."
|
||||
}
|
||||
|
||||
step_resume() {
|
||||
@@ -453,13 +903,20 @@ step_resume() {
|
||||
wait_ready "$n" 900 || { warn "stopping at $n — a partial, understood bring-up beats a forced green."; return 1; }
|
||||
done
|
||||
|
||||
case "${s3}" in
|
||||
""|TBD-*) warn "velero left SUSPENDED: S3_BACKUP_ENDPOINT='${s3}' is not a usable endpoint.
|
||||
# The placeholder test MUST be substring-based. It was `TBD-*` (a PREFIX match) while the placeholder
|
||||
# was `TBD-PROD-…`; the R2 move changed it to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`,
|
||||
# which does not START with TBD- — so this case arm silently stopped matching and `resume` would have
|
||||
# RESUMED velero onto a placeholder hostname on prod. That is the precise failure this guard exists
|
||||
# to prevent, reintroduced by a value change in another file. s3_is_placeholder matches anywhere.
|
||||
if s3_is_placeholder "${s3}"; then
|
||||
warn "velero left SUSPENDED: S3_BACKUP_ENDPOINT='${s3}' is not a usable endpoint.
|
||||
Resuming it would create a BackupStorageLocation that can never reach a bucket, i.e. a tier
|
||||
that reports Ready-ish while backups silently do not exist. Set the real prod endpoint first." ;;
|
||||
*) info "resuming velero"; flux resume kustomization velero --timeout=1s >/dev/null 2>&1 || true
|
||||
wait_ready velero 900 || warn "velero did not converge" ;;
|
||||
esac
|
||||
that reports Ready-ish while backups silently do not exist. Set the real endpoint first, prove
|
||||
it with \`s3check\`, and only then resume (infra/BACKUP-R2.md)."
|
||||
else
|
||||
info "resuming velero"; flux resume kustomization velero --timeout=1s >/dev/null 2>&1 || true
|
||||
wait_ready velero 900 || warn "velero did not converge"
|
||||
fi
|
||||
}
|
||||
|
||||
step_verify() {
|
||||
|
||||
Reference in new issue
Block a user