feat(backup): s3check proves a real round trip — and revives a guard I had killed
*** THE BUG, AND IT WAS MINE. *** The placeholder guards were `case "$s3" in
""|TBD-*)` — a PREFIX glob. Two commits earlier (32d6766) I changed the
placeholder from `TBD-PROD-S3-BACKUP-ENDPOINT` to
`https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`, which does NOT start with
`TBD-`. Verified: fnmatch('https://TBD-R2-...', 'TBD-*') is False. All three
guards silently stopped matching, so **step_resume would have resumed velero on
prod straight onto a nonexistent hostname** — velero installs, reports healthy,
and no backup exists. That is precisely the failure the guard was written to
prevent, disarmed by a value change in a DIFFERENT file with nothing tying the
two together. Replaced with substring-matching s3_is_placeholder in preflight,
s3check and resume.
s3check rewritten from a reachability ping into a real proof. Per credential, from
a throwaway in-cluster pod (pod egress is the only egress that matters — the
workstation sits on a different network side and a host-side reading has already
misled this project once):
1. the credential LOADS AND AUTHENTICATES in the exact shape its consumer reads
— Velero's INI mounted from the `cloud` key, CNPG's discrete env keys;
2. the bucket EXISTS and is LISTABLE under this environment's own prefix, via
ListObjectsV2 — the op Velero's BSL sync and Barman's WAL scan actually use.
ListBuckets deliberately avoided: a bucket-scoped R2 token may not grant it;
3. PUT -> GET -> sha256+length compare -> DELETE -> confirm-gone. A HEAD would
pass with a read-only token and with every write being rejected;
4. the R2 checksum trap probed ON PURPOSE by repeating the PUT with checksums
enabled, and named either way.
Buckets and prefixes are DERIVED (velero's from the authored HelmRelease via yq,
cnpg's parsed from CNPG_BACKUP_PATH), not restated.
Fails closed, with no "unknown, carrying on" arm left: no pod, unfinished pod,
empty log, missing/duplicate/malformed verdict, any failed assertion, a claimed
PASS with failures>0, an assertion count below the caller-computed minimum, an
OK-line count disagreeing with the claimed count, or a PASS from a pod that ended
Failed. Failures are classified by cause — no egress, NoSuchBucket,
InvalidAccessKeyId, AccessDenied (token not scoped to BOTH buckets),
InvalidArgument -> the checksum trap.
`amazon/aws-cli` pinned rather than `minio/mc`: mc lacks sed/awk (which already
broke one probe) AND cannot send checksum headers, so it cannot test for the trap.
TIER_ORDER now includes platform-config-cdi, between platform-config and the leaf
tiers. Without it `resume` returned success without ever waiting on the tier that
guarantees the CDI StorageProfiles — and apps-vms depends on them: unifi
provisions on longhorn-vm, where an unapplied profile means CDI infers RWO instead
of the authored RWX, permanently non-live-migratable.
CREDENTIALS: one key, four committed Secrets, two shapes — and NO manifest change
is needed, only encrypted values. platform/backup/set-r2-credentials.sh writes all
four with `sops set`, which encrypts IN PLACE, so no decrypted Secret ever exists
in the working tree to be committed by accident — strictly better than staging a
plaintext template. Prompts with `read -rs`, rejects anything outside the S3
credential charset (a paste that swallowed a newline authenticates nowhere), never
prints a value, and verifies by decrypting back and comparing fingerprints.
`--check` is read-only. The MinIO key is RETAINED — it is what reads the
pre-migration backups.
THE ACCOUNT ID IS ALREADY IN THE REPO: infra/tofu/backend.r2.hcl commits
https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com for the tofu
state bucket. If the backup buckets go in the same account, only the two token
values are needed. The placeholder is KEPT anyway — fail-closed beats convenience,
and the guard above just demonstrated why.
Corroborating the checksum fix independently: infra/tofu/backend.tf already carries
`region = "auto"` and `skip_s3_checksum = true # R2 rejects AWS trailing
(streaming) checksums`. This repo had learned both R2 facts for tofu state already.
VERSION FINDINGS, documented not measured against R2: plugin v1.14.2 documents
checksumAlgorithm defaulting to CRC32, so the authored "" is load-bearing;
Velero's kopia backend has NO checksum knob, so BSL metadata and PVC-data
compatibility are SEPARATE questions; and in the live CNPG 16.4 image,
boto3/botocore 1.35.60 is below the 1.36 line that turned on default request
checksums — so CNPG should be clear today, but barman-cloud exposes no escape
hatch, making a CNPG image bump an R2 re-verification event.
VERIFIED BY ME, independently of the agent: the dead-guard glob (fnmatch False);
the placeholder now refused on prod before any pod is created; a full round trip
against dev's MinIO — 7/7 assertions on BOTH credential paths, byte-identical,
pod Succeeded, nothing left behind, checksum trap correctly reported absent; all
four Secrets and the committed account id present. Full CI gate PASS.
STILL UNPROVEN AND SAID SO IN THE DOC: Velero-to-R2 (metadata or kopia data),
CNPG-barman-to-R2, and any restore from R2. R-STATE-03 is PROVEN MinIO-ONLY and
its ledger row predates this move. infra/BACKUP-R2.md §5 is the re-proof sequence
and it ends in a RESTORE, not an upload.
Refs R-STATE-03, R-IDP-05, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
59ffbaf11a
commit
8f9b2e4270
7 files changed
+1088
-63
No files matched your search
+505
-48
@@ -26,6 +26,11 @@
|
||||
#
|
||||
# --confirm-destructive is required by `unwedge` only. Everything else is additive or reversible.
|
||||
#
|
||||
# --s3-endpoint <url> makes `s3check` probe a CANDIDATE backup endpoint instead of the live
|
||||
# cluster-vars value, so a proposed target can be PROVEN before it is authored into cluster-vars.
|
||||
# `s3check` writes and then deletes ONE throwaway object under `<prefix>/.s3check/`; it changes
|
||||
# nothing else and applies nothing. See infra/BACKUP-R2.md.
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
# ---------------------------------------------------------------------------- args
|
||||
@@ -85,10 +90,18 @@ say "target confirmed: ${live_server}"
|
||||
|
||||
# Dependency order. Flux enforces the edges itself via dependsOn; resuming in this order simply means
|
||||
# each wait is meaningful instead of blocking on a dependency that is still suspended.
|
||||
# flux-system -> cilium -> platform -> platform-config -> {leaf tiers, apps-wave-0} -> wave-1 -> wave-2
|
||||
TIER_ORDER=(platform platform-config oauth2-proxy headlamp homepage crafty apps-wave-0 apps-wave-1 apps-wave-2 apps-vms)
|
||||
# velero is NOT in that list on purpose — see step `resume`: it consumes S3_BACKUP_ENDPOINT, which on
|
||||
# prod is an intentionally-invalid TBD placeholder pending an operator decision.
|
||||
# flux-system -> cilium -> platform -> platform-config -> platform-config-cdi
|
||||
# -> {leaf tiers, apps-wave-0} -> wave-1 -> wave-2 -> apps-vms
|
||||
#
|
||||
# `platform-config-cdi` MUST stay in this list and MUST stay before apps-vms. It carries the CDI
|
||||
# StorageProfiles, which were split out of platform-config because StorageProfile's CRD does not exist
|
||||
# until cdi-operator has reconciled the CDI CR (clusters/base/platform-config-cdi.yaml). apps-vms
|
||||
# dependsOn it: apps/vms/unifi provisions on `longhorn-vm`, and without the profile applied CDI infers
|
||||
# ReadWriteOnce instead of the authored ReadWriteMany — permanently non-live-migratable for that disk.
|
||||
# Omitting it here would make `resume` return success without ever waiting on the tier that guarantees it.
|
||||
TIER_ORDER=(platform platform-config platform-config-cdi oauth2-proxy headlamp homepage crafty apps-wave-0 apps-wave-1 apps-wave-2 apps-vms)
|
||||
# velero is NOT in that list on purpose — see step `resume`: it consumes S3_BACKUP_ENDPOINT, which is
|
||||
# an intentionally-invalid placeholder until the operator supplies the real R2 endpoint.
|
||||
|
||||
cvar() { kubectl -n flux-system get cm cluster-vars -o jsonpath="{.data.$1}" 2>/dev/null || true; }
|
||||
|
||||
@@ -150,6 +163,354 @@ dns_diff() { # <before> <after> — returns non-zero if anything changed
|
||||
return 1
|
||||
}
|
||||
|
||||
# ======================================================================= s3check plumbing
|
||||
# WHY THIS IS THIS BIG. The previous version of `s3check` curl'd /minio/health/live and reported
|
||||
# REACHABLE. That probe could not pass against Cloudflare R2 (no such path), and reachability is not
|
||||
# the property anyone cares about: a reachable endpoint with a wrong key, a missing bucket, or R2's
|
||||
# checksum rejection still yields a velero tier that reports Ready while NO backup exists. What must
|
||||
# be proven is the write path Velero and CNPG-barman actually use, with the credentials they actually
|
||||
# hold, from the network position they actually occupy.
|
||||
|
||||
# Pinned. The AWS CLI is a deliberate choice over `minio/mc`:
|
||||
# - it prints the S3 ERROR CODE (NoSuchBucket / AccessDenied / InvalidAccessKeyId / InvalidArgument),
|
||||
# so a failure can be NAMED and turned into a fix instead of guessed at from an HTTP status;
|
||||
# - it exposes the FLEXIBLE-CHECKSUM behaviour that is the R2 trap
|
||||
# (AWS_REQUEST_CHECKSUM_CALCULATION), so the trap can be reproduced on purpose here instead of
|
||||
# discovered by a silently-failing backup at 03:00;
|
||||
# - its image carries bash + coreutils. `minio/mc` ships no sed/awk/diff — that minimalism has
|
||||
# already broken one probe in this repo — and it cannot be made to send the checksum headers, so
|
||||
# it cannot test for the trap at all.
|
||||
S3CHECK_IMAGE="amazon/aws-cli:2.36.40"
|
||||
S3CHECK_RUN_ID="$(date +%s)"
|
||||
|
||||
# Throwaway probe pods, tracked so they are removed on EVERY exit path (including `die`). A probe
|
||||
# must never leave a pod behind on a live cluster.
|
||||
S3CHECK_PODS=()
|
||||
s3_cleanup_pods() {
|
||||
local p
|
||||
for p in ${S3CHECK_PODS[@]+"${S3CHECK_PODS[@]}"}; do
|
||||
kubectl -n "${p%%/*}" delete pod "${p#*/}" --ignore-not-found --wait=false >/dev/null 2>&1 || true
|
||||
done
|
||||
S3CHECK_PODS=()
|
||||
}
|
||||
trap s3_cleanup_pods EXIT
|
||||
|
||||
# TRUE when the endpoint is not a usable target. `TBD-*` alone was WRONG once the placeholder became
|
||||
# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`: that does not START with TBD-, so the old
|
||||
# prefix pattern matched nothing and `resume` would have resumed velero straight onto a placeholder
|
||||
# hostname — the exact silent-no-backups failure the guard exists to prevent. Match anywhere.
|
||||
s3_is_placeholder() { # <endpoint>
|
||||
case "$1" in
|
||||
""|*TBD*|*tbd-*|*REPLACE-ME*|*replace-me*|*CHANGE-ME*|*change-me*) return 0 ;;
|
||||
esac
|
||||
return 1
|
||||
}
|
||||
|
||||
# Refuse any value that is not a plain URL/name fragment. This is BOTH a safety guard — these values
|
||||
# are interpolated into the probe pod's JSON overrides AND into a shell script that runs inside it, so
|
||||
# a quote, a backslash or a newline could rewrite either — and a correctness guard: a cluster-vars
|
||||
# value with a trailing space or a stray quote is a defect, not something to paper over.
|
||||
s3_safe_token() { # <label> <value>
|
||||
case "$2" in
|
||||
*[!A-Za-z0-9:/._-]*) die "refusing to probe: $1 contains a character outside [A-Za-z0-9:/._-].
|
||||
That is either a corrupt cluster-vars value or an injection attempt. Value (quoted): $(printf '%q' "$2")" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- the in-pod probe
|
||||
# Emitted to the pod base64-encoded, so nothing in it has to survive JSON or shell quoting.
|
||||
# CONTRACT WITH THE CALLER (this is what makes a silent no-op impossible):
|
||||
# - every proven property prints exactly one `S3CHECK-OK[n] ...` line and increments n;
|
||||
# - every failure prints `S3CHECK-FAIL ...` and is counted;
|
||||
# - the LAST line is `S3CHECK-RESULT PASS|FAIL assertions=<n> failures=<n>`.
|
||||
# The caller independently knows how many assertions this environment must produce and re-counts the
|
||||
# OK lines itself, so neither a truncated log, a skipped bucket, an empty log, nor a lying result line
|
||||
# can be read as success. (A CRD guard in this repo once counted a blank line as "1 CRD" and sailed
|
||||
# past; that class of bug is designed out here rather than commented about.)
|
||||
s3_probe_script() {
|
||||
cat <<'PROBE'
|
||||
set -u
|
||||
umask 077
|
||||
A=0; F=0
|
||||
ok() { A=$((A+1)); printf 'S3CHECK-OK[%d] %s\n' "$A" "$*"; }
|
||||
bad() { F=$((F+1)); printf 'S3CHECK-FAIL %s\n' "$*"; }
|
||||
note() { printf 'S3CHECK-NOTE %s\n' "$*"; }
|
||||
verdict() {
|
||||
if [ "${F}" -eq 0 ] && [ "${A}" -gt 0 ]; then
|
||||
printf 'S3CHECK-RESULT PASS assertions=%d failures=0\n' "${A}"; exit 0
|
||||
fi
|
||||
printf 'S3CHECK-RESULT FAIL assertions=%d failures=%d\n' "${A}" "${F}"; exit 1
|
||||
}
|
||||
|
||||
for v in S3EP S3REGION S3TARGETS S3KEY CRED_MODE; do
|
||||
eval "x=\${${v}-}"
|
||||
[ -n "${x}" ] || { bad "required env ${v} is empty — the probe was launched wrong"; verdict; }
|
||||
done
|
||||
|
||||
export AWS_EC2_METADATA_DISABLED=true
|
||||
export AWS_PAGER=""
|
||||
export AWS_DEFAULT_REGION="${S3REGION}"
|
||||
export AWS_REGION="${S3REGION}"
|
||||
export AWS_MAX_ATTEMPTS=2
|
||||
export AWS_RETRY_MODE=standard
|
||||
export AWS_CONFIG_FILE=/tmp/s3check-aws-config
|
||||
# Path-style addressing, because both targets need it: MinIO on a bare IP has no virtual-host names,
|
||||
# and R2's S3 API is documented path-style. Checksums OFF, mirroring `checksumAlgorithm: ""` on the
|
||||
# Velero BSL — this run must exercise the same wire shape the real backup will use. Written to the
|
||||
# CONFIG FILE as well as the environment so the setting holds even if one channel is ignored by a
|
||||
# future CLI version.
|
||||
{
|
||||
echo '[default]'
|
||||
echo 'request_checksum_calculation = when_required'
|
||||
echo 'response_checksum_validation = when_required'
|
||||
echo 's3 ='
|
||||
echo ' addressing_style = path'
|
||||
} > "${AWS_CONFIG_FILE}"
|
||||
|
||||
# ---- (1) the credential loads at all -------------------------------------------------
|
||||
# NEVER print a credential value, in either `key: value` or `key=value` shape. Only booleans,
|
||||
# byte counts and lengths leave this pod.
|
||||
case "${CRED_MODE}" in
|
||||
file)
|
||||
cf="${AWS_SHARED_CREDENTIALS_FILE-}"
|
||||
if [ -z "${cf}" ] || [ ! -s "${cf}" ]; then
|
||||
bad "credential file '${cf:-<unset>}' is missing or empty — the Secret key did not mount"
|
||||
elif ! grep -Eq '^[[:space:]]*\[default\][[:space:]]*$' "${cf}"; then
|
||||
bad "credential file has no [default] profile. velero-plugin-for-aws reads an AWS INI file from
|
||||
the 'cloud' key of velero-cloud-credentials; without [default] every backup fails at RUNTIME"
|
||||
elif ! grep -Eq '^[[:space:]]*aws_access_key_id[[:space:]]*=[[:space:]]*[^[:space:]]' "${cf}" \
|
||||
|| ! grep -Eq '^[[:space:]]*aws_secret_access_key[[:space:]]*=[[:space:]]*[^[:space:]]' "${cf}"; then
|
||||
bad "credential file [default] lacks a non-empty aws_access_key_id / aws_secret_access_key"
|
||||
else
|
||||
ok "credential LOADS: AWS INI with [default] + both keys ($(wc -c < "${cf}") bytes; values never printed)"
|
||||
fi
|
||||
;;
|
||||
env)
|
||||
if [ -z "${AWS_ACCESS_KEY_ID-}" ] || [ -z "${AWS_SECRET_ACCESS_KEY-}" ]; then
|
||||
bad "AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY did not arrive from the Secret (missing ACCESS_KEY_ID / ACCESS_SECRET_KEY key?)"
|
||||
else
|
||||
ok "credential LOADS: access key id present (${#AWS_ACCESS_KEY_ID} chars; values never printed)"
|
||||
fi
|
||||
;;
|
||||
*) bad "unknown CRED_MODE '${CRED_MODE}'" ;;
|
||||
esac
|
||||
[ "${F}" -eq 0 ] || verdict # no point talking to S3 with no credential
|
||||
|
||||
OUT=""; RC=0
|
||||
awsx() { OUT="$(aws --endpoint-url "${S3EP}" "$@" 2>&1)"; RC=$?; return 0; }
|
||||
|
||||
detail() { printf '%s\n' "${OUT}" | head -n 12 | while IFS= read -r l; do printf 'S3CHECK-DETAIL %s\n' "${l}"; done; }
|
||||
|
||||
# Every failure is NAMED and given the fix. An unclassified failure is still a failure.
|
||||
diagnose() { # <operation label>
|
||||
local op="$1"
|
||||
case "${OUT}" in
|
||||
*InvalidArgument*|*checksum*|*Checksum*)
|
||||
bad "${op}: 400 InvalidArgument — THE R2 CHECKSUM TRAP. velero-plugin-for-aws v1.14+ bundles an
|
||||
AWS SDK that sends x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT, and
|
||||
Cloudflare R2 REJECTS those headers with HTTP 400 InvalidArgument, so every Velero backup and
|
||||
every kopia blob upload fails. FIX: checksumAlgorithm: \"\" must be present in the
|
||||
backupStorageLocation config in platform/velero/helmrelease.yaml. This probe already disables
|
||||
checksums, so seeing this here means the client sent one anyway — treat it as a hard blocker." ;;
|
||||
*"Could not connect"*|*EndpointConnectionError*|*ConnectTimeoutError*|*"Failed to establish"*|\
|
||||
*"Name or service not known"*|*"nodename nor servname"*|*"Temporary failure in name resolution"*|\
|
||||
*"Connection reset"*|*"Read timeout"*)
|
||||
bad "${op}: this POD cannot reach ${S3EP} (connect / DNS / timeout). Velero and CNPG would
|
||||
install perfectly clean and then fail every backup at RUNTIME — a tier that looks healthy
|
||||
while no backup exists. This is the reading that matters: the workstation's own reachability
|
||||
is irrelevant (prod sits on the SOHO net, pfSense's WAN; the dev LAN's MinIO sits INSIDE
|
||||
pfSense) and a host-side probe has already misled this project once. Fix pod egress/routing,
|
||||
or give this environment its own endpoint." ;;
|
||||
*NoSuchBucket*|*"(404)"*|*"Not Found"*)
|
||||
bad "${op}: the bucket does not exist at this endpoint. Create it before enabling velero/CNPG
|
||||
(R2: dashboard -> R2 -> Create bucket; see infra/BACKUP-R2.md)." ;;
|
||||
*InvalidAccessKeyId*|*SignatureDoesNotMatch*|*"(401)"*|*Unauthorized*)
|
||||
bad "${op}: the endpoint REJECTED these credentials. Wrong key, or a key belonging to a
|
||||
different R2 account / MinIO. Re-encrypt the credential Secrets — infra/BACKUP-R2.md." ;;
|
||||
*AccessDenied*|*"(403)"*|*Forbidden*)
|
||||
bad "${op}: authenticated but FORBIDDEN. The token is not scoped to this bucket. An R2 API token
|
||||
needs Object Read & Write on BOTH the velero and cnpg buckets." ;;
|
||||
*) bad "${op}: unclassified failure (rc=${RC}). Treated as a FAILURE, never as unknown-so-fine." ;;
|
||||
esac
|
||||
detail
|
||||
}
|
||||
|
||||
PAY=/tmp/s3check.payload
|
||||
GOT=/tmp/s3check.got
|
||||
head -c 4096 /dev/urandom | base64 > "${PAY}"
|
||||
[ -s "${PAY}" ] || { bad "could not generate a probe payload inside the pod"; verdict; }
|
||||
|
||||
TRAP_DONE=0
|
||||
for t in ${S3TARGETS}; do
|
||||
B="${t%%:*}"; P="${t#*:}"
|
||||
[ "${P}" = "${B}" ] && P=""
|
||||
if [ -n "${P}" ]; then K="${P}/.s3check/${S3KEY}"; else K=".s3check/${S3KEY}"; fi
|
||||
|
||||
# ---- (2) the bucket exists and is LISTABLE. This is the operation Velero's BSL sync and Barman's
|
||||
# WAL scan both perform; ListBuckets deliberately is NOT used, because neither consumer
|
||||
# needs it and a correctly bucket-scoped R2 token may not grant it.
|
||||
args=(s3api list-objects-v2 --bucket "${B}" --max-keys 1)
|
||||
[ -n "${P}" ] && args+=(--prefix "${P}/")
|
||||
awsx "${args[@]}"
|
||||
if [ "${RC}" -eq 0 ]; then
|
||||
ok "${B}: bucket EXISTS, AUTHENTICATES and is LISTABLE (prefix '${P:-<bucket root>}')"
|
||||
else
|
||||
diagnose "LIST ${B}"
|
||||
continue # a short assertion count is how the caller learns this target was abandoned
|
||||
fi
|
||||
|
||||
# ---- (3) PUT -> GET -> byte-compare -> DELETE. A real write, not a HEAD: HEAD on a bucket proves
|
||||
# nothing about whether an object can be STORED (wrong scope, checksum rejection and
|
||||
# read-only tokens all pass a HEAD and fail a backup).
|
||||
awsx s3api put-object --bucket "${B}" --key "${K}" --body "${PAY}"
|
||||
if [ "${RC}" -eq 0 ]; then ok "${B}: PUT ${K} accepted (genuine write path)"
|
||||
else diagnose "PUT ${B}/${K}"; continue; fi
|
||||
|
||||
rm -f "${GOT}"
|
||||
awsx s3api get-object --bucket "${B}" --key "${K}" "${GOT}"
|
||||
if [ "${RC}" -eq 0 ]; then ok "${B}: GET ${K} returned an object"
|
||||
else diagnose "GET ${B}/${K}"
|
||||
aws --endpoint-url "${S3EP}" s3api delete-object --bucket "${B}" --key "${K}" >/dev/null 2>&1 || true
|
||||
continue; fi
|
||||
|
||||
sp="$(wc -c < "${PAY}")"; sg="$(wc -c < "${GOT}" 2>/dev/null || echo 0)"
|
||||
hp="$(sha256sum < "${PAY}")"; hp="${hp%% *}"
|
||||
hg="$(sha256sum < "${GOT}")"; hg="${hg%% *}"
|
||||
if [ "${sp}" = "${sg}" ] && [ -n "${hp}" ] && [ "${hp}" = "${hg}" ]; then
|
||||
ok "${B}: round trip is BYTE-IDENTICAL (${sp} bytes, sha256 matches)"
|
||||
else
|
||||
bad "${B}: round trip CORRUPTED — wrote ${sp} bytes (sha256 ${hp}), read back ${sg} bytes (sha256 ${hg})"
|
||||
fi
|
||||
|
||||
awsx s3api delete-object --bucket "${B}" --key "${K}"
|
||||
if [ "${RC}" -eq 0 ]; then ok "${B}: DELETE ${K} accepted"
|
||||
else diagnose "DELETE ${B}/${K}"; fi
|
||||
|
||||
awsx s3api head-object --bucket "${B}" --key "${K}"
|
||||
if [ "${RC}" -ne 0 ]; then ok "${B}: scratch object is GONE — the probe left nothing behind"
|
||||
else bad "${B}: scratch object ${K} STILL EXISTS after DELETE. Remove it by hand; the probe leaked."; fi
|
||||
|
||||
# ---- (4) the R2 checksum trap, probed ON PURPOSE. Informational by design: the authored BSL sets
|
||||
# checksumAlgorithm: "", so a rejection here does NOT mean backups are broken — it means
|
||||
# that line is LOAD-BEARING. Reported either way so the operator learns which world this
|
||||
# endpoint lives in before, not after, enabling velero.
|
||||
if [ "${TRAP_DONE}" -eq 0 ]; then
|
||||
TRAP_DONE=1
|
||||
TK="${K}.checksum-trap-probe"
|
||||
OUT="$(AWS_REQUEST_CHECKSUM_CALCULATION=when_supported \
|
||||
aws --endpoint-url "${S3EP}" s3api put-object --bucket "${B}" --key "${TK}" --body "${PAY}" 2>&1)"; RC=$?
|
||||
if [ "${RC}" -eq 0 ]; then
|
||||
note "checksum probe: this endpoint ACCEPTS a checksum-bearing PUT — the R2 checksum trap is NOT
|
||||
present here (expected for MinIO). checksumAlgorithm: \"\" stays harmless."
|
||||
aws --endpoint-url "${S3EP}" s3api delete-object --bucket "${B}" --key "${TK}" >/dev/null 2>&1 \
|
||||
|| note "checksum probe: could NOT delete ${TK} — remove it by hand"
|
||||
else
|
||||
case "${OUT}" in
|
||||
*InvalidArgument*)
|
||||
note "checksum probe: R2 CHECKSUM TRAP CONFIRMED at this endpoint — a PUT carrying
|
||||
x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 is rejected with 400 InvalidArgument.
|
||||
checksumAlgorithm: \"\" in platform/velero/helmrelease.yaml is LOAD-BEARING: delete it, or bump
|
||||
velero-plugin-for-aws without re-checking it, and EVERY backup breaks." ;;
|
||||
*) note "checksum probe: inconclusive (rc=${RC}) — neither confirms nor rules out the trap." ;;
|
||||
esac
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
verdict
|
||||
PROBE
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- run one probe pod
|
||||
# <ns> <file|env> <secret> <space-separated bucket:prefix targets> <endpoint> <region>
|
||||
s3_probe() {
|
||||
local ns="$1" mode="$2" sec="$3" targets="$4" ep="$5" region="$6"
|
||||
# $$ alone is NOT unique here: inside the dev container this script is frequently PID 1, so two runs
|
||||
# would reuse the same pod name AND the same scratch object key. Mix in the epoch second.
|
||||
local pod="s3check-$$-${S3CHECK_RUN_ID}-${mode}"
|
||||
|
||||
# What this environment MUST produce: 1 credential assertion + 6 per bucket (list, put, get,
|
||||
# byte-compare, delete, delete-verified). Computed HERE, by the caller, from the target list — the
|
||||
# pod's own claim is never the only source of the expected count.
|
||||
local t n=0
|
||||
for t in ${targets}; do n=$((n+1)); done
|
||||
local min=$((1 + 6 * n))
|
||||
|
||||
local volsjson="" mountsjson="" creds=""
|
||||
case "${mode}" in
|
||||
file) # exactly how velero consumes it: the `cloud` key as an AWS INI credentials FILE
|
||||
volsjson='"volumes":[{"name":"s3creds","secret":{"secretName":"'"${sec}"'","items":[{"key":"cloud","path":"credentials"}],"defaultMode":256}}],'
|
||||
mountsjson='"volumeMounts":[{"name":"s3creds","mountPath":"/s3creds","readOnly":true}],'
|
||||
creds='{"name":"AWS_SHARED_CREDENTIALS_FILE","value":"/s3creds/credentials"},{"name":"CRED_MODE","value":"file"}' ;;
|
||||
env) # exactly how CNPG/barman consumes it: discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY keys
|
||||
creds='{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"name":"'"${sec}"'","key":"ACCESS_KEY_ID"}}},{"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"name":"'"${sec}"'","key":"ACCESS_SECRET_KEY"}}},{"name":"CRED_MODE","value":"env"}' ;;
|
||||
*) die "internal: unknown probe mode '${mode}'" ;;
|
||||
esac
|
||||
|
||||
local b64; b64="$(s3_probe_script | base64 | tr -d '\n')"
|
||||
[ -n "${b64}" ] || die "internal: probe script encoded to nothing"
|
||||
|
||||
# egress-class label: harmless here (the velero namespace is not an app namespace) but required by
|
||||
# the require-egress-class-label ClusterPolicy in any namespace labelled app-namespace=true, which
|
||||
# the CNPG credential namespaces are. Without it the probe pod would be REJECTED by admission and
|
||||
# the check would report "unknown" instead of a result.
|
||||
local overrides
|
||||
overrides='{"apiVersion":"v1","metadata":{"labels":{"app.kubernetes.io/name":"s3check","homelab.keithjackson.dev/egress-class":"internet"}},"spec":{"restartPolicy":"Never","automountServiceAccountToken":false,'"${volsjson}"'"containers":[{"name":"s3check","image":"'"${S3CHECK_IMAGE}"'","imagePullPolicy":"IfNotPresent","command":["/bin/bash","-c","printf %s \"$PROBE_B64\" | base64 -d > /tmp/probe.sh && exec bash /tmp/probe.sh"],'"${mountsjson}"'"env":['"${creds}"',{"name":"PROBE_B64","value":"'"${b64}"'"},{"name":"S3EP","value":"'"${ep}"'"},{"name":"S3REGION","value":"'"${region}"'"},{"name":"S3TARGETS","value":"'"${targets}"'"},{"name":"S3KEY","value":"'"${pod}"'.probe"},{"name":"AWS_REQUEST_CHECKSUM_CALCULATION","value":"when_required"},{"name":"AWS_RESPONSE_CHECKSUM_VALIDATION","value":"when_required"}],"resources":{"requests":{"cpu":"50m","memory":"64Mi"},"limits":{"memory":"256Mi"}}}]}}'
|
||||
|
||||
say "probe: ns=${ns} secret=${sec} (${mode}) targets='${targets}' → ${min} assertions required"
|
||||
kubectl -n "${ns}" delete pod "${pod}" --ignore-not-found >/dev/null 2>&1 || true
|
||||
S3CHECK_PODS+=("${ns}/${pod}")
|
||||
kubectl -n "${ns}" run "${pod}" --restart=Never --image="${S3CHECK_IMAGE}" \
|
||||
--overrides="${overrides}" >/dev/null 2>&1 \
|
||||
|| die "could NOT create the probe pod ${ns}/${pod} (admission policy? RBAC? quota?).
|
||||
That is a FAILURE, not an unknown: nothing about ${ep} has been proven."
|
||||
|
||||
local i=0 phase=""
|
||||
while [ "${i}" -lt 300 ]; do
|
||||
phase="$(kubectl -n "${ns}" get pod "${pod}" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
case "${phase}" in Succeeded|Failed) break ;; esac
|
||||
sleep 5; i=$((i+5))
|
||||
done
|
||||
case "${phase}" in
|
||||
Succeeded|Failed) ;;
|
||||
*) local reason
|
||||
reason="$(kubectl -n "${ns}" get pod "${pod}" \
|
||||
-o jsonpath='{.status.containerStatuses[0].state.waiting.reason}' 2>/dev/null || true)"
|
||||
die "probe pod ${ns}/${pod} did not finish in ${i}s (phase=${phase:-<none>}, waiting=${reason:-none}).
|
||||
NOT a pass. If waiting=ErrImagePull/ImagePullBackOff this cluster cannot pull ${S3CHECK_IMAGE}." ;;
|
||||
esac
|
||||
|
||||
local log; log="$(kubectl -n "${ns}" logs "${pod}" 2>/dev/null || true)"
|
||||
[ -n "${log}" ] || die "probe pod ${ns}/${pod} produced NO output (phase=${phase}). Failing closed —
|
||||
an empty log is the shape of a probe that never ran, not of a healthy backup target."
|
||||
printf '%s\n' "${log}" | while IFS= read -r l; do printf ' %s\n' "${l}"; done
|
||||
|
||||
# ---- the four independent accounting checks that make a silent no-op impossible ----
|
||||
local okcount failcount resline nres verd a f
|
||||
okcount="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-OK\[' || true)"
|
||||
failcount="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-FAIL' || true)"
|
||||
nres="$(printf '%s\n' "${log}" | grep -c '^S3CHECK-RESULT ' || true)"
|
||||
[ "${nres}" = "1" ] || die "expected exactly ONE S3CHECK-RESULT line from ${ns}/${pod}, found ${nres}.
|
||||
A missing verdict line means the probe died mid-way; it is never a pass."
|
||||
resline="$(printf '%s\n' "${log}" | grep '^S3CHECK-RESULT ')"
|
||||
read -r _ verd a f <<<"${resline}"
|
||||
a="${a#assertions=}"; f="${f#failures=}"
|
||||
case "${a}" in ''|*[!0-9]*) die "malformed verdict from ${ns}/${pod}: '${resline}'" ;; esac
|
||||
case "${f}" in ''|*[!0-9]*) die "malformed verdict from ${ns}/${pod}: '${resline}'" ;; esac
|
||||
[ "${verd}" = "PASS" ] || die "S3 ROUND TRIP FAILED against ${ep} (${ns}/${pod}: ${f} failure(s)).
|
||||
Read the S3CHECK-FAIL lines above — each one names the cause and the fix. Do NOT enable velero
|
||||
or CNPG backups against this endpoint until they are all gone."
|
||||
[ "${f}" = "0" ] || die "${ns}/${pod} claimed PASS while reporting ${f} failures — refusing to believe it."
|
||||
[ "${failcount}" = "0" ] || die "${ns}/${pod} claimed PASS but the log carries ${failcount} S3CHECK-FAIL line(s)."
|
||||
[ "${a}" -ge "${min}" ] || die "${ns}/${pod} claimed PASS with only ${a} assertions; ${min} are required
|
||||
(1 credential + 6 per bucket). A short count means steps were SKIPPED — exactly the
|
||||
degrade-to-a-successful-no-op this check exists to prevent."
|
||||
[ "${okcount}" = "${a}" ] || die "assertion accounting mismatch for ${ns}/${pod}: ${okcount} S3CHECK-OK
|
||||
lines in the log vs a claimed ${a}. Refusing to pass on a self-report that does not add up."
|
||||
[ "${phase}" = "Succeeded" ] || die "${ns}/${pod} printed PASS but the pod ended ${phase}."
|
||||
info "PROVEN: ${a} assertions, 0 failures, pod ${phase}, no object left behind."
|
||||
}
|
||||
|
||||
# ============================================================================ steps
|
||||
step_preflight() {
|
||||
say "PREFLIGHT (read-only)"
|
||||
@@ -200,13 +561,14 @@ step_preflight() {
|
||||
info "owner id is distinct and non-default: OK"
|
||||
fi
|
||||
|
||||
case "${s3}" in
|
||||
""|TBD-*) warn "S3_BACKUP_ENDPOINT is unset/placeholder ('${s3}'). velero AND apps-wave-0 consume it.
|
||||
if s3_is_placeholder "${s3}"; then
|
||||
warn "S3_BACKUP_ENDPOINT is unset/placeholder ('${s3}'). velero AND apps-wave-0 consume it.
|
||||
velero will be SKIPPED by the resume step. apps-wave-0 will still apply (the token IS defined,
|
||||
so strict substitution succeeds) but anything writing to that endpoint fails at RUNTIME, not at
|
||||
apply time — so a green apps-wave-0 does NOT mean backups work. Operator decision required." ;;
|
||||
*) info "S3 backup endpoint set: OK" ;;
|
||||
esac
|
||||
apply time — so a green apps-wave-0 does NOT mean backups work. Operator decision required."
|
||||
else
|
||||
info "S3 backup endpoint set: OK"
|
||||
fi
|
||||
|
||||
# COLD-START CRD ORDERING. Flux server-side dry-runs EVERY object in a Kustomization before applying
|
||||
# any of them, so a single unknown `kind` aborts the whole tier. platform-config applies the CDI CR
|
||||
@@ -394,51 +756,139 @@ step_externaldns() {
|
||||
}
|
||||
|
||||
step_s3check() {
|
||||
say "S3 BACKUP ENDPOINT — reachability FROM INSIDE THE CLUSTER"
|
||||
# Velero runs in-cluster, so the only reachability that counts is the pods' own. Probing from the
|
||||
# workstation proves nothing either way: prod sits on the SOHO network (pfSense's WAN) while the
|
||||
# dev LAN's MinIO sits inside pfSense, so host and pod egress differ; a healthy reading there
|
||||
# network, and a healthy result there has already misled this project once.
|
||||
local ep live; live="$(cvar S3_BACKUP_ENDPOINT)"
|
||||
say "S3 BACKUP TARGET — PROVE A REAL ROUND TRIP FROM INSIDE THE CLUSTER"
|
||||
# Velero and CNPG run in-cluster, so pod egress is the only reachability that counts: prod sits on
|
||||
# the SOHO network (pfSense's WAN) while the dev LAN's MinIO sits INSIDE pfSense, so host and pod
|
||||
# egress differ — and a healthy host-side reading has already misled this project once.
|
||||
#
|
||||
# WHAT THIS PROVES (and it fails closed on every one of them):
|
||||
# 1. the committed credential loads and AUTHENTICATES, in the exact shape each consumer reads it
|
||||
# (velero: an AWS INI file from the `cloud` key; CNPG: discrete ACCESS_KEY_ID/ACCESS_SECRET_KEY);
|
||||
# 2. each required bucket EXISTS and is LISTABLE under this environment's own prefix;
|
||||
# 3. PUT -> GET -> byte-compare -> DELETE of a throwaway object under that prefix — the genuine
|
||||
# write path, not a HEAD (a HEAD passes with a read-only token and with R2 rejecting writes);
|
||||
# 4. whether the R2 CHECKSUM TRAP is present at this endpoint, named explicitly either way.
|
||||
#
|
||||
# WHAT IT STILL DOES NOT PROVE: that Velero's kopia uploader and CNPG's barman-cloud themselves work
|
||||
# against this target. Both are UNPROVEN against R2 in this repo. See infra/BACKUP-R2.md.
|
||||
local ep live region
|
||||
live="$(cvar S3_BACKUP_ENDPOINT)"
|
||||
info "live S3_BACKUP_ENDPOINT = ${live:-<unset>}"
|
||||
if [ -n "${S3_ENDPOINT_OVERRIDE}" ]; then
|
||||
# Probe a CANDIDATE endpoint before adopting it. The whole point is to find out whether a proposed
|
||||
# backup target is reachable from this cluster's pods BEFORE authoring it into cluster-vars —
|
||||
# otherwise the first evidence is silently-missing backups.
|
||||
# backup target works from this cluster's pods BEFORE authoring it into cluster-vars — otherwise
|
||||
# the first evidence is silently-missing backups.
|
||||
ep="${S3_ENDPOINT_OVERRIDE}"
|
||||
info "probing CANDIDATE endpoint from --s3-endpoint: ${ep}"
|
||||
else
|
||||
ep="${live}"
|
||||
fi
|
||||
if s3_is_placeholder "${ep}"; then
|
||||
die "no usable endpoint to probe (endpoint='${ep:-<unset>}').
|
||||
cluster-vars still carries the loud placeholder. Pass --s3-endpoint https://<account-id>.r2.cloudflarestorage.com
|
||||
to test a CANDIDATE before authoring it. See infra/BACKUP-R2.md."
|
||||
fi
|
||||
s3_safe_token S3_BACKUP_ENDPOINT "${ep}"
|
||||
case "${ep}" in
|
||||
""|TBD-*) die "no usable endpoint to probe (live='${live}'). Pass --s3-endpoint <url> to test a candidate." ;;
|
||||
http://*|https://*) ;;
|
||||
*) die "endpoint '${ep}' carries no http:// or https:// scheme — the AWS SDK needs one." ;;
|
||||
esac
|
||||
|
||||
local ns=velero pod=s3check-$$
|
||||
kubectl get ns "${ns}" >/dev/null 2>&1 || ns=default
|
||||
info "probing from a throwaway pod in namespace '${ns}'"
|
||||
# --restart=Never + explicit delete in a trap: a probe must never leave a pod behind on a cluster.
|
||||
trap 'kubectl -n "${ns}" delete pod "${pod}" --ignore-not-found --wait=false >/dev/null 2>&1 || true' RETURN
|
||||
if kubectl -n "${ns}" run "${pod}" --restart=Never --image=curlimages/curl:8.11.1 --command --quiet \
|
||||
-- curl -sS -o /dev/null -w '%{http_code}' --max-time 15 "${ep}/minio/health/live" >/dev/null 2>&1; then
|
||||
local i=0 phase=""
|
||||
while [ "$i" -lt 90 ]; do
|
||||
phase="$(kubectl -n "${ns}" get pod "${pod}" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
|
||||
case "${phase}" in Succeeded|Failed) break ;; esac
|
||||
sleep 5; i=$((i+5))
|
||||
done
|
||||
local out; out="$(kubectl -n "${ns}" logs "${pod}" 2>/dev/null || true)"
|
||||
info "pod phase=${phase:-unknown} response=${out:-<none>}"
|
||||
case "${out}" in
|
||||
200|2??) info "REACHABLE: ${ep} answered from inside the cluster." ;;
|
||||
*) warn "NOT REACHABLE from inside the cluster (phase=${phase}, body='${out}').
|
||||
Velero would install cleanly and then fail every backup at RUNTIME — a tier that looks healthy
|
||||
while no backup exists. Fix routing, or give this environment its own endpoint, before
|
||||
resuming velero." ;;
|
||||
esac
|
||||
else
|
||||
warn "could not create the probe pod (admission policy? image pull?). Reachability UNKNOWN."
|
||||
# Region is a knob, not a constant: MinIO ignores it, R2 requires exactly `auto`, and an EMPTY value
|
||||
# in cluster-vars silently builds a BSL with no region.
|
||||
region="$(cvar S3_BACKUP_REGION)"
|
||||
case "${ep}" in
|
||||
*.r2.cloudflarestorage.com*)
|
||||
if [ "${region}" != "auto" ]; then
|
||||
warn "endpoint is Cloudflare R2 but cluster-vars S3_BACKUP_REGION='${region:-<unset>}'. R2 requires
|
||||
exactly 'auto'. Probing with 'auto' anyway; FIX clusters/base/cluster-vars.yaml before enabling
|
||||
velero, or the BackupStorageLocation is built with the wrong region."
|
||||
fi
|
||||
region=auto ;;
|
||||
*)
|
||||
if [ -z "${region}" ]; then
|
||||
region="us-east-1"
|
||||
warn "cluster-vars S3_BACKUP_REGION is EMPTY on this cluster. Probing with 'us-east-1' (MinIO
|
||||
ignores it) — but the BSL would be rendered with region: \"\". Set it explicitly."
|
||||
fi ;;
|
||||
esac
|
||||
s3_safe_token S3_BACKUP_REGION "${region}"
|
||||
|
||||
# Buckets come from the AUTHORED manifests + cluster-vars, never from a hardcoded name here: the
|
||||
# whole point is to probe what this environment will really write to (R-ENV-02, no dev/prod fork).
|
||||
local vbucket vprefix cpath cbucket cprefix rest
|
||||
vbucket="$(yq -r '.spec.values.configuration.backupStorageLocation[0].bucket' \
|
||||
platform/velero/helmrelease.yaml 2>/dev/null | tr -d '"')"
|
||||
[ -n "${vbucket}" ] && [ "${vbucket}" != "null" ] \
|
||||
|| die "could not read the Velero bucket name from platform/velero/helmrelease.yaml"
|
||||
vprefix="$(cvar VELERO_BACKUP_PREFIX)"
|
||||
|
||||
cpath="$(cvar CNPG_BACKUP_PATH)"
|
||||
if [ -z "${cpath}" ]; then
|
||||
warn "cluster-vars CNPG_BACKUP_PATH is EMPTY on this cluster. barmanObjectStore.destinationPath would
|
||||
substitute to the empty string and EVERY WAL archive would fail at runtime. Probing the historical
|
||||
default s3://cnpg so the round trip is still proven — but SET THIS VALUE."
|
||||
cpath="s3://cnpg"
|
||||
fi
|
||||
case "${cpath}" in s3://*) ;; *) die "CNPG_BACKUP_PATH='${cpath}' is not an s3:// URL." ;; esac
|
||||
rest="${cpath#s3://}"; cbucket="${rest%%/*}"
|
||||
if [ "${rest}" = "${cbucket}" ]; then cprefix=""; else cprefix="${rest#*/}"; fi
|
||||
cprefix="${cprefix%/}"
|
||||
[ -n "${cbucket}" ] || die "CNPG_BACKUP_PATH='${cpath}' names no bucket."
|
||||
|
||||
s3_safe_token velero-bucket "${vbucket}"; s3_safe_token VELERO_BACKUP_PREFIX "${vprefix}"
|
||||
s3_safe_token cnpg-bucket "${cbucket}"; s3_safe_token CNPG_BACKUP_PATH-prefix "${cprefix}"
|
||||
|
||||
say "what is about to be proven"
|
||||
info "endpoint ${ep}"
|
||||
info "region ${region}"
|
||||
info "velero bucket '${vbucket}' prefix '${vprefix:-<bucket root>}' (authored bucket + VELERO_BACKUP_PREFIX)"
|
||||
info "cnpg bucket '${cbucket}' prefix '${cprefix:-<bucket root>}' (from CNPG_BACKUP_PATH)"
|
||||
info "scratch key <prefix>/.s3check/<pod>.probe — written, read back, byte-compared, DELETED"
|
||||
|
||||
local ran=0 keys
|
||||
# (a) Velero's own credential, from its own namespace, mounted the way the aws plugin reads it.
|
||||
if kubectl get ns velero >/dev/null 2>&1 \
|
||||
&& kubectl -n velero get secret velero-cloud-credentials >/dev/null 2>&1; then
|
||||
# List KEY NAMES only. Never pipe or print a value (a careless redaction has leaked a password
|
||||
# into a transcript on this project already). go-template, not jsonpath: kubectl's jsonpath
|
||||
# cannot range over a map's keys (`range $k,$v :=` is rejected, U+002C) and returns EMPTY, which
|
||||
# silently looked like "the key is missing".
|
||||
keys="$(kubectl -n velero get secret velero-cloud-credentials \
|
||||
-o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}' 2>/dev/null || true)"
|
||||
printf '%s\n' "${keys}" | grep -qx 'cloud' \
|
||||
|| die "Secret velero/velero-cloud-credentials carries no 'cloud' key (keys: $(printf '%s' "${keys}" | tr '\n' ' ')).
|
||||
velero-plugin-for-aws reads exactly that key as an AWS INI file; without it every backup fails
|
||||
at RUNTIME while the HelmRelease reports Ready."
|
||||
s3_probe velero file velero-cloud-credentials "${vbucket}:${vprefix}" "${ep}" "${region}"
|
||||
ran=$((ran+1))
|
||||
else
|
||||
warn "no Secret velero/velero-cloud-credentials on this cluster — Velero's own credential could NOT
|
||||
be proven. That is expected before the velero tier has ever reconciled here; it is not a pass."
|
||||
fi
|
||||
|
||||
# (b) A CNPG/barman credential, discovered rather than hardcoded (R-ENV-02: no per-environment list
|
||||
# of app names in this script). Any one of them proves the key + the cnpg bucket + the discrete
|
||||
# ACCESS_KEY_ID/ACCESS_SECRET_KEY shape; they are all kept in sync by design.
|
||||
local pick cns csec
|
||||
pick="$(kubectl get secret -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\n"}{end}' 2>/dev/null \
|
||||
| grep -- '-db-backup-credentials$' | head -n 1 || true)"
|
||||
if [ -n "${pick}" ]; then
|
||||
cns="${pick%%/*}"; csec="${pick#*/}"
|
||||
s3_probe "${cns}" env "${csec}" "${cbucket}:${cprefix}" "${ep}" "${region}"
|
||||
ran=$((ran+1))
|
||||
else
|
||||
warn "no *-db-backup-credentials Secret found in any namespace — the CNPG/barman credential could
|
||||
NOT be proven."
|
||||
fi
|
||||
|
||||
[ "${ran}" -gt 0 ] || die "NO backup credential Secret exists on this cluster, so NOTHING could be
|
||||
proven about ${ep}. That is a FAILURE, not a skip: a green s3check must never mean 'I found
|
||||
nothing to check'."
|
||||
|
||||
say "S3 ROUND TRIP PROVEN from inside the cluster: ${ep} (${ran} credential path(s))"
|
||||
info "STILL UNPROVEN by this step: Velero's kopia uploader and CNPG barman-cloud against this target,"
|
||||
info "and a RESTORE. A backup that has never been restored is not a backup — see infra/BACKUP-R2.md."
|
||||
}
|
||||
|
||||
step_resume() {
|
||||
@@ -453,13 +903,20 @@ step_resume() {
|
||||
wait_ready "$n" 900 || { warn "stopping at $n — a partial, understood bring-up beats a forced green."; return 1; }
|
||||
done
|
||||
|
||||
case "${s3}" in
|
||||
""|TBD-*) warn "velero left SUSPENDED: S3_BACKUP_ENDPOINT='${s3}' is not a usable endpoint.
|
||||
# The placeholder test MUST be substring-based. It was `TBD-*` (a PREFIX match) while the placeholder
|
||||
# was `TBD-PROD-…`; the R2 move changed it to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`,
|
||||
# which does not START with TBD- — so this case arm silently stopped matching and `resume` would have
|
||||
# RESUMED velero onto a placeholder hostname on prod. That is the precise failure this guard exists
|
||||
# to prevent, reintroduced by a value change in another file. s3_is_placeholder matches anywhere.
|
||||
if s3_is_placeholder "${s3}"; then
|
||||
warn "velero left SUSPENDED: S3_BACKUP_ENDPOINT='${s3}' is not a usable endpoint.
|
||||
Resuming it would create a BackupStorageLocation that can never reach a bucket, i.e. a tier
|
||||
that reports Ready-ish while backups silently do not exist. Set the real prod endpoint first." ;;
|
||||
*) info "resuming velero"; flux resume kustomization velero --timeout=1s >/dev/null 2>&1 || true
|
||||
wait_ready velero 900 || warn "velero did not converge" ;;
|
||||
esac
|
||||
that reports Ready-ish while backups silently do not exist. Set the real endpoint first, prove
|
||||
it with \`s3check\`, and only then resume (infra/BACKUP-R2.md)."
|
||||
else
|
||||
info "resuming velero"; flux resume kustomization velero --timeout=1s >/dev/null 2>&1 || true
|
||||
wait_ready velero 900 || warn "velero did not converge"
|
||||
fi
|
||||
}
|
||||
|
||||
step_verify() {
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
# Off-cluster backup on Cloudflare R2 — operator procedure
|
||||
|
||||
**Owner:** WS-BACKUP · **Serves:** R-STATE-03 (DR restore), R-IDP-05 (Keycloak DB), R-ENV-02 (one
|
||||
environment-agnostic path) · **Status:** authored and staged, **NOT switched on**.
|
||||
|
||||
> **Read this first.** Velero-against-R2 and CNPG/barman-against-R2 are **UNPROVEN in this repo**.
|
||||
> Nothing here claims they work. What *is* proven is in [§7](#7-what-is-proven-and-what-is-not).
|
||||
> R-STATE-03 is currently PROVEN **against MinIO only** and must be re-proven against R2 before this
|
||||
> migration can be called done.
|
||||
|
||||
## Why R2 at all
|
||||
|
||||
The off-cluster target used to be a MinIO on the Proxmox host (`http://192.168.1.9:9000`). Two things
|
||||
killed it, both recorded on 2026-09-05:
|
||||
|
||||
1. **Prod physically cannot reach it.** Prod is on the SOHO network `192.168.2.0/24`, which is
|
||||
pfSense's **WAN**; the MinIO sits inside pfSense's LAN `192.168.1.0/24`. Measured *from a pod inside
|
||||
prod* — the workstation's own reachability proves nothing about pod egress and has misled this
|
||||
project once already — `curl http://192.168.1.9:9000/minio/health/live` timed out after 15 s.
|
||||
Reaching it would need a WAN→LAN rule that does not exist.
|
||||
2. **Dev's "off-cluster" backup was not off-cluster.** That MinIO runs on the *same Proxmox host* as
|
||||
dev's VMs, so it shared a failure domain with the thing it was backing up.
|
||||
|
||||
R2 needs no firewall change, is genuinely off-site, and is already trusted with the OpenTofu remote
|
||||
state (`infra/tofu/REMOTE-STATE.md`) — the same account, the same kind of token. It is explicitly an
|
||||
interim decision: *"that's where backups will go for the time being until a better on-prem backup
|
||||
solution is created."*
|
||||
|
||||
---
|
||||
|
||||
## 1. Cloudflare: buckets, token, account id
|
||||
|
||||
All of this is in the Cloudflare dashboard; none of it touches the cluster.
|
||||
|
||||
1. **R2 → Create bucket** → `velero`. Location hint: whatever is nearest; jurisdiction default.
|
||||
2. **R2 → Create bucket** → `cnpg`.
|
||||
Both names are fixed by the manifests: `bucket: velero` in `platform/velero/helmrelease.yaml`, and
|
||||
`cnpg` via `CNPG_BACKUP_PATH` (`s3://cnpg/dev`, `s3://cnpg/prod`). Use exactly those names or change
|
||||
the manifests — do not improvise.
|
||||
3. **R2 → Manage API Tokens → Create API Token**
|
||||
- Permission: **Object Read & Write** (not "Admin Read & Write" — the cluster never needs to create
|
||||
or delete buckets, and a token that can is a token that can delete your backups).
|
||||
- **Specify bucket(s):** `velero` **and** `cnpg`. One token scoped to both — the manifests already
|
||||
assume one shared key, and `s3check` proves both buckets with it.
|
||||
- Copy the **Access Key ID** and **Secret Access Key**. They are shown **once**. Put them in the
|
||||
password manager before you do anything else.
|
||||
4. **Account id** — 32 hex characters, visible in the dashboard URL and in the R2 S3 endpoint the
|
||||
token page shows you.
|
||||
|
||||
**This repo already knows it.** `infra/tofu/backend.r2.hcl` commits
|
||||
`https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com` for the OpenTofu state bucket,
|
||||
with the (correct) note that an account id is not a credential. **If the backup buckets live in the
|
||||
same Cloudflare account as the tofu state — they should; there is no reason for a second account —
|
||||
then the endpoint is already known and you do not need to look anything up:**
|
||||
|
||||
```
|
||||
https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
|
||||
```
|
||||
|
||||
Confirm it matches the endpoint on the token page. If it does not, you have two accounts and the
|
||||
`velero`/`cnpg` buckets are in the other one — use that one's id, and say so in the DECISIONS-LOG.
|
||||
|
||||
The endpoint form is always `https://<account-id>.r2.cloudflarestorage.com` — **no bucket in the
|
||||
hostname** (path-style; see [§6](#6-r2-specific-gotchas)).
|
||||
|
||||
---
|
||||
|
||||
## 2. Repo values to set
|
||||
|
||||
Only **one** non-secret value is missing, and it is not in a secret file:
|
||||
|
||||
| File | Key | From | To |
|
||||
|---|---|---|---|
|
||||
| `clusters/base/cluster-vars.yaml` | `S3_BACKUP_ENDPOINT` | `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` | `https://<account-id>.r2.cloudflarestorage.com` |
|
||||
|
||||
Already authored, **do not change**: `S3_BACKUP_REGION: "auto"`, `VELERO_BACKUP_PREFIX` (dev `dev` /
|
||||
prod `prod`), `CNPG_BACKUP_PATH` (`s3://cnpg/dev` / `s3://cnpg/prod`), and `checksumAlgorithm: ""` on
|
||||
the Velero BackupStorageLocation.
|
||||
|
||||
The placeholder is deliberately loud, and the guards match it as a **substring** — `bootstrap/standup.sh`
|
||||
refuses to probe it and refuses to resume `velero` while it is there.
|
||||
|
||||
---
|
||||
|
||||
## 3. Credentials: four Secrets, one command
|
||||
|
||||
The R2 key is read by **four** committed SOPS Secrets, in **two** different shapes. No manifest changes
|
||||
are needed — only the encrypted values change:
|
||||
|
||||
| File | Consumer | Shape |
|
||||
|---|---|---|
|
||||
| `platform/velero/velero-cloud.sops.yaml` | Velero (`velero-cloud-credentials`, ns `velero`) | `stringData.cloud` — an **AWS INI file**: `[default]` + `aws_access_key_id=` / `aws_secret_access_key=` |
|
||||
| `platform/keycloak/keycloak-db-backup-credentials.sops.yaml` | CNPG `keycloak-db` (ns `keycloak`) | `ACCESS_KEY_ID` / `ACCESS_SECRET_KEY` |
|
||||
| `apps/wave-0/nextcloud-db-backup-credentials.sops.yaml` | CNPG `nextcloud-db` | `ACCESS_KEY_ID` / `ACCESS_SECRET_KEY` |
|
||||
| `apps/wave-0/semaphore-db-backup-credentials.sops.yaml` | CNPG `semaphore-db` | `ACCESS_KEY_ID` / `ACCESS_SECRET_KEY` |
|
||||
|
||||
A Secret is namespace-scoped and a Pod reads only its own namespace's, which is why there are four
|
||||
copies of one key rather than one.
|
||||
|
||||
### The one command
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run platform/backup/set-r2-credentials.sh
|
||||
```
|
||||
|
||||
It prompts for the two values (never echoed, never in shell history, never written to a file), writes
|
||||
them into all four Secrets with `sops set` — which encrypts in place, so **no decrypted copy of any
|
||||
Secret is ever created** — then decrypts each file back and proves all four now hold the same key by
|
||||
comparing 32-bit fingerprints. It never prints a credential.
|
||||
|
||||
There is deliberately **no plaintext template to fill in and encrypt**. The
|
||||
`cp x.example x.yaml && $EDITOR && sops -e -i` pattern leaves a plaintext Kubernetes Secret sitting in
|
||||
the working tree, one `git add -A` away from being in git forever; `sops set` removes that window
|
||||
entirely. If you would rather edit by hand, `sops platform/velero/velero-cloud.sops.yaml` decrypts to
|
||||
`$EDITOR` and re-encrypts on save — also with no plaintext on disk — but then you must repeat it four
|
||||
times and keep them identical yourself.
|
||||
|
||||
To check the four files agree without changing anything:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run platform/backup/set-r2-credentials.sh --check
|
||||
```
|
||||
|
||||
### What is NOT changed
|
||||
|
||||
`platform/backup/minio-credentials.sops.yaml` keeps the **MinIO** key. That is not an oversight — it
|
||||
is what you need to read the pre-migration backups ([§8](#8-going-back-to-minio)). Its
|
||||
`.yaml.example` sibling documents the MinIO shape and stays accurate for the MinIO target.
|
||||
|
||||
Review the diff before committing: you should see **only** `ENC[...]` blobs and `lastmodified`/`mac`
|
||||
change in four files. If a key name or a `metadata.name` moved, stop.
|
||||
|
||||
---
|
||||
|
||||
## 4. Prove the target BEFORE merging
|
||||
|
||||
```bash
|
||||
K=infra/ansible/.cache/dev.kubeconfig # or prod.kubeconfig
|
||||
./infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||||
--kubeconfig $K --expect-server https://192.168.2.50:6443 \
|
||||
--s3-endpoint https://<account-id>.r2.cloudflarestorage.com s3check
|
||||
```
|
||||
|
||||
`--s3-endpoint` probes a **candidate** endpoint instead of the live `cluster-vars` value, so the target
|
||||
is proven before it is authored. Run it against **both** clusters — prod's egress is the whole reason
|
||||
this migration exists, and it is a different network position from dev's.
|
||||
|
||||
`s3check` runs a throwaway pod per credential and, for each, proves:
|
||||
|
||||
1. the committed credential **loads and authenticates**, in the exact shape its consumer reads it
|
||||
(Velero's INI file mounted from the `cloud` key; CNPG's discrete `ACCESS_KEY_ID`/`ACCESS_SECRET_KEY`);
|
||||
2. the bucket **exists and is listable** under this environment's own prefix (`ListObjectsV2`, the
|
||||
operation Velero's BSL sync and Barman's WAL scan both perform);
|
||||
3. **PUT → GET → byte-compare (sha256 + length) → DELETE → confirm-gone** of one throwaway object at
|
||||
`<prefix>/.s3check/<pod>.probe` — a genuine write, not a HEAD. A HEAD passes with a read-only token
|
||||
and with R2 rejecting every write;
|
||||
4. whether the **R2 checksum trap** ([§6](#6-r2-specific-gotchas)) is present, probed on purpose by
|
||||
repeating the PUT with checksums *enabled*, and named explicitly either way.
|
||||
|
||||
It **fails closed**: a pod that cannot be created, an empty log, a missing verdict line, a short
|
||||
assertion count, an OK-line count that disagrees with the claimed count, or any single failed
|
||||
assertion all exit non-zero. There is no "reachability unknown, carrying on". Each failure names its
|
||||
own cause and fix — wrong key, unscoped token, missing bucket, no pod egress, checksum rejection.
|
||||
|
||||
It writes and deletes exactly one object per bucket and leaves no pod behind. It applies nothing and
|
||||
does not enable `velero`.
|
||||
|
||||
---
|
||||
|
||||
## 5. The verification sequence — and it ends in a RESTORE
|
||||
|
||||
**A backup that has never been restored is not a backup.** R-STATE-03 is PROVEN against MinIO; that
|
||||
proof does not transfer to a different object store with a different S3 implementation.
|
||||
|
||||
1. **`s3check` on dev, then on prod.** Both must pass. (§4)
|
||||
2. Merge the endpoint + the four re-encrypted Secrets. Let Flux reconcile.
|
||||
3. **Resume `velero`** — `bootstrap/standup.sh … resume` does it once the endpoint is no longer a
|
||||
placeholder, or `flux resume kustomization velero`.
|
||||
4. `velero backup-location get` → `default` **Available**. *Available only means the BSL could be
|
||||
read;* it is not a backup.
|
||||
5. `velero backup create --from-schedule daily` → wait for **Completed**, then
|
||||
`velero backup logs <name>`. Watch specifically for `PodVolumeBackup` failures: the PVC-data path
|
||||
is kopia, a **different S3 client** from the one the BSL uses (see §7), so metadata can succeed
|
||||
while volume data fails.
|
||||
6. **Restore.** `velero restore create --from-backup <name> --include-namespaces <scratch-ns>` into a
|
||||
*scratch* namespace, and verify the PVC contents actually arrived — not just that the Restore says
|
||||
Completed. This is the R-STATE-03 re-proof.
|
||||
7. **CNPG PITR.** Confirm WAL is arriving:
|
||||
`kubectl -n keycloak get cluster keycloak-db -o jsonpath='{.status.conditions}'` (look for
|
||||
`ContinuousArchiving`), and that objects exist under `s3://cnpg/<env>/keycloak-db/`. Then do the
|
||||
restore that matters: a `Cluster` with `bootstrap.recovery` from the barman object store into a
|
||||
scratch namespace, and check the recovered database actually contains data. The existing
|
||||
`conformance/tests/l2-behavioral/keycloak-db-dr` and `l3-fault-injection/dr-restore` tests are the
|
||||
harness for this; they must be re-run **against R2**, and their previous PROVEN state records MinIO,
|
||||
not R2.
|
||||
8. Only after 6 and 7 pass may this migration be described as done, and the ledger updated. Until
|
||||
then the honest status is "R2 accepts objects; recovery from R2 is untested".
|
||||
|
||||
---
|
||||
|
||||
## 6. R2-specific gotchas
|
||||
|
||||
**`region: auto`.** R2 accepts nothing else, and S3 clients require *some* region. Authored as
|
||||
`S3_BACKUP_REGION: "auto"`. This repo already learned it — `infra/tofu/backend.tf` carries
|
||||
`region = "auto" # R2 ignores region, but the S3 backend requires a value.`
|
||||
|
||||
**`checksumAlgorithm: ""` — do not delete this line.** `velero-plugin-for-aws`'s own
|
||||
`backupstoragelocation.md` for **v1.14.2** (the pinned version) documents:
|
||||
|
||||
> `checksumAlgorithm` — The checksum algorithm to use for uploading objects to S3. Supported values
|
||||
> "CRC32", "CRC32C", "SHA1", "SHA256". If the value is set as empty string "", no checksum will be
|
||||
> calculated and attached to the request headers. **Optional (defaults to "CRC32")**
|
||||
|
||||
So *omitting* the key means CRC32 **is** sent: `x-amz-sdk-checksum-algorithm` +
|
||||
`x-amz-checksum-crc32`. This repo has already measured that R2 rejects AWS checksums — that is exactly
|
||||
why `infra/tofu/backend.tf` needs `skip_s3_checksum = true # R2 rejects AWS trailing (streaming)
|
||||
checksums.` for the state bucket. The plugin's own compatibility table lists `checksumAlgorithm=""` as
|
||||
the remedy for the same failure on Ceph S3, Backblaze B2 and IBM COS. If backups start failing with
|
||||
**HTTP 400 `InvalidArgument`** after a plugin bump, look here first. `s3check` probes for this on
|
||||
purpose and names it.
|
||||
|
||||
**Path-style.** `s3ForcePathStyle: "true"` is authored and correct for R2 as well as MinIO. Never put
|
||||
the bucket in the hostname.
|
||||
|
||||
**No bucket auto-creation.** Velero does not create buckets and the token deliberately cannot. A
|
||||
missing bucket is a `NoSuchBucket` at runtime; `s3check` catches it first.
|
||||
|
||||
**Token scope is per-bucket.** A token scoped only to `velero` authenticates fine and then 403s on
|
||||
`cnpg` — the two consumers fail independently and quietly. `s3check` proves both.
|
||||
|
||||
**Both clusters share one endpoint.** Isolation is by path only (`VELERO_BACKUP_PREFIX`,
|
||||
`CNPG_BACKUP_PATH`), because R2 is one environment-independent URL. Those two knobs are load-bearing:
|
||||
without them prod would adopt and TTL-delete dev's Velero backups, and both clusters' identically-named
|
||||
CNPG Clusters would interleave their WAL into one Barman stream, destroying PITR for both.
|
||||
|
||||
---
|
||||
|
||||
## 7. What is proven, and what is not
|
||||
|
||||
**Proven (2026-09-05, WS-BACKUP, against the live dev cluster and its MinIO at
|
||||
`http://192.168.1.9:9000`):**
|
||||
|
||||
- the rewritten `s3check` itself: 7 assertions per credential path, both credential shapes, from
|
||||
in-cluster pods — credential load, bucket list, PUT/GET/byte-compare/DELETE/confirm-gone, no object
|
||||
and no pod left behind;
|
||||
- its failure paths: the `TBD` placeholder is refused; an unreachable endpoint (`:9099`) produces a
|
||||
named no-egress failure and exit 1, not a pass;
|
||||
- MinIO accepts a checksum-bearing PUT, i.e. the trap is a *non-*MinIO problem, and
|
||||
`checksumAlgorithm: ""` is harmless there.
|
||||
|
||||
**Established from the pinned versions (documentation and image inspection, not measured against R2):**
|
||||
|
||||
- `velero-plugin-for-aws` **v1.14.2** pairs with Velero **v1.18.x** (upstream compatibility table) and
|
||||
defaults `checksumAlgorithm` to `CRC32` — so the authored `""` is load-bearing, not decoration.
|
||||
- That plugin's "Non-AWS S3 compatible provider known issues with plugin v1.14.x" table **does not
|
||||
mention Cloudflare R2 at all** — neither as working nor as broken. There is no upstream statement to
|
||||
lean on in either direction.
|
||||
- **The PVC-data path is a different S3 client.** `checksumAlgorithm` is a BackupStorageLocation
|
||||
*object-store plugin* setting. Velero's File System Backup uploads through the unified repository's
|
||||
kopia S3 backend (`pkg/repository/udmrepo/kopialib/backend/s3.go` in Velero v1.18.0), which
|
||||
references only endpoint/region-style options and has **no checksum knob at all**. So R2
|
||||
compatibility for backup *metadata* and for *volume data* are two separate questions, and only step
|
||||
5 of §5 answers the second.
|
||||
- **CNPG/barman is likely clear of the checksum trap, by version.** Inspected inside the live
|
||||
`ghcr.io/cloudnative-pg/postgresql:16.4` image on dev: `barman-cloud-wal-archive 3.11.1`,
|
||||
`botocore 1.35.60`, `boto3 1.35.60`. That botocore predates the 1.36 line in which botocore began
|
||||
attaching CRC32 request checksums by default (the change that introduced
|
||||
`AWS_REQUEST_CHECKSUM_CALCULATION`), so this image should not send them.
|
||||
**Forward risk:** barman-cloud exposes **no** `checksumAlgorithm` equivalent. If the CNPG image is
|
||||
ever bumped to one carrying botocore ≥ 1.36, WAL archiving to R2 could start failing with 400
|
||||
`InvalidArgument` and there is no config knob to turn it off — only `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`
|
||||
in the Pod environment, which CNPG does not currently set. Treat a CNPG image bump as a change that
|
||||
must be re-verified against R2.
|
||||
|
||||
**Not proven, at all:**
|
||||
|
||||
- that Velero can write a usable backup to R2 (metadata or kopia volume data);
|
||||
- that CNPG/barman can archive WAL to R2 and that a `bootstrap.recovery` from R2 succeeds;
|
||||
- that a restore from R2 yields correct data. R-STATE-03's PROVEN state is **MinIO-only**.
|
||||
|
||||
---
|
||||
|
||||
## 8. Going back to MinIO
|
||||
|
||||
The MinIO copy is **not deleted** and stays restorable. To read pre-migration backups:
|
||||
|
||||
1. `clusters/base/cluster-vars.yaml` → `S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"`
|
||||
2. `VELERO_BACKUP_PREFIX` → `""` (the MinIO backups are at the **bucket root**, not under `dev/`) and
|
||||
`CNPG_BACKUP_PATH` → `s3://cnpg` (Barman objects are at `s3://cnpg/<clusterName>/`).
|
||||
3. Restore the MinIO credential into the four Secrets. `platform/backup/minio-credentials.sops.yaml`
|
||||
still holds it: `sops -d --extract '["stringData"]["ACCESS_KEY_ID"]'` it and feed the two values to
|
||||
`platform/backup/set-r2-credentials.sh` (the script cares about the *shape*, not the provider).
|
||||
4. `S3_BACKUP_REGION` may stay `auto` — MinIO ignores the region.
|
||||
5. **Dev only.** Prod cannot reach that MinIO at all (§Why R2); pointing prod back is not an option.
|
||||
|
||||
Reverting for a restore and then returning to R2 costs another PITR reset. Prefer restoring from MinIO
|
||||
into a *scratch* namespace on dev with a second, temporary BackupStorageLocation over flipping the
|
||||
whole cluster's endpoint back and forth.
|
||||
|
||||
---
|
||||
|
||||
## 9. Merging this resets dev's PITR window
|
||||
|
||||
Say it out loud before merging, because it is not recoverable by retrying:
|
||||
|
||||
- Dev's R2 prefix starts **empty**. Velero writes a **fresh kopia repository** — the first backup is a
|
||||
full upload of every included PVC, and no earlier restore point exists in R2.
|
||||
- Every CNPG Cluster takes **new base backups** (`ScheduledBackup … immediate: true`), and the 30-day
|
||||
Barman retention window (`retentionPolicy: "30d"`) rebuilds **from zero**. For roughly a month, dev's
|
||||
point-in-time recovery reaches back only as far as the merge.
|
||||
- The MinIO copy still holds the old window (§8), but reaching it means reverting the endpoint — so it
|
||||
is a deliberate excursion, not a fallback that happens automatically.
|
||||
|
||||
**Do not merge this while dev's proven MinIO restore is the only recovery path you are relying on.**
|
||||
The same applies to prod the day it first backs up anywhere.
|
||||
@@ -2,6 +2,20 @@
|
||||
# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
|
||||
# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
|
||||
#
|
||||
# *** SUPERSEDED AS THE LIVE TARGET (2026-09-05), RETAINED ON PURPOSE. ***
|
||||
# Off-cluster DR moved to Cloudflare R2: prod physically cannot reach this MinIO (prod is on the SOHO
|
||||
# net 192.168.2.0/24, pfSense's WAN; the MinIO is inside pfSense's LAN — measured from a pod inside
|
||||
# prod: timed out), and this MinIO runs on the SAME Proxmox host as dev's VMs, so dev's "off-cluster"
|
||||
# backup shared a failure domain with dev. The R2 procedure is infra/BACKUP-R2.md.
|
||||
#
|
||||
# This file and its encrypted sibling minio-credentials.sops.yaml are KEPT because the MinIO copy of
|
||||
# the backups is NOT deleted and stays restorable: this is the key that reads the pre-migration
|
||||
# backups. Do NOT overwrite it with the R2 key — infra/BACKUP-R2.md §8 is how you point back here.
|
||||
#
|
||||
# DO NOT hand-fill this for R2. Four Secrets in three namespaces hold that one key, and
|
||||
# platform/backup/set-r2-credentials.sh writes all four with `sops set` — encrypted in place, so no
|
||||
# plaintext Secret ever exists in the working tree to be committed by accident.
|
||||
#
|
||||
# WHAT THIS IS: the scoped (non-root) MinIO S3 access key for the off-cluster backup target that
|
||||
# roles/minio_backup_target stands up on the Proxmox host (playbooks/backup-minio.yml). It is the
|
||||
# credential Velero (R-STATE-03) and CloudNativePG / Barman (R-IDP-05) use to write backups. The
|
||||
|
||||
Executable
+195
@@ -0,0 +1,195 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# set-r2-credentials.sh — put ONE Cloudflare R2 API token into every credential Secret that consumes
|
||||
# the off-cluster backup target, without plaintext ever touching the disk.
|
||||
#
|
||||
# WHY THIS EXISTS
|
||||
# The same S3 key is read by four different Secrets in three namespaces, in TWO different shapes
|
||||
# (Velero wants an AWS INI file; CNPG/barman wants discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY keys).
|
||||
# Hand-editing four SOPS files is four chances to typo one of them — and a typo'd backup credential
|
||||
# does not fail at apply time, it fails at 03:00 with no backup and a green HelmRelease.
|
||||
#
|
||||
# It uses `sops set`, which edits an ALREADY-ENCRYPTED file in place: the new value is encrypted as
|
||||
# it is written and NO decrypted copy of the file is ever created. That is the whole point — the
|
||||
# `cp template plaintext.yaml && edit && sops -e -i` dance leaves a plaintext Secret sitting in the
|
||||
# working tree, one `git add -A` away from being committed forever.
|
||||
#
|
||||
# USAGE (inside the dev container — it has sops and mounts the age key)
|
||||
# ./infra/devcontainer/dev.sh run platform/backup/set-r2-credentials.sh
|
||||
# → prompts for the two values (never echoed, never in shell history, never in a file)
|
||||
#
|
||||
# ./infra/devcontainer/dev.sh run platform/backup/set-r2-credentials.sh --check
|
||||
# → touches nothing; just reports whether the four Secrets already agree with each other
|
||||
#
|
||||
# R2_ACCESS_KEY_ID=… R2_SECRET_ACCESS_KEY=… …/set-r2-credentials.sh --no-prompt
|
||||
# → non-interactive. Prefer the prompts: an env var is visible in /proc and in `history`.
|
||||
#
|
||||
# WHAT IT DOES NOT DO
|
||||
# - It does not touch clusters/base/cluster-vars.yaml. The endpoint is not a secret and belongs in
|
||||
# a reviewable plain manifest; the script prints the exact edit to make.
|
||||
# - It does not commit. Review `git diff` (you will see only ENC[...] blobs change) and commit.
|
||||
# - It does not delete or rewrite platform/backup/minio-credentials.sops.yaml. That is the RETAINED
|
||||
# pre-migration MinIO key and it is what you need to read the pre-migration backups.
|
||||
#
|
||||
# SEE infra/BACKUP-R2.md for the whole procedure, including the restore that actually proves it.
|
||||
set -euo pipefail
|
||||
|
||||
# Belt and braces: never let a stray `set -x` in a parent shell echo a credential out of this script.
|
||||
set +x
|
||||
|
||||
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
root="$(git -C "${here}" rev-parse --show-toplevel 2>/dev/null || (cd "${here}/.." && pwd))"
|
||||
cd "${root}"
|
||||
|
||||
MODE=set
|
||||
PROMPT=1
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--check) MODE=check; shift ;;
|
||||
--no-prompt) PROMPT=0; shift ;;
|
||||
-h|--help) sed -n '2,40p' "${BASH_SOURCE[0]}"; exit 0 ;;
|
||||
*) echo "ERROR: unknown argument '$1'" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
|
||||
say() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||
info() { printf ' %s\n' "$*"; }
|
||||
warn() { printf '\033[33mWARN: %s\033[0m\n' "$*" >&2; }
|
||||
|
||||
command -v sops >/dev/null 2>&1 || die "'sops' not found — run this inside ./infra/devcontainer/dev.sh."
|
||||
|
||||
# ---------------------------------------------------------------- the consumers
|
||||
# file|shape. shape=ini -> stringData.cloud, an AWS INI credentials file (velero-plugin-for-aws)
|
||||
# shape=discrete -> stringData.ACCESS_KEY_ID + ACCESS_SECRET_KEY (CNPG barmanObjectStore)
|
||||
CONSUMERS=(
|
||||
"platform/velero/velero-cloud.sops.yaml|ini"
|
||||
"platform/keycloak/keycloak-db-backup-credentials.sops.yaml|discrete"
|
||||
"apps/wave-0/nextcloud-db-backup-credentials.sops.yaml|discrete"
|
||||
"apps/wave-0/semaphore-db-backup-credentials.sops.yaml|discrete"
|
||||
)
|
||||
|
||||
for entry in "${CONSUMERS[@]}"; do
|
||||
f="${entry%%|*}"
|
||||
[ -f "${f}" ] || die "missing consumer file ${f} — has the layout changed? Fix this script, do not skip the file."
|
||||
grep -q '^sops:' "${f}" || die "${f} has no top-level sops: block — refusing to touch a file that is not SOPS-encrypted."
|
||||
done
|
||||
|
||||
# A 32-bit fingerprint of a high-entropy secret: enough to prove two files hold the SAME value,
|
||||
# nowhere near enough to recover it. Values themselves are NEVER printed.
|
||||
fp() { local h; h="$(printf '%s' "$1" | sha256sum)"; printf '%s' "${h:0:8}"; }
|
||||
|
||||
# Read one value out of an encrypted file without ever writing plaintext anywhere.
|
||||
xtract() { sops -d --extract "$2" "$1"; }
|
||||
|
||||
# Pull the two values out of an AWS INI blob using shell only.
|
||||
ini_field() { # <ini-text> <aws_access_key_id|aws_secret_access_key>
|
||||
local line v=""
|
||||
while IFS= read -r line; do
|
||||
case "${line}" in
|
||||
"$2"=*|"$2 "*|*[![:alnum:]_]"$2"=*) v="${line#*=}"; v="${v#"${v%%[![:space:]]*}"}" ;;
|
||||
esac
|
||||
done <<<"$1"
|
||||
printf '%s' "${v}"
|
||||
}
|
||||
|
||||
report() { # print the fingerprint table; returns non-zero when the four files disagree
|
||||
local entry f shape id sec akid_fp sec_fp first_id="" first_sec="" agree=0
|
||||
printf ' %-58s %-10s %-10s\n' FILE ID-FP SECRET-FP
|
||||
for entry in "${CONSUMERS[@]}"; do
|
||||
f="${entry%%|*}"; shape="${entry##*|}"
|
||||
if [ "${shape}" = ini ]; then
|
||||
local blob; blob="$(xtract "${f}" '["stringData"]["cloud"]' 2>/dev/null || true)"
|
||||
id="$(ini_field "${blob}" aws_access_key_id)"
|
||||
sec="$(ini_field "${blob}" aws_secret_access_key)"
|
||||
blob=""
|
||||
else
|
||||
id="$(xtract "${f}" '["stringData"]["ACCESS_KEY_ID"]' 2>/dev/null || true)"
|
||||
sec="$(xtract "${f}" '["stringData"]["ACCESS_SECRET_KEY"]' 2>/dev/null || true)"
|
||||
fi
|
||||
[ -n "${id}" ] || { warn "${f}: could not read the access key id (age key available?)"; agree=1; }
|
||||
[ -n "${sec}" ] || { warn "${f}: could not read the secret access key"; agree=1; }
|
||||
akid_fp="$(fp "${id}")"; sec_fp="$(fp "${sec}")"
|
||||
printf ' %-58s %-10s %-10s\n' "${f}" "${akid_fp}" "${sec_fp}"
|
||||
if [ -z "${first_id}" ]; then first_id="${akid_fp}"; first_sec="${sec_fp}"
|
||||
elif [ "${akid_fp}" != "${first_id}" ] || [ "${sec_fp}" != "${first_sec}" ]; then agree=1; fi
|
||||
id=""; sec=""
|
||||
done
|
||||
return "${agree}"
|
||||
}
|
||||
|
||||
if [ "${MODE}" = check ]; then
|
||||
say "current state of the four backup credential Secrets (values never printed)"
|
||||
if report; then
|
||||
info "ALL FOUR AGREE. (Agreeing is not the same as being right — prove it with bootstrap/standup.sh s3check.)"
|
||||
exit 0
|
||||
fi
|
||||
die "the four Secrets do NOT all hold the same key. Velero and CNPG would be writing to the same
|
||||
target with different credentials — one of them is stale. Re-run this script without --check."
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------- the two values
|
||||
AKID="${R2_ACCESS_KEY_ID-}"
|
||||
SAK="${R2_SECRET_ACCESS_KEY-}"
|
||||
if [ "${PROMPT}" = 1 ]; then
|
||||
say "paste the R2 API token (Cloudflare dashboard -> R2 -> Manage API Tokens)"
|
||||
info "Object Read & Write, scoped to the velero + cnpg buckets. Neither value is echoed or stored."
|
||||
[ -n "${AKID}" ] || { printf ' Access Key ID: '; IFS= read -rs AKID; printf '\n'; }
|
||||
[ -n "${SAK}" ] || { printf ' Secret Access Key: '; IFS= read -rs SAK; printf '\n'; }
|
||||
fi
|
||||
[ -n "${AKID}" ] || die "no access key id given (prompt, or R2_ACCESS_KEY_ID with --no-prompt)."
|
||||
[ -n "${SAK}" ] || die "no secret access key given (prompt, or R2_SECRET_ACCESS_KEY with --no-prompt)."
|
||||
|
||||
# FAIL CLOSED on anything that is not a plain S3 credential. These values are passed to `sops set` as
|
||||
# JSON, so a quote, a backslash or a newline would corrupt the Secret rather than error — and the
|
||||
# usual cause is a copy-paste that swallowed a newline or a trailing space, which would produce a
|
||||
# credential that authenticates nowhere and a backup that silently never runs.
|
||||
for pair in "access key id:${AKID}" "secret access key:${SAK}"; do
|
||||
case "${pair#*:}" in
|
||||
*[!A-Za-z0-9._~+/=-]*) die "the ${pair%%:*} contains a character outside [A-Za-z0-9._~+/=-].
|
||||
Almost certainly a stray space, quote or newline from the paste. Nothing was changed." ;;
|
||||
esac
|
||||
done
|
||||
# R2 issues 32-hex ids and 64-hex secrets today. A mismatch is a warning, not a refusal: the format is
|
||||
# Cloudflare's to change, and refusing outright would make this script the thing that blocks a rotation.
|
||||
case "${AKID}" in *[!0-9a-f]*|"") warn "the access key id is not 32 lowercase hex chars — unusual for R2. Continuing." ;;
|
||||
*) [ "${#AKID}" = 32 ] || warn "the access key id is ${#AKID} chars, not the usual 32. Continuing." ;;
|
||||
esac
|
||||
[ "${#SAK}" -ge 40 ] || warn "the secret access key is only ${#SAK} chars — unusual for R2 (64). Continuing."
|
||||
|
||||
# ---------------------------------------------------------------- write
|
||||
say "writing the credential into ${#CONSUMERS[@]} SOPS Secrets (encrypted in place; no plaintext file)"
|
||||
for entry in "${CONSUMERS[@]}"; do
|
||||
f="${entry%%|*}"; shape="${entry##*|}"
|
||||
case "${shape}" in
|
||||
ini)
|
||||
# velero-plugin-for-aws reads this key VERBATIM as an AWS credentials file. The [default]
|
||||
# profile name and the two underscore_names are the plugin's contract, not a style choice.
|
||||
sops set "${f}" '["stringData"]["cloud"]' \
|
||||
"\"[default]\\naws_access_key_id=${AKID}\\naws_secret_access_key=${SAK}\\n\"" ;;
|
||||
discrete)
|
||||
sops set "${f}" '["stringData"]["ACCESS_KEY_ID"]' "\"${AKID}\""
|
||||
sops set "${f}" '["stringData"]["ACCESS_SECRET_KEY"]' "\"${SAK}\"" ;;
|
||||
esac
|
||||
info "updated ${f}"
|
||||
done
|
||||
AKID_FP="$(fp "${AKID}")"; SAK_FP="$(fp "${SAK}")"
|
||||
AKID=""; SAK=""
|
||||
|
||||
say "verifying by decrypting each file back (values never printed)"
|
||||
report || die "the files still disagree after writing — do NOT commit. Inspect with 'sops <file>'."
|
||||
info "every file now reads id-fp=${AKID_FP} secret-fp=${SAK_FP} — matching what was pasted."
|
||||
|
||||
say "NEXT — in this order, and do not skip the restore"
|
||||
cat <<'NEXT'
|
||||
1. Set the endpoint (NOT a secret, so it lives in a plain reviewable manifest):
|
||||
clusters/base/cluster-vars.yaml
|
||||
S3_BACKUP_ENDPOINT: "https://<account-id>.r2.cloudflarestorage.com"
|
||||
S3_BACKUP_REGION: "auto" # R2 accepts nothing else
|
||||
2. Prove the target BEFORE merging anything (writes and deletes one throwaway object):
|
||||
bootstrap/standup.sh --kubeconfig <kc> --expect-server <url> \
|
||||
--s3-endpoint https://<account-id>.r2.cloudflarestorage.com s3check
|
||||
3. Only then commit + merge, and follow infra/BACKUP-R2.md through the RESTORE.
|
||||
A backup that has never been restored is not a backup, and R-STATE-03 is currently PROVEN
|
||||
against MinIO only.
|
||||
NEXT
|
||||
+42
-10
@@ -9,17 +9,43 @@
|
||||
|
||||
## What this is (LIVE)
|
||||
|
||||
Flux `HelmRelease` (Velero chart 12.x / Velero 1.18.x) wired to the on-Proxmox **MinIO** S3 target:
|
||||
Flux `HelmRelease` (Velero chart 12.x / Velero 1.18.x) wired to whatever `${S3_BACKUP_ENDPOINT}` names.
|
||||
The target is a **cluster-vars knob, not a constant**:
|
||||
|
||||
- **Endpoint:** `http://192.168.1.9:9000` (plain HTTP, **path-style**, region `us-east-1` placeholder)
|
||||
- **BucketStorageLocation `default`:** provider `aws`, bucket `velero`, `s3ForcePathStyle=true`.
|
||||
- **AWS plugin** (`velero/velero-plugin-for-aws`) loaded as an init container.
|
||||
- **Endpoint:** `${S3_BACKUP_ENDPOINT}` — dev has been running against the on-Proxmox **MinIO**
|
||||
`http://192.168.1.9:9000`; as of **2026-09-05** the authored target is **Cloudflare R2**
|
||||
(`https://<account-id>.r2.cloudflarestorage.com`). The switch-on procedure, the R2 gotchas and the
|
||||
honest proven/unproven split all live in **[`infra/BACKUP-R2.md`](../../infra/BACKUP-R2.md)**.
|
||||
Always **path-style**; region `${S3_BACKUP_REGION}` (MinIO ignores it, R2 requires exactly `auto`).
|
||||
- **BucketStorageLocation `default`:** provider `aws`, bucket `velero`, prefix
|
||||
`${VELERO_BACKUP_PREFIX}` (dev/prod — both clusters share one R2 endpoint and are isolated by path
|
||||
only), `s3ForcePathStyle=true`, **`checksumAlgorithm: ""`**.
|
||||
That last one is load-bearing: `velero-plugin-for-aws` v1.14.2 defaults it to `CRC32` and R2 rejects
|
||||
AWS checksums with HTTP 400 `InvalidArgument`. Do not remove it, and re-check it on every plugin bump.
|
||||
- **AWS plugin** (`velero/velero-plugin-for-aws:v1.14.2`) loaded as an init container.
|
||||
- **File System Backup (kopia):** `deployNodeAgent=true`, `uploaderType=kopia`, `snapshotsEnabled=false`
|
||||
— storage-agnostic PVC backup, **no** CSI snapshot / data-mover.
|
||||
— storage-agnostic PVC backup, **no** CSI snapshot / data-mover. This path uploads through kopia's
|
||||
**own** S3 client, which does not read `checksumAlgorithm`: metadata compatibility and volume-data
|
||||
compatibility with a given object store are two separate questions.
|
||||
- **Credentials:** `velero-cloud-credentials` — a SOPS Secret (`velero-cloud.sops.yaml`) holding the
|
||||
`cloud` INI key, the **scoped** MinIO key limited to the `velero`+`cnpg` buckets. Kept in sync with
|
||||
the staging source of truth `platform/backup/minio-credentials.sops.yaml` and the Proxmox host
|
||||
ansible-vault.
|
||||
`cloud` INI key, the **scoped** key limited to the `velero`+`cnpg` buckets. Four Secrets across three
|
||||
namespaces hold this one key; `platform/backup/set-r2-credentials.sh` writes all four at once with
|
||||
`sops set` (no plaintext ever on disk) and proves they agree.
|
||||
`platform/backup/minio-credentials.sops.yaml` retains the **pre-migration MinIO** key — that is
|
||||
deliberate, it is how the pre-migration backups stay readable.
|
||||
|
||||
## Proving the target before trusting it
|
||||
|
||||
```bash
|
||||
bootstrap/standup.sh --kubeconfig <kc> --expect-server <url> \
|
||||
[--s3-endpoint <candidate>] s3check
|
||||
```
|
||||
|
||||
`s3check` runs a throwaway in-cluster pod per credential and proves credential load + authentication,
|
||||
bucket existence and listability under this environment's prefix, and a real
|
||||
**PUT → GET → byte-compare → DELETE → confirm-gone** round trip — plus whether the R2 checksum trap is
|
||||
present. It fails closed and loud; it never reports "unknown, carrying on". Pod egress is the only
|
||||
reading that counts (prod sits on pfSense's WAN, the MinIO inside its LAN).
|
||||
|
||||
Reconciled by its **own non-gating** Flux Kustomization `clusters/base/backup.yaml`
|
||||
(`dependsOn: platform-config`, `wait:false`) — a backup controller must never gate the platform graph.
|
||||
@@ -46,7 +72,13 @@ PITR to the `cnpg` bucket — see the DB manifests.)
|
||||
|
||||
## Post-deploy (PM)
|
||||
|
||||
- `velero backup-location get` → `default` Available.
|
||||
- `velero backup create --from-schedule daily` → objects land under `s3://velero/`.
|
||||
- `velero backup-location get` → `default` Available. **Available only means the BSL could be read.**
|
||||
- `velero backup create --from-schedule daily` → Completed, then read `velero backup logs <name>` and
|
||||
look for `PodVolumeBackup` failures specifically (see the kopia note above). Objects land under
|
||||
`s3://velero/${VELERO_BACKUP_PREFIX}/`.
|
||||
- DR drill (R-STATE-03, Phase-4 gate): `velero restore create --from-backup <name>` into a scratch
|
||||
namespace and verify PVC data restores via the node-agent.
|
||||
|
||||
**R-STATE-03 is PROVEN against MinIO only.** A restore from R2 has never been executed in this repo,
|
||||
and Velero-against-R2 is unproven here — `infra/BACKUP-R2.md` §5 is the re-proof sequence and §7 is the
|
||||
honest list of what is and is not established.
|
||||
@@ -5,7 +5,10 @@ resources:
|
||||
- namespace.yaml
|
||||
- helmrepository.yaml
|
||||
- helmrelease.yaml
|
||||
# Scoped MinIO S3 credential (the `cloud` INI key the AWS plugin reads) — SOPS-encrypted, decrypted
|
||||
# per-env by the `velero` Flux Kustomization (clusters/dev). Kept in sync with the staging source of
|
||||
# truth platform/backup/minio-credentials.sops.yaml + the Proxmox host ansible-vault.
|
||||
# Scoped S3 credential for ${S3_BACKUP_ENDPOINT} (the `cloud` INI key the AWS plugin reads) —
|
||||
# SOPS-encrypted, decrypted per-env by the `velero` Flux Kustomization (clusters/dev).
|
||||
# ONE key lives in FOUR Secrets across three namespaces (this one + the three CNPG
|
||||
# *-db-backup-credentials). platform/backup/set-r2-credentials.sh writes all four with `sops set`
|
||||
# and proves they agree; do not hand-edit one of them. platform/backup/minio-credentials.sops.yaml
|
||||
# retains the PRE-MIGRATION MinIO key on purpose. See infra/BACKUP-R2.md.
|
||||
- velero-cloud.sops.yaml
|
||||
@@ -2,11 +2,20 @@
|
||||
# doc pointer, kept for parity with the other components' layout).
|
||||
#
|
||||
# LIVE wiring (see helmrelease.yaml):
|
||||
# - provider aws + velero-plugin-for-aws (init container) → MinIO S3 ${S3_BACKUP_ENDPOINT} (dev: http://192.168.1.9:9000)
|
||||
# (path-style, bucket `velero`, region us-east-1 placeholder).
|
||||
# - provider aws + velero-plugin-for-aws v1.14.2 (init container) → the S3 endpoint
|
||||
# ${S3_BACKUP_ENDPOINT} from cluster-vars. That is a KNOB, not MinIO: dev has been running against
|
||||
# the on-Proxmox MinIO http://192.168.1.9:9000, and as of 2026-09-05 the authored target is
|
||||
# Cloudflare R2 (https://<account-id>.r2.cloudflarestorage.com) — see infra/BACKUP-R2.md.
|
||||
# Always path-style; bucket `velero`; region ${S3_BACKUP_REGION} (MinIO ignores it, R2 demands
|
||||
# exactly `auto`); per-environment prefix ${VELERO_BACKUP_PREFIX}.
|
||||
# - checksumAlgorithm: "" — REQUIRED for R2 and harmless for MinIO. The plugin defaults it to CRC32,
|
||||
# which R2 rejects with HTTP 400 InvalidArgument. Do not drop it; see helmrelease.yaml.
|
||||
# - deployNodeAgent:true + uploaderType:kopia → File System Backup of PVC data (no CSI data-mover).
|
||||
# NOTE this path uses kopia's OWN S3 client, which does not read checksumAlgorithm — backup
|
||||
# metadata and volume data are two separate S3-compatibility questions (infra/BACKUP-R2.md §7).
|
||||
# - snapshotsEnabled:false → no VolumeSnapshotLocation.
|
||||
# - credentials.existingSecret: velero-cloud-credentials (SOPS `cloud` INI key, ns velero).
|
||||
# - two daily Schedules (${VELERO_BACKUP_SCHEDULE} / ttl ${VELERO_RETENTION}): the whole cluster
|
||||
# via FSB, with monitoring+unifi backed up RESOURCES-ONLY (their volume data is reconstructable).
|
||||
# DR drill (tested restore) remains a Phase-4 acceptance gate (R-STATE-03) — the PM runs it post-deploy.
|
||||
# It is PROVEN against MinIO only; a restore from R2 has never been executed (infra/BACKUP-R2.md §5).
|
||||
Reference in new issue
Block a user