From 2e259a808fd995fa6196de2da07fd4a1ad5ac0a7 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 17:12:37 -0500 Subject: [PATCH 1/2] fix(velero): prod backups could not write metadata to R2, and opt-out FSB would have shipped 8.6 TiB MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Velero was resumed on prod (live `flux resume`; suspension was never in git) and its BackupStorageLocation validated Available against Cloudflare R2 on the first try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "". Backups still failed, and a second, unrelated hazard was armed. Both are fixed here; both were measured on prod, not reasoned about. 1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all. It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty `x-amz-tagging` header on every PUT. R2 does not implement object tagging in any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced it on the write of velero/backups//velero-backup.json: kopia uploaded the volume data fine (it uses its own S3 client), then the backup went phase=Failed with no metadata object — i.e. unrestorable. No config avoids it; `config.tagging` with a real value sends a tag set R2 also rejects. Upstream PR #309 makes the field conditional and landed on release-1.14 after v1.14.2 was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it, with the exit condition (v1.14.3 GA) stated at the pin. 2. The opt-out FSB schedule would have uploaded the NFS bulk tier. Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring and unifi. Since 2026-09-11 prod also carries four static NFS PVs — plex-media, sonarr-media and radarr-media all bound to the SAME export /mnt/Storage/plex (8.32 TiB measured via zfs list), plus nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained 720h. Namespace exclusion cannot express the fix: those four namespaces also hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose loss is unrecoverable. So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy (ConfigMap velero-volume-policies) skips any volume whose source is NFS, and schedule (1) references it. That tests the property that makes a volume bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list the next PV would slip past. The per-pod backup-volumes-excludes annotation was rejected for failing OPEN — a new NFS mount would be backed up in full until someone remembered. PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with defaultVolumesToFsBackup:true plus this policy produced exactly ONE PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes, Completed). The NFS `legacy-archive` volume produced none. Residual gap, stated rather than hidden: the rule matches NFS only. A future bulk volume on another out-of-cluster source needs its own condition. Co-Authored-By: Claude Opus 5 --- platform/velero/helmrelease.yaml | 45 +++++++++++++++++++++++-- platform/velero/kustomization.yaml | 3 ++ platform/velero/values.yaml | 8 +++++ platform/velero/volume-policies.yaml | 50 ++++++++++++++++++++++++++++ 4 files changed, 103 insertions(+), 3 deletions(-) create mode 100644 platform/velero/volume-policies.yaml diff --git a/platform/velero/helmrelease.yaml b/platform/velero/helmrelease.yaml index f41a0da..975d0d4 100644 --- a/platform/velero/helmrelease.yaml +++ b/platform/velero/helmrelease.yaml @@ -65,11 +65,30 @@ spec: # acceptable: Velero's CRDs are stable, and this is far less fragile than pinning a shell-bearing # kubectl mirror (rancher/kubectl is distroless → no /bin/sh for the hook's copy-binary step). --- upgradeCRDs: false - # --- AWS object-store plugin (BSL provider=aws → MinIO S3-compatible), loaded as an init container. - # Image minor MUST track the Velero minor (chart 12.x = Velero 1.18 → plugin v1.14.x). --- + # --- AWS object-store plugin (BSL provider=aws → the S3-compatible target), loaded as an init + # container. Image minor MUST track the Velero minor (chart 12.x = Velero 1.18 → plugin v1.14.x). + # + # *** WHY AN -rc TAG, AND WHY IT CANNOT BE v1.14.2 (measured on prod, 2026-09-12). *** + # v1.14.2 sets the S3 PutObject `Tagging` field UNCONDITIONALLY: + # Tagging: aws.String(o.tagging) // o.tagging == "" when config.tagging is unset + # so the SDK emits `x-amz-tagging: ` (empty) on EVERY PutObject. Cloudflare R2 does not implement + # object tagging AT ALL — the S3 compatibility matrix lists PutObject's `x-amz-tagging`, plus + # Put/Get/DeleteObjectTagging, as unimplemented — so R2 answers HTTP 501: + # NotImplemented: Header 'x-amz-tagging' with value '' not implemented + # Observed live on prod twice, on the write of `velero/backups//velero-backup.json`. The + # Backup lands its kopia volume DATA (the node-agent's kopia client sends no such header) and then + # goes phase=Failed with NO metadata object — an unrestorable backup. There is NO config that + # avoids this: setting `config.tagging` to a real value only sends a tag set R2 also rejects. + # The fix is upstream PR #309 ("Only set PutObject Tagging when tags are configured"), which makes + # the field conditional (`if o.tagging != ""`). It landed on release-1.14 AFTER v1.14.2 was cut, so + # the earliest build carrying it is v1.14.3-rc.2. There is no GA release with the fix yet. + # EXIT CONDITION: move to v1.14.3 the moment it ships GA (Renovate will offer it). Do NOT + # "clean this up" back to v1.14.2 — that silently breaks every prod backup's metadata. + # Harmless on dev's MinIO, which ignores the header either way, so it is stated unconditionally + # rather than becoming a fifth per-environment knob. --- initContainers: - name: velero-plugin-for-aws - image: velero/velero-plugin-for-aws:v1.14.2 + image: velero/velero-plugin-for-aws:v1.14.3-rc.2 imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /target @@ -128,6 +147,23 @@ spec: schedules: # (1) The whole cluster EXCEPT monitoring+unifi: all resources + all pod-volume data via kopia. # defaultVolumesToFsBackup:true is the OPT-OUT model — nothing with real data is forgotten. + # + # THE BULK TIER IS OPTED OUT BY VOLUME TYPE, NOT BY NAMESPACE (added 2026-09-12). Opt-out + # + the external NFS bulk tier is a 8.6 TiB collision: prod carries four static NFS PVs + # (plex-media, sonarr-media, radarr-media — all three bound to the SAME export + # /mnt/Storage/plex, measured 8.32 TiB — plus nextcloud-legacy-archive at 270 GiB), and + # their namespaces ALSO hold real Longhorn config/data PVCs that MUST be backed up. So + # namespace granularity cannot express this: moving plex/sonarr/radarr/nextcloud into + # schedule (2) would drop their real config PVCs, and leaving them here uploaded the media + # library to R2 three times over (kopia dedups the STORED copy, not the three NFS reads). + # `resourcePolicy` below therefore skips volumes whose source is NFS, whatever they are + # called and whoever adds the next one — the bulk tier IS the NFS export (R-STATE-04), so + # the rule tests the property rather than a name list that a future PV would slip past. + # Media on that ZFS pool has its own protection and is reconstructable; it is deliberately + # NOT cluster-DR scope. See platform/velero/volume-policies.yaml. + # RESIDUAL GAP, stated rather than hidden: the rule matches NFS only. A future bulk volume + # on some OTHER out-of-cluster source (iSCSI, CSI-attached object gateway) would be picked + # up by opt-out again and needs its own condition added there. daily: disabled: false schedule: "${VELERO_BACKUP_SCHEDULE}" @@ -137,6 +173,9 @@ spec: storageLocation: default snapshotVolumes: false defaultVolumesToFsBackup: true + resourcePolicy: + kind: configmap + name: velero-volume-policies excludedNamespaces: - monitoring # volume DATA excluded here; resources captured by schedule (2) - unifi diff --git a/platform/velero/kustomization.yaml b/platform/velero/kustomization.yaml index feacb73..b812ac7 100644 --- a/platform/velero/kustomization.yaml +++ b/platform/velero/kustomization.yaml @@ -5,6 +5,9 @@ resources: - namespace.yaml - helmrepository.yaml - helmrelease.yaml + # Volume policies for schedule (1)'s opt-out FSB: skip the external NFS bulk tier by SOURCE TYPE. + # Must exist before the Schedule references it by name (same Kustomization, so one apply). + - volume-policies.yaml # --- PER-ENVIRONMENT `cloud` INI credential (split 2026-09-07) ----------------------------------- # BOTH Secrets are applied to BOTH clusters; the HelmRelease picks ONE by name via # `existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}` (cluster-vars: dev "", diff --git a/platform/velero/values.yaml b/platform/velero/values.yaml index e73db5e..d6de81b 100644 --- a/platform/velero/values.yaml +++ b/platform/velero/values.yaml @@ -9,6 +9,14 @@ # sharing one bucket with CNPG/barman under a disjoint prefix. # - config.checksumAlgorithm: "" — MANDATORY for R2, which rejects the AWS SDK's # x-amz-checksum-* headers with HTTP 400 InvalidArgument. Harmless against MinIO. +# - velero-plugin-for-aws PINNED TO v1.14.3-rc.2 — also MANDATORY for R2. v1.14.2 sends an empty +# `x-amz-tagging` header on every PutObject, which R2 answers with HTTP 501 NotImplemented, so +# every Backup lands its kopia volume data and then FAILS writing velero-backup.json. Upstream +# PR #309 makes the field conditional; rc.2 is the earliest build carrying it. Move to v1.14.3 +# when it ships GA. Measured on prod 2026-09-12. +# - resourcePolicy on the FSB schedule -> ConfigMap velero-volume-policies (volume-policies.yaml): +# skips volumes whose source is NFS, so the 8.6 TiB external bulk tier is not uploaded nightly +# while the same namespaces' real Longhorn PVCs still are. Proven on prod 2026-09-12. # - deployNodeAgent:true + uploaderType:kopia → File System Backup of PVC data (no CSI data-mover). # - snapshotsEnabled:false → no VolumeSnapshotLocation. # - credentials.existingSecret: velero-cloud-credentials (SOPS `cloud` INI key, ns velero). diff --git a/platform/velero/volume-policies.yaml b/platform/velero/volume-policies.yaml new file mode 100644 index 0000000..1848efc --- /dev/null +++ b/platform/velero/volume-policies.yaml @@ -0,0 +1,50 @@ +--- +# Velero VOLUME POLICIES — the opt-out escape hatch for the external bulk tier (R-STATE-04). +# +# WHY THIS EXISTS. platform/velero/helmrelease.yaml schedule (1) runs +# `defaultVolumesToFsBackup: true` — the OPT-OUT model, so no volume with real data is ever +# forgotten. That model collides with the NFS bulk tier prod grew on 2026-09-11: +# +# plex-media 9Ti claim -> 192.168.2.102:/mnt/Storage/plex (8.32 TiB measured) +# sonarr-media 9Ti claim -> the SAME export +# radarr-media 9Ti claim -> the SAME export +# nextcloud-legacy-archive 270Gi -> 192.168.2.102:/mnt/Storage/nextcloud +# +# Left to opt-out, one nightly run reads that library THREE TIMES over NFS and pushes ~8.6 TiB at +# Cloudflare R2, every night, held for ${VELERO_RETENTION} (prod: 720h). kopia dedups the STORED +# bytes, not the reads. That is not DR; it is a self-inflicted outage plus a recurring bill. +# +# WHY NOT namespace exclusion. plex / sonarr / radarr / nextcloud each ALSO hold a real Longhorn +# PVC (plex-config 8Gi, sonarr-config 5Gi, radarr-config 5Gi, nextcloud-data 20Gi) whose loss is +# unrecoverable. Excluding those namespaces — the shape schedule (2) uses for monitoring+unifi — +# would silently drop exactly the data that matters. The opt-out has to be per VOLUME. +# +# WHY NOT the per-pod `backup.velero.io/backup-volumes-excludes` annotation. It works, but it fails +# OPEN: the next NFS PV someone mounts is backed up in full until a human remembers to annotate its +# pod, and nothing tells them. A volume policy tests the PROPERTY that makes a volume bulk-tier — +# its source is the out-of-cluster NFS export — so it covers volumes that do not exist yet. +# +# `nfs: {}` is the documented "match any NFS volume source" condition (server/path may be given to +# narrow it; deliberately not, so a second export is covered too). First matching rule wins. +# +# PROVEN, not assumed (prod, 2026-09-12, velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud +# with defaultVolumesToFsBackup:true and this policy produced exactly ONE PodVolumeBackup — +# `data` (nextcloud-data, Longhorn, 875,417,102 bytes, Completed). The NFS `legacy-archive` volume +# produced NO PodVolumeBackup at all. +# +# Referenced by name from schedule (1)'s template (`resourcePolicy: {kind: configmap, name: ...}`). +# Applied to BOTH environments: dev has no NFS PVs today, so the rule is a no-op there, and it is +# already correct if dev ever grows one. +apiVersion: v1 +kind: ConfigMap +metadata: + name: velero-volume-policies + namespace: velero +data: + policies.yaml: | + version: v1 + volumePolicies: + - conditions: + nfs: {} + action: + type: skip From 2d5171e064c8ff2c6ac9fedd73e2c73faf201672 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 17:17:06 -0500 Subject: [PATCH 2/2] =?UTF-8?q?feat(velero):=20scrape=20the=20backup=20con?= =?UTF-8?q?troller=20=E2=80=94=20three=20critical=20alerts=20could=20never?= =?UTF-8?q?=20fire?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit platform/monitoring/alerts.yaml has shipped VeleroBackupFailure (critical), VeleroBackupPartialFailure and VeleroNoRecentBackup since the observability wave. None of them could ever have fired. The velero chart defaults `metrics.serviceMonitor.enabled: false`, and prod confirms it live: `kubectl -n velero get servicemonitor` returns no resources, so velero_backup_failure_total and velero_backup_last_successful_timestamp were never scraped. A nightly backup could fail — or silently stop running — for weeks with nothing to say so. That is exactly what R-STATE-03 is for, which makes the scrape part of the backup path rather than an observability extra. Both flags verified against chart 12.1.0: `helm template` emits neither object without the monitoring.coreos.com/v1 capability, and emits both PodMonitor and ServiceMonitor with it — so the chart gates on capability detection AND `enabled`, and helm-controller supplies real capabilities from the API server. No additionalLabels are needed: platform/monitoring already sets serviceMonitorSelectorNilUsesHelmValues and podMonitorSelectorNilUsesHelmValues to false, so Prometheus selects every monitor regardless of labels. Independent of the R2/bulk-tier fix in the previous commit; droppable on its own if the PM wants to sequence it separately. Co-Authored-By: Claude Opus 5 --- platform/velero/helmrelease.yaml | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/platform/velero/helmrelease.yaml b/platform/velero/helmrelease.yaml index 975d0d4..27a794b 100644 --- a/platform/velero/helmrelease.yaml +++ b/platform/velero/helmrelease.yaml @@ -198,6 +198,26 @@ spec: includedNamespaces: - monitoring - unifi + # --- SCRAPE THE BACKUP CONTROLLER, or its failures are silent (added 2026-09-12) ------------ + # platform/monitoring/alerts.yaml already ships VeleroBackupFailure (critical), + # VeleroBackupPartialFailure and VeleroNoRecentBackup — and NONE of them could ever fire, + # because the chart defaults `metrics.serviceMonitor.enabled: false` and no ServiceMonitor + # existed in ns velero (verified live on prod, 2026-09-12: `get servicemonitor -n velero` → + # no resources). The series velero_backup_failure_total / velero_backup_last_successful_timestamp + # were never scraped, so a nightly backup could fail — or stop running — indefinitely with no + # alert. That is the precise failure mode R-STATE-03 exists to prevent, so the scrape is part of + # the backup path, not an observability nice-to-have. + # The chart gates these on BOTH `enabled` and capability detection of monitoring.coreos.com/v1 + # (verified: `helm template` emits neither object without --api-versions, and emits both with + # it), so `enabled: true` is required and sufficient — helm-controller supplies real capabilities + # from the API server. No additionalLabels needed: platform/monitoring sets + # serviceMonitorSelectorNilUsesHelmValues / podMonitorSelectorNilUsesHelmValues false, so + # Prometheus selects every ServiceMonitor/PodMonitor regardless of labels. + metrics: + serviceMonitor: + enabled: true # velero server: backup/restore success+failure counters + nodeAgentPodMonitor: + enabled: true # node-agent: kopia FSB (PodVolumeBackup) progress + failures # Modest home-dev footprint (mirror the other platform components). resources: requests: