home-network-k3s-impl/platform/velero/helmrelease.yaml

---
# Velero — off-cluster backup/DR of cluster resources + PVC data (ADR-0003, R-STATE-03). LIVE.
#
# TARGET: the S3 endpoint ${S3_BACKUP_ENDPOINT} (cluster-vars) at
# s3://${VELERO_BACKUP_BUCKET}/${VELERO_BACKUP_PREFIX}, in region ${S3_BACKUP_REGION}. Four knobs,
# because the two environments' object stores differ in shape as well as address:
#   dev   http://192.168.1.9:9000  bucket `velero`, NO prefix, region us-east-1  (on-Proxmox MinIO,
#                                  plain HTTP, path-style; MinIO ignores the region)
#   prod  https://<acct>.r2.cloudflarestorage.com  bucket `k3s-prod-backup`, prefix `velero`,
#                                  region `auto`   (Cloudflare R2; prod cannot route to that MinIO,
#                                  and R2 demands exactly `auto`)
# Prod's bucket is SHARED with CNPG/barman (the operator created one bucket), so the `velero` prefix
# is what keeps this controller's retention sweep away from the Postgres WAL stream. See the
# "OBJECT-STORE TENANCY" block in clusters/base/cluster-vars.yaml.
#
# WHY THIS SHAPE:
#  - provider aws + velero-plugin-for-aws (init container) drives the S3-compatible MinIO target.
#  - File System Backup (kopia) via the node-agent DaemonSet — storage-agnostic PVC backup that does
#    NOT depend on CSI VolumeSnapshots or a data-mover (the simplest path that still captures the
#    Longhorn PVC *data*). snapshotsEnabled:false → no VolumeSnapshotLocation is created/needed.
#  - Credentials: the `cloud` INI key of the SOPS Secret velero-cloud-credentials (ns velero). ONE
#    Secret serves both environments, and it now carries the Cloudflare R2 API token (written by
#    platform/backup/set-r2-credentials.sh, 2026-09-07) rather than the scoped MinIO key. That is
#    safe only because the R2 token is bucket-scoped to k3s-prod-backup, which dev never names:
#    dev's Velero would fail loudly against its MinIO endpoint with an R2 key, it could not
#    silently write somewhere unexpected. Rotating either target means re-running that script.
#
# VOLUME OPT-OUT (per the sizing decision) is implemented with TWO schedules (a single Backup has one
# cluster-wide defaultVolumesToFsBackup, so per-namespace volume policy needs two): see `schedules`.
#
# NON-GATING TIER: reconciled by clusters/base/backup.yaml (dependsOn platform-config, wait:false).
# A backup controller must NEVER gate the platform graph (mirror headlamp/oauth2-proxy).
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: velero
  namespace: velero
spec:
  interval: 30m
  chart:
    spec:
      chart: velero
      version: ">=12.0.0 <13.0.0"   # Velero v1.18.x line (k8s-1.36 catch-up). CRDs pre-applied server-side (upgradeCRDs:false). Renovate maintains
      sourceRef:
        kind: HelmRepository
        name: vmware-tanzu
        namespace: velero
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
  values:
    # --- CRD install strategy. The chart's pre-install `velero-upgrade-crds` HOOK job runs
    #     docker.io/bitnami/kubectl, whose public Docker Hub catalog Bitnami REMOVED (Aug 2025) —
    #     `docker.io/bitnami/kubectl:1.31` now 404s, leaving the hook in Init:ImagePullBackOff so Velero
    #     never installs. The hook exists ONLY to UPDATE CRDs on chart upgrades; it is NOT how CRDs are
    #     first installed. The chart ships all 13 Velero CRDs (backups/restores/backupstoragelocations/
    #     schedules/podvolumebackups/…) in its native `crds/` directory, which Helm applies
    #     UNCONDITIONALLY on install — before hooks — regardless of this flag.
    #     Disabling the hook therefore drops the fragile external-kubectl-image dependency entirely while
    #     the CRDs still install via Helm's crds/ handling. Trade-off: Helm never manages crds/ after the
    #     initial install, so CRD *schema upgrades* on future chart bumps become a manual/Renovate step —
    #     acceptable: Velero's CRDs are stable, and this is far less fragile than pinning a shell-bearing
    #     kubectl mirror (rancher/kubectl is distroless → no /bin/sh for the hook's copy-binary step). ---
    upgradeCRDs: false
    # --- AWS object-store plugin (BSL provider=aws → the S3-compatible target), loaded as an init
    #     container. Image minor MUST track the Velero minor (chart 12.x = Velero 1.18 → plugin v1.14.x).
    #
    # *** WHY AN -rc TAG, AND WHY IT CANNOT BE v1.14.2 (measured on prod, 2026-09-12). ***
    # v1.14.2 sets the S3 PutObject `Tagging` field UNCONDITIONALLY:
    #     Tagging: aws.String(o.tagging)            // o.tagging == "" when config.tagging is unset
    # so the SDK emits `x-amz-tagging: ` (empty) on EVERY PutObject. Cloudflare R2 does not implement
    # object tagging AT ALL — the S3 compatibility matrix lists PutObject's `x-amz-tagging`, plus
    # Put/Get/DeleteObjectTagging, as unimplemented — so R2 answers HTTP 501:
    #     NotImplemented: Header 'x-amz-tagging' with value '' not implemented
    # Observed live on prod twice, on the write of `velero/backups/<name>/velero-backup.json`. The
    # Backup lands its kopia volume DATA (the node-agent's kopia client sends no such header) and then
    # goes phase=Failed with NO metadata object — an unrestorable backup. There is NO config that
    # avoids this: setting `config.tagging` to a real value only sends a tag set R2 also rejects.
    # The fix is upstream PR #309 ("Only set PutObject Tagging when tags are configured"), which makes
    # the field conditional (`if o.tagging != ""`). It landed on release-1.14 AFTER v1.14.2 was cut, so
    # the earliest build carrying it is v1.14.3-rc.2. There is no GA release with the fix yet.
    # EXIT CONDITION: move to v1.14.3 the moment it ships GA (Renovate will offer it). Do NOT
    # "clean this up" back to v1.14.2 — that silently breaks every prod backup's metadata.
    # Harmless on dev's MinIO, which ignores the header either way, so it is stated unconditionally
    # rather than becoming a fifth per-environment knob. ---
    initContainers:
      - name: velero-plugin-for-aws
        image: velero/velero-plugin-for-aws:v1.14.3-rc.2
        imagePullPolicy: IfNotPresent
        volumeMounts:
          - mountPath: /target
            name: plugins
    # --- File System Backup (kopia): the node-agent DaemonSet mounts each pod's volumes and copies
    #     the data into S3. Storage-agnostic; no CSI-snapshot / data-mover wiring. ---
    deployNodeAgent: true
    snapshotsEnabled: false        # no VolumeSnapshotLocation — kopia FSB is the volume-data path
    configuration:
      uploaderType: kopia
      backupStorageLocation:
        - name: default
          provider: aws
          # Bucket and prefix are BOTH knobs (cluster-vars), because the two environments' object
          # stores are shaped differently. Dev's MinIO has a dedicated `velero` bucket and dev's
          # backups sit at its ROOT, so dev renders bucket=velero with an EMPTY prefix — and the
          # chart omits `objectStorage.prefix` entirely when it is empty (verified, chart 12.1.0),
          # leaving dev's live BackupStorageLocation byte-unchanged. Prod's operator created ONE R2
          # bucket shared with CNPG/barman, so prod renders bucket=k3s-prod-backup with prefix
          # `velero` — a disjoint top-level prefix from barman's. Getting this wrong is not a
          # cosmetic error: Velero ADOPTS every backup it finds under its location and TTL-DELETES
          # it on retention, so a prefix that overlaps another tenant deletes that tenant's data.
          bucket: ${VELERO_BACKUP_BUCKET}
          prefix: "${VELERO_BACKUP_PREFIX}"
          default: true
          config:
            # Region is a knob, not a constant: S3 clients require SOME region, MinIO ignores
            # whatever it is, and Cloudflare R2 requires exactly `auto`. Hardcoding `us-east-1`
            # was correct-by-accident and broke the moment the target stopped being MinIO.
            region: ${S3_BACKUP_REGION}
            s3Url: ${S3_BACKUP_ENDPOINT}      # cluster-vars; dev MinIO (http) or prod R2 (https)
            s3ForcePathStyle: "true"          # MinIO is path-style; R2 accepts path-style too
            # *** REQUIRED FOR CLOUDFLARE R2 — DO NOT DROP THIS LINE. ***
            # velero-plugin-for-aws v1.14.2 bundles an AWS SDK with flexible checksums ON by
            # default, so it sends `x-amz-sdk-checksum-algorithm` / `x-amz-checksum-crc32` on every
            # PUT. R2 REJECTS those headers with HTTP 400 InvalidArgument, which would fail every
            # backup AND every kopia blob upload. The empty string is the documented opt-out, and
            # it must be PRESENT-but-empty rather than absent — the plugin branches on the key
            # existing, and the chart does emit an empty value (verified, chart 12.1.0). Harmless
            # against MinIO, which ignores it, so it is stated unconditionally rather than as a
            # fourth knob. If backups start failing with InvalidArgument after a plugin bump, look
            # here first.
            checksumAlgorithm: ""
    credentials:
      # PER-ENVIRONMENT credential (2026-09-07). dev renders velero-cloud-credentials (the scoped
      # MinIO key); prod renders velero-cloud-credentials-prod (the Cloudflare R2 token). BOTH
      # Secrets are applied to BOTH clusters by platform/velero/kustomization.yaml — the suffix
      # only chooses which one this Deployment mounts. cluster-vars: BACKUP_CREDENTIALS_SUFFIX,
      # dev "" | prod "-prod". A name that resolves to a MISSING Secret does not degrade the
      # backup, it wedges the pod in ContainerCreating (the volume never mounts), which is why
      # the base default is the empty suffix — the unsuffixed Secret always exists.
      existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}   # SOPS Secret carrying the `cloud` INI key (ns velero)
    # --- Daily backups. TWO schedules implement the opt-out volume policy: capture every PVC's data
    #     by default (opt-out), but skip the two reconstructable heavy volumes' DATA while still
    #     capturing their Kubernetes resources. ---
    schedules:
      # (1) The whole cluster EXCEPT monitoring+unifi: all resources + all pod-volume data via kopia.
      #     defaultVolumesToFsBackup:true is the OPT-OUT model — nothing with real data is forgotten.
      #
      #     THE BULK TIER IS OPTED OUT BY VOLUME TYPE, NOT BY NAMESPACE (added 2026-09-12). Opt-out
      #     + the external NFS bulk tier is a 8.6 TiB collision: prod carries four static NFS PVs
      #     (plex-media, sonarr-media, radarr-media — all three bound to the SAME export
      #     /mnt/Storage/plex, measured 8.32 TiB — plus nextcloud-legacy-archive at 270 GiB), and
      #     their namespaces ALSO hold real Longhorn config/data PVCs that MUST be backed up. So
      #     namespace granularity cannot express this: moving plex/sonarr/radarr/nextcloud into
      #     schedule (2) would drop their real config PVCs, and leaving them here uploaded the media
      #     library to R2 three times over (kopia dedups the STORED copy, not the three NFS reads).
      #     `resourcePolicy` below therefore skips volumes whose source is NFS, whatever they are
      #     called and whoever adds the next one — the bulk tier IS the NFS export (R-STATE-04), so
      #     the rule tests the property rather than a name list that a future PV would slip past.
      #     Media on that ZFS pool has its own protection and is reconstructable; it is deliberately
      #     NOT cluster-DR scope. See platform/velero/volume-policies.yaml.
      #     RESIDUAL GAP, stated rather than hidden: the rule matches NFS only. A future bulk volume
      #     on some OTHER out-of-cluster source (iSCSI, CSI-attached object gateway) would be picked
      #     up by opt-out again and needs its own condition added there.
      daily:
        disabled: false
        schedule: "${VELERO_BACKUP_SCHEDULE}"
        useOwnerReferencesInBackup: false
        template:
          ttl: "${VELERO_RETENTION}"
          storageLocation: default
          snapshotVolumes: false
          defaultVolumesToFsBackup: true
          resourcePolicy:
            kind: configmap
            name: velero-volume-policies
          excludedNamespaces:
            - monitoring        # volume DATA excluded here; resources captured by schedule (2)
            - unifi
            # velero ITSELF (added 2026-09-19). Not a sizing decision — a correctness one. Velero runs
            # transient kopia repository-maintenance Jobs in its own namespace, one per BackupRepository
            # (`<ns>-default-kopia-maintain-job-<ts>`), each with `plugins` and `scratch` emptyDir volumes.
            # Under defaultVolumesToFsBackup:true the backup tries to fs-backup those volumes, the Job
            # completes, its pod disappears mid-backup, and the PodVolumeBackup FAILS. Measured on prod:
            # velero-daily-20260919030041 came back PartiallyFailed with exactly 2 errors, both PVBs for
            # `plugins` + `scratch` on homepage-default-kopia-maintain-job-1789786837269-qhsvh, with 4 such
            # PVB failures accumulated. 3079/3079 items were backed up — nothing real was missed — but
            # PartiallyFailed does not advance velero_backup_last_successful_timestamp, so
            # VeleroNoRecentBackup fired CRITICAL over a backup that had in fact protected everything. A
            # backup alert that cries wolf nightly is how a real backup failure gets ignored.
            #
            # SAFE BY INSPECTION: ns velero holds NO PVCs (verified live), so there is no volume data here
            # to lose. Its RESOURCES still matter for DR (BSL, credentials, Backup CRs) and are captured by
            # schedule (2) — which is why `velero` is added there too.
            #
            # NARROWER THAN THE GENERAL RULE, deliberately. The general fix is a volume policy skipping
            # emptyDir outright (the property: emptyDir never holds durable state), covering a transient Job
            # in any namespace. That needs Velero's `volumeTypes` policy condition, unproven here on 1.18.1,
            # and a policies.yaml Velero cannot parse breaks EVERY backup — a far worse failure than the one
            # being fixed. All 4 observed failures were kopia-maintain pods in ns velero, so this uses only
            # mechanisms already proven in this repo. Revisit if the shape shows up elsewhere.
            - velero
      # (2) monitoring + unifi: RESOURCES ONLY (defaultVolumesToFsBackup:false → no volume data). Their
      #     PVCs are reconstructable and excluded from data backup per the sizing decision:
      #       - monitoring: the Prometheus TSDB PVC (ephemeral/reconstructable) and the Loki 72h-log PVC
      #         (Grafana/Alertmanager are emptyDir — no PVC). Grafana dashboards are GitOps-provisioned.
      #       - unifi: the ~7.4 GiB KubeVirt VM disk (huge + reconstructable; UniFi is protected by its
      #         own config export, not this disk).
      #     Backing up their K8s objects (VM/DataVolume/Services/HelmRelease-owned CRs) still aids DR.
      daily-resources-only:
        disabled: false
        schedule: "${VELERO_BACKUP_SCHEDULE}"
        useOwnerReferencesInBackup: false
        template:
          ttl: "${VELERO_RETENTION}"
          storageLocation: default
          snapshotVolumes: false
          defaultVolumesToFsBackup: false
          includedNamespaces:
            - monitoring
            - unifi
            - velero              # resources only (BSL/credentials/Backup CRs); see schedule (1)
    # --- SCRAPE THE BACKUP CONTROLLER, or its failures are silent (added 2026-09-12) ------------
    # platform/monitoring/alerts.yaml already ships VeleroBackupFailure (critical),
    # VeleroBackupPartialFailure and VeleroNoRecentBackup — and NONE of them could ever fire,
    # because the chart defaults `metrics.serviceMonitor.enabled: false` and no ServiceMonitor
    # existed in ns velero (verified live on prod, 2026-09-12: `get servicemonitor -n velero` →
    # no resources). The series velero_backup_failure_total / velero_backup_last_successful_timestamp
    # were never scraped, so a nightly backup could fail — or stop running — indefinitely with no
    # alert. That is the precise failure mode R-STATE-03 exists to prevent, so the scrape is part of
    # the backup path, not an observability nice-to-have.
    # The chart gates these on BOTH `enabled` and capability detection of monitoring.coreos.com/v1
    # (verified: `helm template` emits neither object without --api-versions, and emits both with
    # it), so `enabled: true` is required and sufficient — helm-controller supplies real capabilities
    # from the API server. No additionalLabels needed: platform/monitoring sets
    # serviceMonitorSelectorNilUsesHelmValues / podMonitorSelectorNilUsesHelmValues false, so
    # Prometheus selects every ServiceMonitor/PodMonitor regardless of labels.
    metrics:
      serviceMonitor:
        enabled: true            # velero server: backup/restore success+failure counters
      nodeAgentPodMonitor:
        enabled: true            # node-agent: kopia FSB (PodVolumeBackup) progress + failures
    # Modest home-dev footprint (mirror the other platform components).
    resources:
      requests:
        cpu: 50m
        memory: 128Mi
      limits:
        memory: 512Mi
    nodeAgent:
      resources:
        requests:
          cpu: 50m
          memory: 128Mi
        limits:
          memory: 512Mi