feat(backup): wire Velero + CNPG Barman to the live MinIO S3 target

Author the GitOps side of off-cluster backups against the on-Proxmox MinIO (http://192.168.1.9:9000, path-style; buckets velero + cnpg). Velero (R-STATE-03): - HelmRelease (chart 8.x / Velero 1.15) + velero-plugin-for-aws init container + node-agent kopia File System Backup (deployNodeAgent, uploaderType=kopia, snapshotsEnabled=false). BSL default -> bucket velero, s3ForcePathStyle. - Two daily Schedules: `daily` backs up the whole cluster's resources+PVC data (defaultVolumesToFsBackup opt-out) EXCEPT monitoring+unifi; `daily-resources-only` captures monitoring+unifi RESOURCES only (Prometheus TSDB, Loki 72h logs, and the ~7.4GiB UniFi VM disk are reconstructable — excluded from volume backup). - velero-cloud-credentials SOPS Secret (cloud INI key, ns velero). - Own non-gating Flux tier clusters/base/backup.yaml (dependsOn platform-config, wait:false) + clusters/dev SOPS decryption patch — mirrors headlamp. CNPG native Barman PITR (R-IDP-05 / R-STATE-03): - keycloak-db, nextcloud-db, semaphore-db gain spec.backup.barmanObjectStore (s3://cnpg/<cluster>, endpointURL MinIO, gzip WAL+data, 30d retention) + a daily ScheduledBackup (6-field cron, immediate) each. - Per-namespace scoped-key SOPS Secrets (<db>-backup-credentials), same-ns only. - render.py: emit-waves now recognizes <svc>-db-backup-credentials.sops.yaml as a DB-wave sidecar (+ self-test); wave-0 kustomization regenerated. Creds mirror platform/backup/minio-credentials.sops.yaml (scoped key k3s-backup), SOPS round-trip verified with the cluster age key. gate: green. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 22:37:17 -05:00
1 parent 1bee2554f8
commit d1248e6778
@@ -10,9 +10,11 @@ resources:
 - gitea.yaml
 - jackett.yaml
 - nextcloud-db-app.sops.yaml
+- nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
 - pihole.yaml
 - plex.yaml
 - qbittorrent.yaml
 - semaphore-db-app.sops.yaml
+- semaphore-db-backup-credentials.sops.yaml
 - semaphore-db.yaml
@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: nextcloud-db-backup-credentials
+    namespace: nextcloud-db
+type: Opaque
+stringData:
+    ACCESS_KEY_ID: ENC[AES256_GCM,data:gS533S1h2f5uWQ==,iv:XmcNCYdlkAIewn7Br8Kk4CHcZWbMQxJE0Mlge9/oGRY=,tag:+krC/C+EMQl5pYgwkuut1g==,type:str]
+    ACCESS_SECRET_KEY: ENC[AES256_GCM,data:NTcBY05zUy8ngV3phNF18FBHRCEKQGu2KQ2yQGu4YiuJasllznZ+YvCBn/3i0Z7/,iv:uXyenHmNqBRb5CC/ELgVbFXV/W59ZPOrSeGwIAi8Hz0=,tag:9ZRcKL0ijOPBi4QUbDZyoQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFMlBnNy94T2tlUUtaYTc0
+            Q0FKcDB3K2xPY2QvRWRyWTlTMnhWK0NlekFRCnpacHkwNHVONmRlR0xYNWQyVUhk
+            eUdVNTNVQjRsTm40cHBRWVFvbkJ0eTAKLS0tIGVJaE5ja0F1ejJOUkZKOWlXbXpD
+            ODAyK2g4bGlXeXV0Q1FMbXNiSEpUYncKvwviYgI/XjCBJG5kpS8tOHLTLsnXv1bl
+            b4E4rigNesmreSYxAyhGw9bgQNY6ba/F3Ecexrdx+iV/tYYc/VoDKA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-08-29T03:31:48Z"
+    mac: ENC[AES256_GCM,data:ks4i7zjrJQajKdncnnV5dD4Ap1Q1xdKyiQAiqhao2USXu9S+4flX2kzc9lv8lFG/dTiWvLu6Tqgmj46ssiqtd0E8DnrASNEuKz/Xlw1iCka2008C5BhE/k8oy84kXXDgEuo7GiYdhJU/mAaAnY4p10ZQkvTKddQMmSO65YRwRMo=,iv:skZoS+bhAOnLlY5R5OfZvFV8yCEiEKtLac4mVbdmZ2s=,tag:6T1JIroe5hI8RXWAyzEKLw==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4
@@ -68,7 +68,41 @@ spec:
      memory: 256Mi
    limits:
      memory: 512Mi
-  # --- Off-cluster DR backup (R-STATE-03) — OPERATOR HAND-OFF (OQ-BKP, shared with Velero, Phase 4) ---
-  # When the off-cluster backup target is chosen, add a Barman object store + a ScheduledBackup here
-  # (same hand-off as keycloak-db). Until then the DB is protected by 3-replica Longhorn + (Phase-4)
-  # Velero PVC backups of the namespace; the SD declares backup.policy daily-offcluster as the intent.
+  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
+  # Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
+  # backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
+  # (defaults to the Cluster name), so objects land under s3://cnpg/nextcloud-db/. Path-style HTTP.
+  # Credentials: the scoped key from the in-namespace SOPS Secret nextcloud-db-backup-credentials
+  # (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
+  backup:
+    retentionPolicy: "30d"
+    barmanObjectStore:
+      destinationPath: s3://cnpg
+      endpointURL: http://192.168.1.9:9000
+      s3Credentials:
+        accessKeyId:
+          name: nextcloud-db-backup-credentials
+          key: ACCESS_KEY_ID
+        secretAccessKey:
+          name: nextcloud-db-backup-credentials
+          key: ACCESS_SECRET_KEY
+      wal:
+        compression: gzip
+        maxParallel: 2
+      data:
+        compression: gzip
+---
+# Daily base backup of nextcloud-db (WAL archived continuously by the barmanObjectStore above). CNPG
+# cron is 6-field (sec min hour dom mon dow); immediate:true seeds the first base backup. 02:00 slot.
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: nextcloud-db-daily
+  namespace: nextcloud-db
+spec:
+  immediate: true
+  schedule: "0 0 2 * * *"
+  backupOwnerReference: self
+  cluster:
+    name: nextcloud-db
+  method: barmanObjectStore
@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: semaphore-db-backup-credentials
+    namespace: semaphore-db
+type: Opaque
+stringData:
+    ACCESS_KEY_ID: ENC[AES256_GCM,data:jmhu9U7Lyo0ugA==,iv:EyU67JLLO10za1tM/peju9tvBBINJ7ZDn6F9Zu0g6H8=,tag:9M/adv8Gdu3XNBJ/G4xfeA==,type:str]
+    ACCESS_SECRET_KEY: ENC[AES256_GCM,data:f9qW7BX5OQ2L7FqZ6pr6lqdEhTy+7xabLSMTnHZ/UYj1RnQ9asL+/HNkEWsMatJC,iv:hS2Q5l4EsvMBop/sBnld83tm2MhsTTI5N72BLGLMdtk=,tag:MwfC205a/djUA6HNqobcUw==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwU3VmQmFzQzUzTGFSVkxq
+            enlnVmRhTSt6V2xRMnF5ZXdpWlpsVDZZckg0CnY1eWRVNTVHMDhZaVpiczhLNGdW
+            Rjg3RWp4MWtUdnhjd0ZBUTZmWVY1WHMKLS0tIFlxYk1VcjBCR2Npbi93YklzUGF0
+            cGpweDl4VjNsNHM0Sm8zVDBWcGFJanMKpBBV9cnSZRVmIGjEi9YBMlbm4df253cS
+            osMRBCKizDIz1Q0Awq952y/L97ljUnSE6JWnlKTriHoGtLDct4mg3Q==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-08-29T03:31:48Z"
+    mac: ENC[AES256_GCM,data:uqtBwfubrUqaRN5r5qfMlITBvKxfbdyxZGDcZgyWtmDvX1OulKBvuOYBZaDW1MdIrq++nbKhocQK2PkuAmqZI3NbfUZ5mrGWsIvD3mImrfyiOnASvnCMwIlMTHlcJbJTIY+8JLmzbr+2TPbZ7zVechR4EoX9tEykY+GZml7cTic=,iv:7qoBRE+JlmArMejf++orrccAPDOpJsDp4vA+UHYCkoA=,tag:IwE7rZCXB+UhiP0yWWIFGA==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4
@@ -69,7 +69,41 @@ spec:
      memory: 256Mi
    limits:
      memory: 512Mi
-  # --- Off-cluster DR backup (R-STATE-03) — OPERATOR HAND-OFF (OQ-BKP, shared with Velero, Phase 4) ---
-  # When the off-cluster backup target is chosen, add a Barman object store + a ScheduledBackup here
-  # (same hand-off as keycloak-db / nextcloud-db). Until then the DB is protected by 3-replica Longhorn
-  # + (Phase-4) Velero PVC backups of the namespace; the SD declares backup.policy daily-offcluster.
+  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
+  # Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
+  # backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
+  # (defaults to the Cluster name), so objects land under s3://cnpg/semaphore-db/. Path-style HTTP.
+  # Credentials: the scoped key from the in-namespace SOPS Secret semaphore-db-backup-credentials
+  # (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
+  backup:
+    retentionPolicy: "30d"
+    barmanObjectStore:
+      destinationPath: s3://cnpg
+      endpointURL: http://192.168.1.9:9000
+      s3Credentials:
+        accessKeyId:
+          name: semaphore-db-backup-credentials
+          key: ACCESS_KEY_ID
+        secretAccessKey:
+          name: semaphore-db-backup-credentials
+          key: ACCESS_SECRET_KEY
+      wal:
+        compression: gzip
+        maxParallel: 2
+      data:
+        compression: gzip
+---
+# Daily base backup of semaphore-db (WAL archived continuously by the barmanObjectStore above). CNPG
+# cron is 6-field (sec min hour dom mon dow); immediate:true seeds the first base backup. 04:00 slot.
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: semaphore-db-daily
+  namespace: semaphore-db
+spec:
+  immediate: true
+  schedule: "0 0 4 * * *"
+  backupOwnerReference: self
+  cluster:
+    name: semaphore-db
+  method: barmanObjectStore