feat(backup): wire Velero + CNPG Barman to the live MinIO S3 target

Author the GitOps side of off-cluster backups against the on-Proxmox MinIO (http://192.168.1.9:9000, path-style; buckets velero + cnpg). Velero (R-STATE-03): - HelmRelease (chart 8.x / Velero 1.15) + velero-plugin-for-aws init container + node-agent kopia File System Backup (deployNodeAgent, uploaderType=kopia, snapshotsEnabled=false). BSL default -> bucket velero, s3ForcePathStyle. - Two daily Schedules: `daily` backs up the whole cluster's resources+PVC data (defaultVolumesToFsBackup opt-out) EXCEPT monitoring+unifi; `daily-resources-only` captures monitoring+unifi RESOURCES only (Prometheus TSDB, Loki 72h logs, and the ~7.4GiB UniFi VM disk are reconstructable — excluded from volume backup). - velero-cloud-credentials SOPS Secret (cloud INI key, ns velero). - Own non-gating Flux tier clusters/base/backup.yaml (dependsOn platform-config, wait:false) + clusters/dev SOPS decryption patch — mirrors headlamp. CNPG native Barman PITR (R-IDP-05 / R-STATE-03): - keycloak-db, nextcloud-db, semaphore-db gain spec.backup.barmanObjectStore (s3://cnpg/<cluster>, endpointURL MinIO, gzip WAL+data, 30d retention) + a daily ScheduledBackup (6-field cron, immediate) each. - Per-namespace scoped-key SOPS Secrets (<db>-backup-credentials), same-ns only. - render.py: emit-waves now recognizes <svc>-db-backup-credentials.sops.yaml as a DB-wave sidecar (+ self-test); wave-0 kustomization regenerated. Creds mirror platform/backup/minio-credentials.sops.yaml (scoped key k3s-backup), SOPS round-trip verified with the cluster age key. gate: green. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 22:37:17 -05:00
1 parent 1bee2554f8
commit d1248e6778
@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: keycloak-db-backup-credentials
+    namespace: keycloak
+type: Opaque
+stringData:
+    ACCESS_KEY_ID: ENC[AES256_GCM,data:N7iknaokuZ/tLg==,iv:yaGfsH5vplrNRqGnnpi8lYBGaIPDgTuRr0N2CWXXhh8=,tag:uLS1vvtUFGQu+Lb4b04pPg==,type:str]
+    ACCESS_SECRET_KEY: ENC[AES256_GCM,data:lV9r1ZXtsSwfaGO42x6hH3zE0cvBWSjcN1jzcvfRzpKdXa45eY68YA0tKdSC+0oy,iv:xs/uQJq+NkasynNRtgq+mGz8bFzgw8ZkuWI/oKR5b2g=,tag:SU/XmxCQGCrkcEsFTHbg9A==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLQVpJa1FWSDZudXhxQmJC
+            ZWlHVlorQk8vOWQxVzVLc3ZpTTVQQlBpbUNjClVwc3gvaVpyQzc5YVdYclp4YlVW
+            bWFpcFg3bDRvM1ROV0ZFSUZSU1NoeTAKLS0tIEpNSWlzRUM5VnpVY1cyRlY0T2xF
+            SjNFTy9FVm5GOFZ6WmgrV04ybGQxQjgKSnB2se7Ar1+piB0fn0wKSaOa3MLpetNa
+            gKPHJKP9OR5+KhlTnspBlxX9Un0WZcvZOQqAdewqlDklv4ukO7PgdQ==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-08-29T03:31:48Z"
+    mac: ENC[AES256_GCM,data:sxAENlR+qyBt2cBVG54hLbThXhzfNcKe8k1Ju3NmqWRyYLp/o7WHRwqydWHNYeY0YJJyuv73o1dDkVEPZEI2pxIlhaSwxhx+sDLaJnrYXq3YesbaT7jXkBrzz2MG1xjzV4nXgVYveM4rwCAWLij8qvIsbPMPyCHvaUjG/g0KDQ0=,iv:DoZgcHFsQUzTrStZfYr2bfQR6FGe5whNRF+lpzrXzKE=,tag:vGkrue7N0wjlDFZmQCqVQg==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4
@@ -28,5 +28,9 @@ resources:
  - keycloak-admin-credentials.sops.yaml
  # --- OIDC client secrets (grafana + oauth2-proxy) the realm import injects via placeholders ---
  - keycloak-realm-clients.sops.yaml
+  # --- Scoped MinIO S3 credential for the keycloak-db CNPG Barman backup (R-IDP-05) — SOPS-encrypted,
+  #     decrypted by the `platform` tier (which already has the decryption stanza). Lands in ns
+  #     keycloak; the keycloak-db Cluster (platform-config) reads it by name for WAL archiving. ---
+  - keycloak-db-backup-credentials.sops.yaml
  # --- Grafana dashboard for Keycloak health (plain ConfigMap; R-OBS-03) ---
  - grafana-dashboard.yaml
@@ -34,8 +34,43 @@ spec:
      memory: 256Mi
    limits:
      memory: 512Mi
-  # --- Off-cluster DR backup (R-IDP-05) — OPERATOR HAND-OFF (OQ-BKP, shared with Velero, Phase 4) ---
-  # When the off-cluster backup target is chosen, add a Barman object store + a ScheduledBackup:
-  #   spec.backup.barmanObjectStore = { destinationPath, endpointURL, s3Credentials {…SOPS…} }
-  #   + a postgresql.cnpg.io/v1 ScheduledBackup (cron) referencing this Cluster.
-  # Until then the DB is protected by 3-replica Longhorn + (Phase-4) Velero PVC backups of the namespace.
+  # --- Off-cluster DR backup (R-IDP-05) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
+  # Continuous WAL archiving + periodic base backups give proper Postgres PITR (superior to a Velero
+  # PVC snapshot for a live DB). destinationPath is the BUCKET ROOT; CNPG appends serverName (defaults
+  # to the Cluster name), so objects land under s3://cnpg/keycloak-db/. Path-style HTTP MinIO endpoint.
+  # Credentials: the scoped key from the in-namespace SOPS Secret keycloak-db-backup-credentials
+  # (a Pod only reads same-ns Secrets), kept in sync with platform/backup/minio-credentials.sops.yaml.
+  backup:
+    retentionPolicy: "30d"          # keep 30 days of base backups + WAL (PITR window); prunes older
+    barmanObjectStore:
+      destinationPath: s3://cnpg
+      endpointURL: http://192.168.1.9:9000
+      s3Credentials:
+        accessKeyId:
+          name: keycloak-db-backup-credentials
+          key: ACCESS_KEY_ID
+        secretAccessKey:
+          name: keycloak-db-backup-credentials
+          key: ACCESS_SECRET_KEY
+      wal:
+        compression: gzip
+        maxParallel: 2
+      data:
+        compression: gzip
+---
+# Daily base backup of keycloak-db (WAL is archived continuously by the barmanObjectStore above).
+# CNPG cron is 6-field (sec min hour dom mon dow). immediate:true takes a first base backup as soon
+# as the schedule is created so the PITR chain has a starting point. Staggered 01:00 (nextcloud 02:00,
+# semaphore 04:00, Velero 03:00) to avoid concurrent base backups hammering MinIO.
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: keycloak-db-daily
+  namespace: keycloak
+spec:
+  immediate: true
+  schedule: "0 0 1 * * *"
+  backupOwnerReference: self
+  cluster:
+    name: keycloak-db
+  method: barmanObjectStore