From e7c4a95c2dacbde7a3386065618603c2f123b628 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 7 Sep 2026 01:38:22 -0500 Subject: [PATCH] feat(backup): the four backup credentials become TWO objects each, one per environment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The four backup-credential SOPS Secrets are SHARED BASE manifests: platform/ and apps/ are rendered by the same Flux tiers on both clusters, and the ONLY per-environment difference in this repo is cluster-vars substitution (R-ENV-02). So writing the R2 token into all four (0a6a862) did not "give prod credentials" — it REPLACED DEV'S. Merging that as-is would have handed dev's Velero and all three dev CNPG Clusters an R2 key id to sign MinIO requests with: 403 on every WAL segment, and archive_mode=on turns that into a disk-full clock on the pgdata PVC, not a degraded backup. Dev's PROVEN DR restore (R-STATE-03) is the thing that would have broken. This commit splits the material. Each credential is now TWO Secret objects, both applied to BOTH clusters, differing only in name suffix: dev -> velero-cloud-credentials keycloak/nextcloud/semaphore-db-backup-credentials prod -> velero-cloud-credentials-prod ...-db-backup-credentials-prod The consumer picks one by name; that wiring is the NEXT commit, so this one changes no rendered reference and is a no-op for both live clusters. WHY DEV KEEPS THE UNSUFFIXED NAME. Dev is live. Renaming its Secret would change `s3Credentials.accessKeyId.name` on three running CNPG Clusters — which projects into the instance pods' environment and therefore ROLLS keycloak-db, nextcloud-db and semaphore-db — and would roll the velero Deployment, all to rename an object whose contents do not change. This repo already makes exactly this call for dev's live identifiers (EXTERNALDNS_TXT_OWNER_ID "default", VELERO_BACKUP_PREFIX ""): the live cluster keeps the name it is running, and the NEW environment takes the new one. The asymmetry is deliberate and documented at each kustomization. WHERE THE MATERIAL CAME FROM. Neither side is invented: - the four unsuffixed files are restored byte-for-byte from `main` (the MinIO material as it stood before 0a6a862), so dev's Secrets are IDENTICAL to what dev is running; - the four `-prod` files carry the R2 token from 0a6a862 (already on this branch as 9739e9b), re-encrypted only because metadata.name changed. RE-ENCRYPTION, AND WHY A COPY+SED IS NOT ENOUGH. `.sops.yaml` sets `encrypted_regex: ^(data|stringData)$`, so metadata.name stays cleartext — but SOPS MACs the WHOLE document (there is no `mac_only_encrypted` here), so editing that cleartext line invalidates the MAC and `sops -d` then refuses the file. The `-prod` files were therefore produced inside the dev toolbox (infra/devcontainer, which has sops 3.9.4 and the age key mounted) by decrypt -> rename -> re-encrypt, with the plaintext never leaving a container tmpfile. Note for the next person: `sops -e --output NEW OLD` matches creation rules against the INPUT path, so encrypting a tmpfile into a repo path needs `--filename-override NEW` or it dies with "no matching creation rules found". Verified: all eight files `sops -d` cleanly; the four dev files hash to dev's material and the four prod files to R2's, and the two sets differ. `.sops.yaml` needed no new rule — the existing `(platform|apps|catalog)/.*(secret|sops)\.ya?ml$` rule already matches `*-prod.sops.yaml`, so the new files encrypt to the same age recipient. apps/wave-0/kustomization.yaml is GENERATED, so render.py learns the `-prod` sidecar rather than the golden being hand-edited: `-backup-credentials-prod.sops.yaml` now places into the DB's wave alongside its unsuffixed twin (both must exist before the Cluster that names one), and the selftest asserts it. Co-Authored-By: Claude Opus 5 --- apps/wave-0/kustomization.yaml | 2 ++ ...cloud-db-backup-credentials-prod.sops.yaml | 29 +++++++++++++++++++ .../nextcloud-db-backup-credentials.sops.yaml | 8 ++--- ...phore-db-backup-credentials-prod.sops.yaml | 29 +++++++++++++++++++ .../semaphore-db-backup-credentials.sops.yaml | 8 ++--- catalog/render.py | 15 ++++++++-- ...cloak-db-backup-credentials-prod.sops.yaml | 29 +++++++++++++++++++ .../keycloak-db-backup-credentials.sops.yaml | 8 ++--- platform/keycloak/kustomization.yaml | 14 +++++++-- platform/velero/kustomization.yaml | 18 ++++++++++-- platform/velero/velero-cloud-prod.sops.yaml | 28 ++++++++++++++++++ platform/velero/velero-cloud.sops.yaml | 6 ++-- 12 files changed, 171 insertions(+), 23 deletions(-) create mode 100644 apps/wave-0/nextcloud-db-backup-credentials-prod.sops.yaml create mode 100644 apps/wave-0/semaphore-db-backup-credentials-prod.sops.yaml create mode 100644 platform/keycloak/keycloak-db-backup-credentials-prod.sops.yaml create mode 100644 platform/velero/velero-cloud-prod.sops.yaml diff --git a/apps/wave-0/kustomization.yaml b/apps/wave-0/kustomization.yaml index bd47530..4899214 100644 --- a/apps/wave-0/kustomization.yaml +++ b/apps/wave-0/kustomization.yaml @@ -11,6 +11,7 @@ resources: - jackett-oauth2-proxy-credentials.sops.yaml - jackett.yaml - nextcloud-db-app.sops.yaml +- nextcloud-db-backup-credentials-prod.sops.yaml - nextcloud-db-backup-credentials.sops.yaml - nextcloud-db.yaml - pihole-oauth2-proxy-credentials.sops.yaml @@ -19,5 +20,6 @@ resources: - qbittorrent-oauth2-proxy-credentials.sops.yaml - qbittorrent.yaml - semaphore-db-app.sops.yaml +- semaphore-db-backup-credentials-prod.sops.yaml - semaphore-db-backup-credentials.sops.yaml - semaphore-db.yaml diff --git a/apps/wave-0/nextcloud-db-backup-credentials-prod.sops.yaml b/apps/wave-0/nextcloud-db-backup-credentials-prod.sops.yaml new file mode 100644 index 0000000..0e3bf84 --- /dev/null +++ b/apps/wave-0/nextcloud-db-backup-credentials-prod.sops.yaml @@ -0,0 +1,29 @@ +apiVersion: v1 +kind: Secret +metadata: + name: nextcloud-db-backup-credentials-prod + namespace: nextcloud-db +type: Opaque +stringData: + ACCESS_KEY_ID: ENC[AES256_GCM,data:4lx6et4HcXDegulyIoa7jZR/lVgapGPEkukajfCKjvk=,iv:p6V7tizbikk55tOvpPOsPHTe/p52QLX5A5xrwuB14rA=,tag:oNJuipT/Lb6zynJ2K2Q71A==,type:str] + ACCESS_SECRET_KEY: ENC[AES256_GCM,data:V5bHPoPeBOVyLp1n1Jilyso4i5P2aPSH0hrryfweg2q/SctWZxRbLBQ/FLzOigbJsAhlwOGu069mQPma6xjfdw==,iv:8u4UOvI8dC/cm6mEWDlyN5E3SfRZmwKY9uZY/DjPEsQ=,tag:tw7tGs29TmUvMPBCtCszAA==,type:str] +sops: + kms: [] + gcp_kms: [] + azure_kv: [] + hc_vault: [] + age: + - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg + enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiSUt5N2k2N2lDU2ZsMy9m + M1NFS0M2QjFsNFhnR0hCMzFyaTdiQWFZRFJZCmhBTWRXNmxLMGtsc3Rxd2FSMDZp + aktjcTE2c1hmU1EzM3RWNksyb3Q1YVUKLS0tIDBFS1I5aDBER2cvcWtLZ3BXTEp5 + ZGI5ZmZIcStheUNQamxIVVVpc0ZMOHMKlE5aHewtHJpeU24F5sodc9CEm9KTjHQQ + 8ThEQ/8vyazL9V4ojlICxSHU/SHkHpD1eBaTKcAQd/LZzujIcJ+QfA== + -----END AGE ENCRYPTED FILE----- + lastmodified: "2026-09-07T06:36:33Z" + mac: ENC[AES256_GCM,data:8awz1Jh8t4zjDMyd93/v+6on9iiIaSi01293A7uYhxj9L1+g63zeO+DPmSn6+KbZvHdqU2A7up09H1NzfXAEdGuvVbB36cdGq0zKank3PUi8c7IT6qNuWydxRAcqLnbbD+c30Z6GB+gAuZHstNOe4Rgc4aJvMN6FuP6x6Z0870c=,iv:KGGKJq2QA5xztvoHs/SViXW72DYwT16i+W+dJBUeIJQ=,tag:3UpYqx2+RIK0QdMH/BSIpA==,type:str] + pgp: [] + encrypted_regex: ^(data|stringData)$ + version: 3.9.4 diff --git a/apps/wave-0/nextcloud-db-backup-credentials.sops.yaml b/apps/wave-0/nextcloud-db-backup-credentials.sops.yaml index d1e3625..3cef69f 100644 --- a/apps/wave-0/nextcloud-db-backup-credentials.sops.yaml +++ b/apps/wave-0/nextcloud-db-backup-credentials.sops.yaml @@ -5,8 +5,8 @@ metadata: namespace: nextcloud-db type: Opaque stringData: - ACCESS_KEY_ID: ENC[AES256_GCM,data:LkBD4YP2NXJQNDcxwukoHoawmnVHF+cOZID+T4QSfIU=,iv:mVDSC02WczRjSor+V0dTHDVX3XpWuaAgwvOE45CSu/Q=,tag:n60oC4oW1sHIx3nKUAPnKA==,type:str] - ACCESS_SECRET_KEY: ENC[AES256_GCM,data:IbzhRTEoJUTe/uEguMhaRwtkbw4HCmbrnLg3H4DcfgCo/63LVaq7dWogJaXhvA+P78aO7sRdFW/g/NCThofU+g==,iv:E6gjP/v5WNoXLKTF+Td5szwJ/Q2kPcnRYXda9Ku4nCw=,tag:FYns9o1Y4rtHFXUD8FNICw==,type:str] + ACCESS_KEY_ID: ENC[AES256_GCM,data:gS533S1h2f5uWQ==,iv:XmcNCYdlkAIewn7Br8Kk4CHcZWbMQxJE0Mlge9/oGRY=,tag:+krC/C+EMQl5pYgwkuut1g==,type:str] + ACCESS_SECRET_KEY: ENC[AES256_GCM,data:NTcBY05zUy8ngV3phNF18FBHRCEKQGu2KQ2yQGu4YiuJasllznZ+YvCBn/3i0Z7/,iv:uXyenHmNqBRb5CC/ELgVbFXV/W59ZPOrSeGwIAi8Hz0=,tag:9ZRcKL0ijOPBi4QUbDZyoQ==,type:str] sops: kms: [] gcp_kms: [] @@ -22,8 +22,8 @@ sops: ODAyK2g4bGlXeXV0Q1FMbXNiSEpUYncKvwviYgI/XjCBJG5kpS8tOHLTLsnXv1bl b4E4rigNesmreSYxAyhGw9bgQNY6ba/F3Ecexrdx+iV/tYYc/VoDKA== -----END AGE ENCRYPTED FILE----- - lastmodified: "2026-09-07T05:41:34Z" - mac: ENC[AES256_GCM,data:dO1Z1YF2wYThTMRBUJbg9bMuiDAtDpN2ERpkC2mpvLvH09rDBmb3IWzC7iHKm2GQ9v/8yZ6WEN42RMAQBxD7MtVRtjiTDuuIeOObFuw69FDOfRFx2NR7C6LoXxrUm8/8Q/KcZM9PRHQMM7jpGRIWTyFyeA/2EuUdRooIN7MpIkA=,iv:9hpzLEZ2UdRgVYVEqDY7bjHfWUdjf+kflGgeYx19LBE=,tag:A7ihm4K9rLBgoQ4uvx5Ebg==,type:str] + lastmodified: "2026-08-29T03:31:48Z" + mac: ENC[AES256_GCM,data:ks4i7zjrJQajKdncnnV5dD4Ap1Q1xdKyiQAiqhao2USXu9S+4flX2kzc9lv8lFG/dTiWvLu6Tqgmj46ssiqtd0E8DnrASNEuKz/Xlw1iCka2008C5BhE/k8oy84kXXDgEuo7GiYdhJU/mAaAnY4p10ZQkvTKddQMmSO65YRwRMo=,iv:skZoS+bhAOnLlY5R5OfZvFV8yCEiEKtLac4mVbdmZ2s=,tag:6T1JIroe5hI8RXWAyzEKLw==,type:str] pgp: [] encrypted_regex: ^(data|stringData)$ version: 3.9.4 diff --git a/apps/wave-0/semaphore-db-backup-credentials-prod.sops.yaml b/apps/wave-0/semaphore-db-backup-credentials-prod.sops.yaml new file mode 100644 index 0000000..837038a --- /dev/null +++ b/apps/wave-0/semaphore-db-backup-credentials-prod.sops.yaml @@ -0,0 +1,29 @@ +apiVersion: v1 +kind: Secret +metadata: + name: semaphore-db-backup-credentials-prod + namespace: semaphore-db +type: Opaque +stringData: + ACCESS_KEY_ID: ENC[AES256_GCM,data:OhWEEYqrvmDB40NmiETgeUisBvaO9ocevCZAsurNijw=,iv:kEjOIg9NHbRYm+Ul1WrBAzb3gLbIv+ipiYCxFK0QoVc=,tag:KAB1JAIw0ATwbjDEKZVT9A==,type:str] + ACCESS_SECRET_KEY: ENC[AES256_GCM,data:C0PNKv07LCYpFONLqd55sz+ioyU/i2YJUkbZ8OX/dCErM/DilqchyUAT5EqnT/r3YJmdaA1pX85S3fxlVVHkbg==,iv:4ha28b+xIebvX2JPITmtTovIe4vdmj2XfI3cgkUHH4c=,tag:xbMyflcHojytPzgaoOYcXQ==,type:str] +sops: + kms: [] + gcp_kms: [] + azure_kv: [] + hc_vault: [] + age: + - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg + enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZVTJ3cXR6cks3Y0RNS2hx + K3pjZEduYUxmL0V3SmFUcU9XbGVHSVFITWh3Cjl1eHlTLytoOW5PaVNFVkJoeTBo + T1FnRHNDVzJ6dWpuR09WaElvbWF4YU0KLS0tIEJoQ1VsM1cxRlg0R1dtNGdOenNn + aVNjMjVBTytLdHlFdmZCS2tPelo1Y2MKYt1G5t+sDiHgcg+Q9ECBzJn2gS/paSEC + 7aoX60SBrPgHvnGls5lzc3Tin9nH4jWYl6Nu8DAvIE0YPXDFr5a0UA== + -----END AGE ENCRYPTED FILE----- + lastmodified: "2026-09-07T06:36:33Z" + mac: ENC[AES256_GCM,data:mhVP7tjdZK2FR2imvpJbTyT7BigvcIKEpLSnBS1oSITUirB8xZC7aHBfyFuIoKOrpOJgEe40nHUdDYCHRJtb4dLeVja8+7meVaQWzxC8BWVHsksW+73hsqhjAzlHeobx6Irbk4PAMqIY0/jnf4MHryjnRn3dUuwu9U1CqL30JbE=,iv:57oCcp6FW9EAIVKnYGtkaTrnbijyv54T+vmZxMbMqnw=,tag:agrRP/JLypU60KOeN0s/5w==,type:str] + pgp: [] + encrypted_regex: ^(data|stringData)$ + version: 3.9.4 diff --git a/apps/wave-0/semaphore-db-backup-credentials.sops.yaml b/apps/wave-0/semaphore-db-backup-credentials.sops.yaml index fc97f8e..d7a4574 100644 --- a/apps/wave-0/semaphore-db-backup-credentials.sops.yaml +++ b/apps/wave-0/semaphore-db-backup-credentials.sops.yaml @@ -5,8 +5,8 @@ metadata: namespace: semaphore-db type: Opaque stringData: - ACCESS_KEY_ID: ENC[AES256_GCM,data:sIVd7sJEWKBpu3PYxxmSCZ8OsZQLrys5SiCj0iPor28=,iv:hLmkR7TYyEAWrtuq41RatwqQR1RdUz9Q54BfcGAJ5xI=,tag:oRSTMdOd9SJUf8+7l+YsNw==,type:str] - ACCESS_SECRET_KEY: ENC[AES256_GCM,data:8sBtArxeBu7cDBX2Y18oy9mMigukmJc5NhbUD56ddSPB1eZgJbznZmMwel59+/IkqsMZSsaKDqT8PvVMWPrbxQ==,iv:OSfNZtl+ByJlZBPNUSV2NaF5SEDJLuhAtFpbVdRk4ko=,tag:Aow0/AKLkARx1NiIqc7LKA==,type:str] + ACCESS_KEY_ID: ENC[AES256_GCM,data:jmhu9U7Lyo0ugA==,iv:EyU67JLLO10za1tM/peju9tvBBINJ7ZDn6F9Zu0g6H8=,tag:9M/adv8Gdu3XNBJ/G4xfeA==,type:str] + ACCESS_SECRET_KEY: ENC[AES256_GCM,data:f9qW7BX5OQ2L7FqZ6pr6lqdEhTy+7xabLSMTnHZ/UYj1RnQ9asL+/HNkEWsMatJC,iv:hS2Q5l4EsvMBop/sBnld83tm2MhsTTI5N72BLGLMdtk=,tag:MwfC205a/djUA6HNqobcUw==,type:str] sops: kms: [] gcp_kms: [] @@ -22,8 +22,8 @@ sops: cGpweDl4VjNsNHM0Sm8zVDBWcGFJanMKpBBV9cnSZRVmIGjEi9YBMlbm4df253cS osMRBCKizDIz1Q0Awq952y/L97ljUnSE6JWnlKTriHoGtLDct4mg3Q== -----END AGE ENCRYPTED FILE----- - lastmodified: "2026-09-07T05:41:34Z" - mac: ENC[AES256_GCM,data:QUGHd8pmgMrcdQKaoQDq41VstAoV1Dhb8SmihRAr+p62KNbWTLf7L10mAnwopTSy8ucA+p52bRgcMx6//ZvM7j/DjilmC+NMnMIFqRthVtJ3WWUGbWmlWAGHHg0XyEdims0F3fh1CIkZIx0ONgiOpUT3F/s2VajJhrjKq37peMA=,iv:sgfrDaAsChQlrcSlqD9rZ+8JPCCa+dhYOm6L+hnPfcY=,tag:b879Nimwl+Fdw9NRvbEe+g==,type:str] + lastmodified: "2026-08-29T03:31:48Z" + mac: ENC[AES256_GCM,data:uqtBwfubrUqaRN5r5qfMlITBvKxfbdyxZGDcZgyWtmDvX1OulKBvuOYBZaDW1MdIrq++nbKhocQK2PkuAmqZI3NbfUZ5mrGWsIvD3mImrfyiOnASvnCMwIlMTHlcJbJTIY+8JLmzbr+2TPbZ7zVechR4EoX9tEykY+GZml7cTic=,iv:7qoBRE+JlmArMejf++orrccAPDOpJsDp4vA+UHYCkoA=,tag:IwE7rZCXB+UhiP0yWWIFGA==,type:str] pgp: [] encrypted_regex: ^(data|stringData)$ version: 3.9.4 diff --git a/catalog/render.py b/catalog/render.py index 16b4323..3533e46 100644 --- a/catalog/render.py +++ b/catalog/render.py @@ -1321,6 +1321,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict: - `-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB) - `-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret) - `-db-backup-credentials.sops.yaml` -> wave(svc-db) (CNPG Barman S3 backup secret) + - `-db-backup-credentials-prod.sops.yaml` -> wave(svc-db) (the PROD half of that + secret: the two environments carry DIFFERENT object-store credentials, selected in + the Cluster CR by `${BACKUP_CREDENTIALS_SUFFIX}`. Both objects ship to both clusters, + so both must be placed in the DB wave — the Secret has to exist before the Cluster + that names it.) Non-catalog units (EXTRA_UNITS, e.g. gitea-actions) place by their declared anchor+offset. Only files that actually exist in `present_files` are placed, so the plan is robust to a service having no bootstrap secret or no DB. Deterministic (sorted). @@ -1348,6 +1353,9 @@ def plan_apps_waves(sds: list, present_files: set) -> dict: add(wdb, f"{db}.yaml") add(wdb, f"{db}-app.sops.yaml") add(wdb, f"{db}-backup-credentials.sops.yaml") # CNPG Barman S3 creds (R-STATE-03) + # PROD half of the same credential (2026-09-07 per-env split). Placed only when + # the file exists, so a DB that has not been split is unaffected. + add(wdb, f"{db}-backup-credentials-prod.sops.yaml") for unit, cfg in EXTRA_UNITS.items(): anchor = cfg["anchor"] @@ -1830,6 +1838,7 @@ def _selftest() -> int: present = { "a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml", "svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml", + "svc-db-backup-credentials-prod.sops.yaml", "gitea.yaml", "gitea-actions.yaml", "gitea-actions-runner-secret.sops.yaml", } @@ -1839,8 +1848,10 @@ def _selftest() -> int: flat = {f: wv for wv, fs in plan.items() for f in fs} check(flat.get("a.yaml") == 0, "leaf app must land in wave 0") check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0 - and flat.get("svc-db-backup-credentials.sops.yaml") == 0, - "CNPG DB manifest + its owner secret + its Barman backup secret must be in the DB's (lower) wave") + and flat.get("svc-db-backup-credentials.sops.yaml") == 0 + and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0, + "CNPG DB manifest + its owner secret + BOTH halves of its per-env Barman backup secret " + "must be in the DB's (lower) wave") check(flat.get("svc.yaml") == 1 and flat.get("svc-bootstrap-secret.sops.yaml") == 1, "a DB-backed app + its bootstrap secret must be in the consumer wave, above the DB") check(flat.get("gitea.yaml") == 0, "gitea (leaf) must be wave 0") diff --git a/platform/keycloak/keycloak-db-backup-credentials-prod.sops.yaml b/platform/keycloak/keycloak-db-backup-credentials-prod.sops.yaml new file mode 100644 index 0000000..85fcf84 --- /dev/null +++ b/platform/keycloak/keycloak-db-backup-credentials-prod.sops.yaml @@ -0,0 +1,29 @@ +apiVersion: v1 +kind: Secret +metadata: + name: keycloak-db-backup-credentials-prod + namespace: keycloak +type: Opaque +stringData: + ACCESS_KEY_ID: ENC[AES256_GCM,data:kQ5x5fXb6KdVaDsItGug2cpQMZw9tSvbsYDqTMxoP+A=,iv:dCTfwVxlpnjaMhGBIMa18d4yAiQ6SKhb4EMm1p1JyQ8=,tag:sNqMLs+rGQm/1MyV90InDQ==,type:str] + ACCESS_SECRET_KEY: ENC[AES256_GCM,data:9lGvwvseAWK1bMXDXZJ+bjw4oHH/XVPi8+C8vfXGJ3jX8VW95rJrczPbEfV9rCudVIJFnxAdaQZZWaJCsC8fCQ==,iv:GOvPs2I77yiKh381bcv0kCwT5xbk8Fao6ctPCW2aCHg=,tag:hSjWjaZ+nEJwZyv3sIuq7A==,type:str] +sops: + kms: [] + gcp_kms: [] + azure_kv: [] + hc_vault: [] + age: + - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg + enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBN2NvVXJ5TW1mQlhYWnRC + UlVLVjFtNk1Wdkk1NzdicXBQVGNtc3JZclNBCjl3UlVTZmFJWGJGam54VytVR0lZ + djB3ZjBCOFFNOUFMZGxCcnZiZ3VDMUUKLS0tIFhDSWR3ZXZPOHFzVGNhZGhqdUQv + WmF0NlNIeTJCeVV0ZGkxSFJIL3BqNlEKntEpxCdpy7Xd16ZKe6skQYL6hCqYOAIN + NtwtQ9oF/8i4i0T7B8kcORB71cu8dPu6w6LtaSC+1qBV5TBSSH5U6Q== + -----END AGE ENCRYPTED FILE----- + lastmodified: "2026-09-07T06:36:33Z" + mac: ENC[AES256_GCM,data:dcgB6z9BDzaft4do6f1SqjCqmQopc5mM0ckfU/HfE09eOSIRjpoZ+qMzJW8RIwUZNKrjEAwrrAEtrN67GRxL0jOEsw/6xNrWJRFv6KTeE3ECy5qMO4jLCQTeMvZuvfZE7iqwvHCCnjhkqkedEAD4Hwefq0c0EZ0ZxZK3z1tVdHk=,iv:Ue+ZJq9dJl9qbjJ1l8Q24VmsK+KW/uSRFu04xG6/tM8=,tag:MvJ6Ww3h3b/C2PqA/62abA==,type:str] + pgp: [] + encrypted_regex: ^(data|stringData)$ + version: 3.9.4 diff --git a/platform/keycloak/keycloak-db-backup-credentials.sops.yaml b/platform/keycloak/keycloak-db-backup-credentials.sops.yaml index b65c480..67c5bf0 100644 --- a/platform/keycloak/keycloak-db-backup-credentials.sops.yaml +++ b/platform/keycloak/keycloak-db-backup-credentials.sops.yaml @@ -5,8 +5,8 @@ metadata: namespace: keycloak type: Opaque stringData: - ACCESS_KEY_ID: ENC[AES256_GCM,data:SktHHkzztP4MX+DDaDeT5gAu8Q5gxl0yd2hbJmNlj1k=,iv:OkKvG0tQw99088oU9A8ujy0WkEWzx0WzrRk844xJXJg=,tag:z4u656LsqhMZcoVuWv1hXg==,type:str] - ACCESS_SECRET_KEY: ENC[AES256_GCM,data:26KiA364CugQZj8zae+hw5OPFrHlF7cYnLkWOwqdaUQYUr825pUY5xxN03JPPqjFDndeBdi8eu29UAhiPX7Bjw==,iv:jlp5Fj/4kyolGdfPp4WZUEdXvUE/lp/kxD8JvoUdz0A=,tag:qF/yi2+k4uEvbhsoxgkRJg==,type:str] + ACCESS_KEY_ID: ENC[AES256_GCM,data:N7iknaokuZ/tLg==,iv:yaGfsH5vplrNRqGnnpi8lYBGaIPDgTuRr0N2CWXXhh8=,tag:uLS1vvtUFGQu+Lb4b04pPg==,type:str] + ACCESS_SECRET_KEY: ENC[AES256_GCM,data:lV9r1ZXtsSwfaGO42x6hH3zE0cvBWSjcN1jzcvfRzpKdXa45eY68YA0tKdSC+0oy,iv:xs/uQJq+NkasynNRtgq+mGz8bFzgw8ZkuWI/oKR5b2g=,tag:SU/XmxCQGCrkcEsFTHbg9A==,type:str] sops: kms: [] gcp_kms: [] @@ -22,8 +22,8 @@ sops: SjNFTy9FVm5GOFZ6WmgrV04ybGQxQjgKSnB2se7Ar1+piB0fn0wKSaOa3MLpetNa gKPHJKP9OR5+KhlTnspBlxX9Un0WZcvZOQqAdewqlDklv4ukO7PgdQ== -----END AGE ENCRYPTED FILE----- - lastmodified: "2026-09-07T05:41:34Z" - mac: ENC[AES256_GCM,data:GBGgpKf+cQaySZIzlSa6/cmhjH18Y/yRTUvYrrpVcMTdpIuhy9iJtbc2EsGxUs0YSP69BW61VoZJoZoZXpSqJX5h7p6eFYqT5GAMnCY31FTUhmjIOpCJ5hlDfSmHR7y1wNLDy0DKFj5oeKFzsVLF2TKC1nWjJSWZXIp2MPp2cks=,iv:qqBtl5PHobme+zO+lAwFnVo5z6esojq3qV7NHxIuZCs=,tag:2eHDjzUBQ18pAqTVg9dpFQ==,type:str] + lastmodified: "2026-08-29T03:31:48Z" + mac: ENC[AES256_GCM,data:sxAENlR+qyBt2cBVG54hLbThXhzfNcKe8k1Ju3NmqWRyYLp/o7WHRwqydWHNYeY0YJJyuv73o1dDkVEPZEI2pxIlhaSwxhx+sDLaJnrYXq3YesbaT7jXkBrzz2MG1xjzV4nXgVYveM4rwCAWLij8qvIsbPMPyCHvaUjG/g0KDQ0=,iv:DoZgcHFsQUzTrStZfYr2bfQR6FGe5whNRF+lpzrXzKE=,tag:vGkrue7N0wjlDFZmQCqVQg==,type:str] pgp: [] encrypted_regex: ^(data|stringData)$ version: 3.9.4 diff --git a/platform/keycloak/kustomization.yaml b/platform/keycloak/kustomization.yaml index 20c3bba..de861be 100644 --- a/platform/keycloak/kustomization.yaml +++ b/platform/keycloak/kustomization.yaml @@ -28,9 +28,17 @@ resources: - keycloak-admin-credentials.sops.yaml # --- OIDC client secrets (grafana + oauth2-proxy) the realm import injects via placeholders --- - keycloak-realm-clients.sops.yaml - # --- Scoped MinIO S3 credential for the keycloak-db CNPG Barman backup (R-IDP-05) — SOPS-encrypted, - # decrypted by the `platform` tier (which already has the decryption stanza). Lands in ns - # keycloak; the keycloak-db Cluster (platform-config) reads it by name for WAL archiving. --- + # --- PER-ENVIRONMENT S3 credential for the keycloak-db CNPG Barman backup (R-IDP-05), split + # 2026-09-07. SOPS-encrypted, decrypted by the `platform` tier (which already has the + # decryption stanza). BOTH land in ns keycloak on BOTH clusters; the keycloak-db Cluster + # (platform-config) selects ONE by name via + # `keycloak-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}` (cluster-vars: dev "", + # prod "-prod"). Two objects, so an R2 token rotation cannot silently break dev's MinIO + # WAL archiving — and archive_mode=on makes a broken archive a disk-full clock, not a + # degraded backup (see postgres-cluster.yaml). + # dev -> keycloak-db-backup-credentials (MinIO, s3://cnpg) + # prod -> keycloak-db-backup-credentials-prod (Cloudflare R2, s3://k3s-prod-backup/barman) - keycloak-db-backup-credentials.sops.yaml + - keycloak-db-backup-credentials-prod.sops.yaml # --- Grafana dashboard for Keycloak health (plain ConfigMap; R-OBS-03) --- - grafana-dashboard.yaml diff --git a/platform/velero/kustomization.yaml b/platform/velero/kustomization.yaml index bfd99ae..feacb73 100644 --- a/platform/velero/kustomization.yaml +++ b/platform/velero/kustomization.yaml @@ -5,7 +5,19 @@ resources: - namespace.yaml - helmrepository.yaml - helmrelease.yaml - # Scoped MinIO S3 credential (the `cloud` INI key the AWS plugin reads) — SOPS-encrypted, decrypted - # per-env by the `velero` Flux Kustomization (clusters/dev). Kept in sync with the staging source of - # truth platform/backup/minio-credentials.sops.yaml + the Proxmox host ansible-vault. + # --- PER-ENVIRONMENT `cloud` INI credential (split 2026-09-07) ----------------------------------- + # BOTH Secrets are applied to BOTH clusters; the HelmRelease picks ONE by name via + # `existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}` (cluster-vars: dev "", + # prod "-prod"). They are separate objects so rotating one environment's key can never touch the + # other's — the failure mode that made the split necessary (a single shared Secret carrying R2 keys + # would have pointed dev's Velero at MinIO with credentials MinIO has never issued). + # + # dev -> velero-cloud-credentials : the scoped MinIO key for http://192.168.1.9:9000. + # Kept in sync with the staging source of truth + # platform/backup/minio-credentials.sops.yaml + the + # Proxmox host ansible-vault. + # prod -> velero-cloud-credentials-prod : the Cloudflare R2 token for k3s-prod-backup. + # + # Both are decrypted by the `velero` Flux Kustomization's SOPS stanza (clusters/dev + clusters/prod). - velero-cloud.sops.yaml + - velero-cloud-prod.sops.yaml diff --git a/platform/velero/velero-cloud-prod.sops.yaml b/platform/velero/velero-cloud-prod.sops.yaml new file mode 100644 index 0000000..37cb52d --- /dev/null +++ b/platform/velero/velero-cloud-prod.sops.yaml @@ -0,0 +1,28 @@ +apiVersion: v1 +kind: Secret +metadata: + name: velero-cloud-credentials-prod + namespace: velero +type: Opaque +stringData: + cloud: ENC[AES256_GCM,data:Kq+hfYMq/NZJJqbK2zqwW97XKNW0LbT2Un20yc0B6evPXqp+aC0vV0odwos4XwV0puPIsEhU7IeD+JidVjU1eKv6Qax0rnoj90FVWihxWyLMNAMC+b8PxBnPhWdk6ZlrMKs13roGZwYh4bbrstnJamMqpD4oZpthEl/NEQNWVAb5ogJHsmPNiQOtHhtBWkTubfgPIw==,iv:gn8/4/XpUZ8c6+nlGe6SAODUyrhDqskQ+uVRx2YR3Qc=,tag:aTk3cRJTA0WjdIi1B9S0hg==,type:str] +sops: + kms: [] + gcp_kms: [] + azure_kv: [] + hc_vault: [] + age: + - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg + enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwWmV5UGpmdnFKbEhjeHlZ + QUZCWm13ZlpVYU1URFErSmM5MEJ2WTRxZVFvCkZ5cEdJaXJ1NDNSaEF5d2lxdDQw + NGpzQm5ESGt6cUlIZVdHYTBjRGVwaVUKLS0tIHlpWkVacXdIZndBOXdlVk1LZzlN + NkZyRm1abk9CNlZRUERtbFM2RjVtS1UKWYtI7QZiTjPIHaLOOX9WvDK1cHwP2WfW + 26WdNW90tTmt3cYdaFGnkpAVCMS3BRtdInirulfoGRgvffjWNBDVuQ== + -----END AGE ENCRYPTED FILE----- + lastmodified: "2026-09-07T06:36:33Z" + mac: ENC[AES256_GCM,data:4v4L29KoIhQI94TU/s0u3HKR5C6mMc6mRWWsZzwIfqkl4fayRvLG67tbJ88vVkb6AXgqHoKXKlhbpn6hTJPBrL72f8+fii3CeLloVJyW2AuzHaHCSi92U04ywoSc45hClZrOLvUFf5KbtawPPKfP53oaNq+AVEZ1nqKd8ssUyBA=,iv:WVrdja00ptK/MmCC/Chu+Gq4ZYpBvpFRWW3lSo7Mon0=,tag:GzFfpkBlHu6Nc8ZdnCBuKQ==,type:str] + pgp: [] + encrypted_regex: ^(data|stringData)$ + version: 3.9.4 diff --git a/platform/velero/velero-cloud.sops.yaml b/platform/velero/velero-cloud.sops.yaml index 9841b26..b3142be 100644 --- a/platform/velero/velero-cloud.sops.yaml +++ b/platform/velero/velero-cloud.sops.yaml @@ -5,7 +5,7 @@ metadata: namespace: velero type: Opaque stringData: - cloud: ENC[AES256_GCM,data:+4w1Ta7Yh+JwW4fG3qtHKaGPpjiCNAFJs22gHW1kMt6w7qYxc7ZCTY06IoUh3a2jS+EXAzn5HAZx0mN76C8JwrfPBRTtMq5K9lhhUjZf1iA1Ku2Dwo46YrPWZZvNkwEcfpX3DRuj2P7IY0EduRUrzHlCRxCVbHa4/uTX8mdzXdBuycOAvqM8vRPKqfRpWxjhpv7V2w==,iv:9ikaNLiC6WA4MxrC6A5FmALqzu8wTON8SkfULSxH+X0=,tag:FFIP85+LfElfdWISHYO6Ug==,type:str] + cloud: ENC[AES256_GCM,data:ZnGAkgO3sOXfaL+1QpK6jVixBmR/1iQhhipo+dWcmJBGbxgxFv37L4Y2gmLG0a58G3oghIYWOX++uc/mrH4ZYlLCjARHHBjHCtp4GouXUADwxj2jlVo40bk5GCpsvn3ge8zLdl8dyBF1ZAx66dQ=,iv:FgQOJqJxgchnM7cvl9NeW1o3wf2cos9jl1AR8vnVN9I=,tag:AqFE1a+KAQs5JUlY1FV5wQ==,type:str] sops: kms: [] gcp_kms: [] @@ -21,8 +21,8 @@ sops: dzRUSUZjQ2JpcytuaXF0ZWFlQWZCSUkK+ig7ypo/llaP0Xyd0CaCgHMW4Z05pWMC Xh6jDlRCswRGsLIWLK/nRvQRIArmd4ge1P0WVu+DQAXGL8OtQJ8cZA== -----END AGE ENCRYPTED FILE----- - lastmodified: "2026-09-07T05:41:34Z" - mac: ENC[AES256_GCM,data:F/snASyKPeW5FA4P1a+qfT5qvZ1NSQaa8g/KuEBJzkVia2z/KzHawSBM6Vzrndc96t9SuAz+yFa+9Lh35q90vQc4IxroFr1Kr9FqzuER6b5OrBqehVdQPHTjsoYRKvLHMwOVFb28NeltrSNpYsc+lLK30v6yqG0jVds3laZvAt8=,iv:rHvZt9SGkKJZakVXs/wwPtkcZum8AfmUzCrsZXo6nHY=,tag:yE3JCwCA+nz6Nb90PDQ+Zw==,type:str] + lastmodified: "2026-08-29T03:31:48Z" + mac: ENC[AES256_GCM,data:KtqFiWAYimikxjnUl+oFxSR8tyMr5UwRIlAYte4ZtekLCCJYZ1M389HckAdkgp+r+R/481rghhI4MswtokV30WH3t+nkVVyPB92qRGaHFpdwq/248dXj7WUizyRUoW453+/w/EJhnZzBaN8l2XOAVFj8mM3m/fezzyk82odnsPQ=,iv:EWixHOIwDhYq/BxVRjS1te8k02g6gMtwpmXcPnac1PE=,tag:OtedEDOPxE9qG7r0IK/MIg==,type:str] pgp: [] encrypted_regex: ^(data|stringData)$ version: 3.9.4