Merge: MinIO S3 backup target on Proxmox (ansible role + live) + SOPS creds

infra/ansible/inventory/group_vars/proxmox_hosts/README.md

@@ -0,0 +1,29 @@
+# group_vars for the `proxmox_hosts` group (the Proxmox hypervisor host)
+
+Target of `playbooks/backup-minio.yml` (role `minio_backup_target`). The role needs the MinIO
+credentials as **ansible-vault-encrypted** vars here — encrypted, so safe to commit; the passphrase
+that decrypts them stays out of the repo (`ansible.cfg` → `vault_password_file`).
+
+## `vault.yml` (committed, encrypted)
+
+```bash
+cd infra/ansible
+ansible-vault create inventory/group_vars/proxmox_hosts/vault.yml
+# contents:
+#   minio_backup_target_root_user:     "…"   # MinIO root (host admin only)
+#   minio_backup_target_root_password: "…"
+#   minio_backup_target_access_key:    "k3s-backup"   # scoped key handed to Velero + CNPG
+#   minio_backup_target_secret_key:    "…"
+```
+
+The **scoped** access key/secret is mirrored (SOPS-encrypted) for the GitOps side at
+`platform/backup/minio-credentials.sops.yaml` — the two MUST stay in sync (same key/secret), since
+the cluster authenticates to this MinIO with them. Rotating means updating BOTH and re-running the
+playbook (recreates the MinIO user) + reconciling Flux.
+
+## Notes
+- This is the **Ansible-bootstrap-layer** secret (host credential), distinct from the platform's
+  in-cluster **SOPS + age** secret (ADR-0007) — different layer, different tool, same values for the
+  scoped key.
+- `vault.yml` is a vault blob (not parseable YAML) — excluded from the yamllint gate like the other
+  `**/vault.yml`.

infra/ansible/inventory/group_vars/proxmox_hosts/vault.yml

@@ -0,0 +1,29 @@
+$ANSIBLE_VAULT;1.1;AES256
+30633837396565613861623931383238383836356238326436633738613935326461346138373432
+6234643535303037313337363463633765326239376238380a333331333136396239313733366163
+33353036366461393530633139636430313736366338313331353163616135303930323038636134
+6366353362313537330a373863623861663537353539376664306338643539346139396130396632
+30336231363731616334383863326638636238323565316364303530396636373163393765376438
+63373263623063656662353434623339316238663535386465653164326236616265616638656639
+30613466656439396534316339646139343334656165323965323965313361633933633038376663
+31653539653763623132313533396333643964373032313333613432383433626338376431323731
+63316163363361303439313266356361613737333066643931656564633038306236363439666435
+39343536656336343834663464356566373564636531343534343365376332343832343235663339
+66383561306533323938613738376632626263306337346532643063373166333838333033346365
+32383136363636643163613334366631653238613166303962656463363663643830343531363533
+39333630323765643963396562363537396564666462306163336366643739313035636633646661
+61653637376535393034383631303563363539633265333161363334616239323866353162636632
+65663830323465393138306361333662383934346438663562316235353035633064643861353334
+63646435653639316439366139316539653833656561646439323766633139306565383565656261
+61323230306635343236646438336165643737626132613962633930653461393131303731623438
+63313265653166643334396461353231343235386434363438623034323762653433303932326630
+62633464316139353433366338313837646162323231363135353964623035366132643332366662
+38393465326363616139376465383738626636393639363338653437323832613439313138353033
+39393765613735666331636230363463306462373238623138316263326339366330333332643833
+33363764303939326662343665643963333463356638613231363834386132666339393132356538
+38303362643337366430303165633432666632343536383366653836663135383336316232393936
+66626632386537333739313432313564633866383132353862626433626366633562346464613066
+35376332303132326438313132303264313763376163396137356132306131373335346266646532
+31666230616461386339366461653661323138353662373866316639626134326339643231613337
+30343361643166353162363866653061313966663435373332353562656164663962326562333038
+35363834656361393930

infra/ansible/inventory/hosts.ini

@@ -37,3 +37,11 @@ localhost ansible_connection=local
 # with their ansible_host under k3s_secondary; this group only tags the egress role target.
 [vpn_egress_gateway]
 k3s-dev-3
+
+# The Proxmox HYPERVISOR host itself (NOT a k3s node) — target of playbooks/backup-minio.yml,
+# which stands up the off-cluster MinIO S3 backup target for Velero + CNPG. Reached DIRECTLY
+# over SSH as root (its own 192.168.2.102:22, the pfSense NAT to the host's 192.168.1.9), so
+# it overrides the cluster-wide `ansible_user: k3s` from group_vars/all.yml. Distinct from the
+# `hypervisor` group above (that is `localhost` for host-prereqs.yml, a different machine).
+[proxmox_hosts]
+proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh_private_key_file=~/.ssh/home_network

infra/ansible/playbooks/backup-minio.yml

@@ -0,0 +1,20 @@
+---
+# Stand up the off-cluster S3 backup target (MinIO) on the Proxmox host — the store Velero
+# (R-STATE-03) and CloudNativePG (R-IDP-05) back up into. Host infra, additive + idempotent:
+# ONE new ZFS dataset (Storage/k3s-backups), the minio user/service/binaries, and the
+# velero + cnpg buckets with a scoped non-root access key. It never touches existing
+# datasets, sharenfs, pvesm storage, VMs, or firewall.
+#
+# Credentials come from Ansible vault (inventory/group_vars/proxmox_hosts/vault.yml):
+#   minio_backup_target_root_user/_root_password, minio_backup_target_access_key/_secret_key
+# The scoped key is ALSO staged for the GitOps side (SOPS) at
+#   platform/backup/minio-credentials.sops.yaml
+#
+# Prereqs:  ansible-galaxy collection install -r collections/requirements.yml   (community.general.zfs)
+# Run:      ansible-playbook playbooks/backup-minio.yml
+- name: Provision the MinIO S3 backup target on the Proxmox host
+  hosts: proxmox_hosts
+  become: true
+  gather_facts: true          # need ansible_facts.all_ipv4_addresses to discover the LAN bind addr
+  roles:
+    - role: minio_backup_target

infra/ansible/roles/minio_backup_target/README.md

@@ -0,0 +1,46 @@
+# roles/minio_backup_target
+
+Stands up a **single-node, single-drive MinIO** S3 endpoint on the **Proxmox host** to serve as the
+off-cluster **backup target** for the k3s platform's **Velero** (volume/resource DR, R-STATE-03) and
+**CloudNativePG** (Keycloak DB Barman object store, R-IDP-05). A backup target — not primary storage —
+so a single drive is intentional.
+
+## What it does (all additive + idempotent)
+
+1. **ZFS dataset** `Storage/k3s-backups` — `compression=zstd`, `quota=100G`,
+   `mountpoint=/mnt/Storage/k3s-backups`, **no sharenfs**. Uses `community.general.zfs` so re-runs are
+   no-ops and existing datasets are never touched.
+2. **`minio` system user/group** + a data dir (`…/k3s-backups/data`) it owns.
+3. **MinIO + `mc` binaries** (pinned, checksummed). The Proxmox host has **no outbound internet**, so
+   the binaries are downloaded on the **Ansible controller** and copied to the host.
+4. **systemd unit** `minio.service` (Type=notify, hardened), bound to the host's
+   **cluster-reachable `192.168.1.x` address** on port **9000** (console 9001) — auto-discovered from
+   facts, assert-guarded.
+5. **Buckets** `velero` + `cnpg`, a **scoped non-root access key** limited by an IAM policy to just
+   those two buckets, plus the retained root creds for host-side admin.
+
+## Credentials
+
+Supplied via the vaulted `inventory/group_vars/proxmox_hosts/vault.yml` (ansible-vault). Nothing is
+defaulted — a missing vault fails the play loudly:
+
+| var | purpose |
+|---|---|
+| `minio_backup_target_root_user` / `_root_password` | MinIO root (host admin only) |
+| `minio_backup_target_access_key` / `_secret_key` | scoped key handed to Velero + CNPG |
+
+The **same scoped key** is staged for the GitOps side, SOPS-encrypted, at
+`platform/backup/minio-credentials.sops.yaml`.
+
+## Run
+
+```bash
+cd infra/ansible
+ansible-galaxy collection install -r collections/requirements.yml   # community.general (zfs)
+ansible-playbook playbooks/backup-minio.yml
+```
+
+## Endpoint
+
+`http://<host-192.168.1.x>:9000` — the S3 URL the cluster uses (path-style). Health:
+`GET /minio/health/ready` ⇒ 200. Verified reachable from an in-cluster pod at bring-up.

infra/ansible/roles/minio_backup_target/defaults/main.yml

@@ -0,0 +1,62 @@
+---
+# roles/minio_backup_target — a single-node, single-drive MinIO S3 target on the Proxmox
+# host, used ONLY as an off-cluster backup bucket store for Velero + CloudNativePG (ADR-0003,
+# R-STATE-03 / R-IDP-05). This is a BACKUP TARGET, not primary storage — single drive is fine.
+#
+# SAFETY: every task in this role is additive and idempotent. It creates ONE new ZFS dataset,
+# ONE system user, ONE systemd service and two buckets. It never touches existing datasets,
+# sharenfs, pvesm storage, VMs or firewall. See tasks/main.yml for the guards.
+
+# ── MinIO / mc release pins (Renovate-friendly; both are AGPL community builds) ───────────────
+# The Proxmox host has NO outbound internet (PiHole-only DNS, no default route to WAN), so the
+# binaries are fetched on the ANSIBLE CONTROLLER and copied to the host (tasks/main.yml).
+minio_backup_target_minio_version: "RELEASE.2025-09-07T16-13-09Z"
+minio_backup_target_minio_sha256: "7c5bd8512c6e966455b1d198209358b2d191c77a83ab377c4073281065fb855f"
+minio_backup_target_minio_url: "https://dl.min.io/server/minio/release/linux-amd64/archive/minio.{{ minio_backup_target_minio_version }}"
+
+minio_backup_target_mc_version: "RELEASE.2025-08-13T08-35-41Z"
+minio_backup_target_mc_sha256: "01f866e9c5f9b87c2b09116fa5d7c06695b106242d829a8bb32990c00312e891"
+minio_backup_target_mc_url: "https://dl.min.io/client/mc/release/linux-amd64/archive/mc.{{ minio_backup_target_mc_version }}"
+
+# Controller-side cache the binaries land in before being copied to the host (delegate_to localhost).
+minio_backup_target_controller_cache: "/tmp/minio-backup-target-cache"
+
+# ── ZFS dataset (backup drive) ────────────────────────────────────────────────────────────────
+# Host convention is Storage/<purpose> mounted at /mnt/Storage/<purpose>. NO sharenfs.
+minio_backup_target_zfs_dataset: "Storage/k3s-backups"
+minio_backup_target_mountpoint: "/mnt/Storage/k3s-backups"
+minio_backup_target_zfs_compression: "zstd"
+minio_backup_target_zfs_quota: "100G"
+# MinIO writes its drive (incl. .minio.sys) into a subdir of the dataset, keeping the dataset
+# root clean and owned by root.
+minio_backup_target_data_dir: "{{ minio_backup_target_mountpoint }}/data"
+
+# ── Service identity ────────────────────────────────────────────────────────────────────────
+minio_backup_target_user: "minio"
+minio_backup_target_group: "minio"
+minio_backup_target_minio_bin: "/usr/local/bin/minio"
+minio_backup_target_mc_bin: "/usr/local/bin/mc"
+minio_backup_target_env_file: "/etc/default/minio"
+
+# ── Network binding ──────────────────────────────────────────────────────────────────────────
+# MinIO's S3 API must bind the host address the k3s pods can reach (the 192.168.1.0/24 LAN;
+# nodes are .50/.51/.52). tasks/main.yml auto-discovers the host's address in this CIDR prefix
+# and asserts it exists; override minio_backup_target_bind_address to pin it explicitly.
+minio_backup_target_lan_prefix: "192.168.1."
+minio_backup_target_bind_address: ""          # "" ⇒ auto-discover from facts (see tasks)
+minio_backup_target_api_port: 9000
+minio_backup_target_console_port: 9001
+
+# ── Buckets + scoped (non-root) access key ────────────────────────────────────────────────────
+# Velero and CNPG each get a bucket; a single non-root user scoped to ONLY these two buckets is
+# handed to the cluster (SOPS). Root creds stay on the host for admin.
+minio_backup_target_buckets:
+  - velero
+  - cnpg
+minio_backup_target_policy_name: "k3s-backup-rw"
+
+# Credentials come from the vaulted group_vars (inventory/group_vars/proxmox_hosts/vault.yml):
+#   minio_backup_target_root_user, minio_backup_target_root_password,
+#   minio_backup_target_access_key, minio_backup_target_secret_key
+# They are intentionally NOT defaulted here so a missing vault fails loudly rather than
+# silently provisioning guessable creds.

infra/ansible/roles/minio_backup_target/handlers/main.yml

@@ -0,0 +1,6 @@
+---
+- name: Restart minio
+  ansible.builtin.systemd:
+    name: minio
+    state: restarted
+    daemon_reload: true

infra/ansible/roles/minio_backup_target/tasks/main.yml

@@ -0,0 +1,263 @@
+---
+# ── 0. Preconditions / safety asserts ─────────────────────────────────────────────────────────
+- name: Assert MinIO credentials are supplied (from vaulted group_vars)
+  ansible.builtin.assert:
+    that:
+      - minio_backup_target_root_user | default('') | length > 0
+      - minio_backup_target_root_password | default('') | length >= 8
+      - minio_backup_target_access_key | default('') | length > 0
+      - minio_backup_target_secret_key | default('') | length >= 8
+    fail_msg: >-
+      MinIO credentials are missing. Provide minio_backup_target_root_user/_password and
+      minio_backup_target_access_key/_secret_key via inventory/group_vars/proxmox_hosts/vault.yml.
+    quiet: true
+
+# Determine the host's 192.168.1.0/24 address (the S3 endpoint the cluster reaches). If the
+# operator pinned minio_backup_target_bind_address, honour it; else auto-discover and assert.
+- name: Discover the host LAN (cluster-reachable) IPv4 address
+  ansible.builtin.set_fact:
+    minio_backup_target_bind_address: >-
+      {{ minio_backup_target_bind_address
+         if (minio_backup_target_bind_address | length > 0)
+         else (ansible_facts.all_ipv4_addresses
+               | select('match', '^' ~ (minio_backup_target_lan_prefix | regex_escape))
+               | list | first | default('')) }}
+
+- name: Assert a cluster-reachable LAN address was found
+  ansible.builtin.assert:
+    that:
+      - minio_backup_target_bind_address | length > 0
+    fail_msg: >-
+      Could not find a {{ minio_backup_target_lan_prefix }}x address on this host
+      (interfaces: {{ ansible_facts.all_ipv4_addresses | default([]) | join(', ') }}).
+      The cluster egresses onto that LAN — refusing to bind MinIO to an unreachable address.
+      Set minio_backup_target_bind_address explicitly if discovery is wrong.
+    quiet: true
+
+# ── 1. ZFS dataset (backup drive) — idempotent, never touches other datasets ───────────────────
+- name: Create/ensure the k3s-backups ZFS dataset (compression, quota, mountpoint; NO sharenfs)
+  community.general.zfs:
+    name: "{{ minio_backup_target_zfs_dataset }}"
+    state: present
+    extra_zfs_properties:
+      compression: "{{ minio_backup_target_zfs_compression }}"
+      quota: "{{ minio_backup_target_zfs_quota }}"
+      mountpoint: "{{ minio_backup_target_mountpoint }}"
+
+# ── 2. Service identity ───────────────────────────────────────────────────────────────────────
+- name: Create the minio system group
+  ansible.builtin.group:
+    name: "{{ minio_backup_target_group }}"
+    system: true
+
+- name: Create the minio system user (no login shell, home = data dir)
+  ansible.builtin.user:
+    name: "{{ minio_backup_target_user }}"
+    group: "{{ minio_backup_target_group }}"
+    system: true
+    shell: /usr/sbin/nologin
+    home: "{{ minio_backup_target_data_dir }}"
+    create_home: false
+
+- name: Ensure the MinIO data directory exists (owned by minio, inside the dataset)
+  ansible.builtin.file:
+    path: "{{ minio_backup_target_data_dir }}"
+    state: directory
+    owner: "{{ minio_backup_target_user }}"
+    group: "{{ minio_backup_target_group }}"
+    mode: "0750"
+
+# ── 3. Binaries — fetched on the CONTROLLER (host has no internet), copied to the host ──────────
+- name: Ensure controller-side download cache exists
+  ansible.builtin.file:
+    path: "{{ minio_backup_target_controller_cache }}"
+    state: directory
+    mode: "0700"
+  delegate_to: localhost
+  become: false
+  run_once: true
+
+- name: Fetch MinIO server binary onto the controller (checksummed)
+  ansible.builtin.get_url:
+    url: "{{ minio_backup_target_minio_url }}"
+    dest: "{{ minio_backup_target_controller_cache }}/minio.{{ minio_backup_target_minio_version }}"
+    checksum: "sha256:{{ minio_backup_target_minio_sha256 }}"
+    mode: "0755"
+  delegate_to: localhost
+  become: false
+  run_once: true
+
+- name: Fetch mc client binary onto the controller (checksummed)
+  ansible.builtin.get_url:
+    url: "{{ minio_backup_target_mc_url }}"
+    dest: "{{ minio_backup_target_controller_cache }}/mc.{{ minio_backup_target_mc_version }}"
+    checksum: "sha256:{{ minio_backup_target_mc_sha256 }}"
+    mode: "0755"
+  delegate_to: localhost
+  become: false
+  run_once: true
+
+# Integrity is already guaranteed by the SHA256 verification get_url performed on the controller
+# above; copy transfers that verified artifact faithfully. (The copy module's own `checksum` arg is
+# SHA1, not SHA256 — deliberately not used here to avoid a false mismatch.)
+- name: Install MinIO server binary on the host
+  ansible.builtin.copy:
+    src: "{{ minio_backup_target_controller_cache }}/minio.{{ minio_backup_target_minio_version }}"
+    dest: "{{ minio_backup_target_minio_bin }}"
+    owner: root
+    group: root
+    mode: "0755"
+  notify: Restart minio
+
+- name: Install mc client binary on the host
+  ansible.builtin.copy:
+    src: "{{ minio_backup_target_controller_cache }}/mc.{{ minio_backup_target_mc_version }}"
+    dest: "{{ minio_backup_target_mc_bin }}"
+    owner: root
+    group: root
+    mode: "0755"
+
+# ── 4. systemd service ────────────────────────────────────────────────────────────────────────
+- name: Write the MinIO environment file (root creds, bind address, data dir)
+  ansible.builtin.template:
+    src: minio.env.j2
+    dest: "{{ minio_backup_target_env_file }}"
+    owner: root
+    group: "{{ minio_backup_target_group }}"
+    mode: "0640"
+  no_log: true
+  notify: Restart minio
+
+- name: Install the MinIO systemd unit
+  ansible.builtin.template:
+    src: minio.service.j2
+    dest: /etc/systemd/system/minio.service
+    owner: root
+    group: root
+    mode: "0644"
+  notify: Restart minio
+
+- name: Enable + start MinIO (apply any pending unit/env changes now)
+  ansible.builtin.systemd:
+    name: minio
+    enabled: true
+    state: started
+    daemon_reload: true
+
+- name: Flush handlers so MinIO is (re)started before we configure buckets/users
+  ansible.builtin.meta: flush_handlers
+
+# ── 5. Readiness gate ─────────────────────────────────────────────────────────────────────────
+- name: Wait for the MinIO S3 health endpoint to report ready
+  ansible.builtin.uri:
+    url: "http://{{ minio_backup_target_bind_address }}:{{ minio_backup_target_api_port }}/minio/health/ready"
+    status_code: 200
+  register: minio_backup_target_health
+  retries: 30
+  delay: 2
+  until: minio_backup_target_health.status == 200
+
+# ── 6. Buckets + scoped non-root user (via mc, idempotent) ─────────────────────────────────────
+- name: Configure the local mc alias against the running MinIO
+  ansible.builtin.command:
+    argv:
+      - "{{ minio_backup_target_mc_bin }}"
+      - alias
+      - set
+      - k3sbackup
+      - "http://{{ minio_backup_target_bind_address }}:{{ minio_backup_target_api_port }}"
+      - "{{ minio_backup_target_root_user }}"
+      - "{{ minio_backup_target_root_password }}"
+  no_log: true
+  changed_when: false
+
+- name: Create the Velero + CNPG buckets (idempotent)
+  ansible.builtin.command:
+    argv:
+      - "{{ minio_backup_target_mc_bin }}"
+      - mb
+      - --ignore-existing
+      - "k3sbackup/{{ item }}"
+  loop: "{{ minio_backup_target_buckets }}"
+  register: minio_backup_target_mb
+  changed_when: "'created successfully' in (minio_backup_target_mb.stdout | default(''))"
+
+- name: Render the scoped bucket RW policy to the host
+  ansible.builtin.template:
+    src: bucket-policy.json.j2
+    dest: "/etc/minio-{{ minio_backup_target_policy_name }}.json"
+    owner: root
+    group: "{{ minio_backup_target_group }}"
+    mode: "0640"
+
+- name: Check whether the scoped policy already exists
+  ansible.builtin.command:
+    argv:
+      - "{{ minio_backup_target_mc_bin }}"
+      - admin
+      - policy
+      - info
+      - k3sbackup
+      - "{{ minio_backup_target_policy_name }}"
+  register: minio_backup_target_policy_info
+  changed_when: false
+  failed_when: false
+
+- name: Create the scoped bucket RW policy (velero + cnpg only)
+  ansible.builtin.command:
+    argv:
+      - "{{ minio_backup_target_mc_bin }}"
+      - admin
+      - policy
+      - create
+      - k3sbackup
+      - "{{ minio_backup_target_policy_name }}"
+      - "/etc/minio-{{ minio_backup_target_policy_name }}.json"
+  when: minio_backup_target_policy_info.rc != 0
+  changed_when: true
+
+- name: Check whether the scoped access key already exists
+  ansible.builtin.command:
+    argv:
+      - "{{ minio_backup_target_mc_bin }}"
+      - admin
+      - user
+      - info
+      - k3sbackup
+      - "{{ minio_backup_target_access_key }}"
+  register: minio_backup_target_user_info
+  changed_when: false
+  failed_when: false
+
+- name: Create the scoped non-root access key
+  ansible.builtin.command:
+    argv:
+      - "{{ minio_backup_target_mc_bin }}"
+      - admin
+      - user
+      - add
+      - k3sbackup
+      - "{{ minio_backup_target_access_key }}"
+      - "{{ minio_backup_target_secret_key }}"
+  when: minio_backup_target_user_info.rc != 0
+  no_log: true
+  changed_when: true
+
+- name: Attach the scoped policy to the access key (idempotent)
+  ansible.builtin.command:
+    argv:
+      - "{{ minio_backup_target_mc_bin }}"
+      - admin
+      - policy
+      - attach
+      - k3sbackup
+      - "{{ minio_backup_target_policy_name }}"
+      - --user
+      - "{{ minio_backup_target_access_key }}"
+  register: minio_backup_target_attach
+  changed_when:
+    - minio_backup_target_attach.rc == 0
+    - "'already in effect' not in (minio_backup_target_attach.stderr | default(''))"
+  failed_when:
+    - minio_backup_target_attach.rc != 0
+    - "'already in effect' not in (minio_backup_target_attach.stderr | default(''))"

infra/ansible/roles/minio_backup_target/templates/bucket-policy.json.j2

@@ -0,0 +1,15 @@
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": ["s3:*"],
+      "Resource": [
+{% for b in minio_backup_target_buckets %}
+        "arn:aws:s3:::{{ b }}",
+        "arn:aws:s3:::{{ b }}/*"{{ "," if not loop.last else "" }}
+{% endfor %}
+      ]
+    }
+  ]
+}

infra/ansible/roles/minio_backup_target/templates/minio.env.j2

@@ -0,0 +1,11 @@
+# {{ ansible_managed }}
+# MinIO server environment (single-node, single-drive backup target for k3s Velero + CNPG).
+# Managed by roles/minio_backup_target — do not edit by hand.
+MINIO_ROOT_USER={{ minio_backup_target_root_user }}
+MINIO_ROOT_PASSWORD={{ minio_backup_target_root_password }}
+
+# The S3 drive lives on the ZFS dataset {{ minio_backup_target_zfs_dataset }}.
+MINIO_VOLUMES="{{ minio_backup_target_data_dir }}"
+
+# Bind the S3 API (and console) to the cluster-reachable LAN address only.
+MINIO_OPTS="--address {{ minio_backup_target_bind_address }}:{{ minio_backup_target_api_port }} --console-address {{ minio_backup_target_bind_address }}:{{ minio_backup_target_console_port }}"

infra/ansible/roles/minio_backup_target/templates/minio.service.j2

@@ -0,0 +1,28 @@
+{# {{ ansible_managed }} — MinIO systemd unit (roles/minio_backup_target) #}
+[Unit]
+Description=MinIO S3 backup target (k3s Velero + CNPG)
+Documentation=https://min.io/docs/minio/linux/index.html
+After=network-online.target
+Wants=network-online.target
+AssertFileIsExecutable={{ minio_backup_target_minio_bin }}
+
+[Service]
+Type=notify
+User={{ minio_backup_target_user }}
+Group={{ minio_backup_target_group }}
+EnvironmentFile={{ minio_backup_target_env_file }}
+ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo 'MINIO_VOLUMES unset in {{ minio_backup_target_env_file }}'; exit 1; fi"
+ExecStart={{ minio_backup_target_minio_bin }} server $MINIO_OPTS $MINIO_VOLUMES
+Restart=always
+RestartSec=5s
+LimitNOFILE=65536
+TimeoutStopSec=30s
+# Hardening (backup target holds no other roles on this host).
+NoNewPrivileges=true
+ProtectSystem=full
+ProtectHome=true
+PrivateTmp=true
+ReadWritePaths={{ minio_backup_target_mountpoint }}
+
+[Install]
+WantedBy=multi-user.target

platform/backup/minio-credentials.sops.yaml

@@ -0,0 +1,41 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: minio-backup-credentials
+    namespace: backup
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:V1G4ViUPhrnkiRLFNZyYbSaEPFVLsZ74gB4zo1Y7AM2QP7R5lWVEvDO9VgLVatRZwt7cbeObMAYXeI873tZ9hOi2JNSvYuuXrscC3q8nMuJGswvIMvkM,iv:sGb4E/j5mhtkwY05YP/U7lCa9UHV/35dGCziymGUJHo=,tag:S5KnPKRaxJuGCMwPZkIFBw==,type:comment]
+    S3_ENDPOINT: ENC[AES256_GCM,data:UwzKnjDVv3HoEeAPxlzkhUifL6hmebM=,iv:IWmiCYrvuIBez2i3qyDs6Yu4mQ0YuNYv46Rn3+ksm/k=,tag:dlefmBsY3TuLZPAPmaWs/g==,type:str]
+    S3_REGION: ENC[AES256_GCM,data:GvGIw6qxx9M4,iv:uArcS3uwHKwjif00ICIc0Og0OF98OL10ArhT6teJzh4=,tag:2tn4XykslMtwYk7C4m8ndw==,type:str]
+    S3_FORCE_PATH_STYLE: ENC[AES256_GCM,data:5cEjjA==,iv:k5R7iszw/kxYLSH8aqIAHmq8ePdL7xYYPQcutHloN7s=,tag:DJPE0500UP5ECYuBJM9nBg==,type:str]
+    VELERO_BUCKET: ENC[AES256_GCM,data:Q1Oo4Amz,iv:XnAT/Nn8C9BmnE5e0d1LI0I0BVSUL7C+mI+s31FrMYI=,tag:oWDHC2jjKYpKNXA98wYqpQ==,type:str]
+    CNPG_BUCKET: ENC[AES256_GCM,data:P3BUXg==,iv:BPuFgI9iL9lWZhQZ0k3skrBwyp1UDLyz0aIyq6/V7nI=,tag:R4n30UxwcCY1k+X/bu7pxw==,type:str]
+    #ENC[AES256_GCM,data:A6h/rEzdn5Sp1BLDnxoHrYdEwiB9Eb/5NAxCMZRXvoVcnTfOvLHVA0ntno+9GwMueIhwN9WgG7fnCHhG7A==,iv:1+VfHF59PppwpLiYW4gUTwgGcvXMEzAnvowVvE/awFU=,tag:skLHPchOW2PZzjqqaJ2TSQ==,type:comment]
+    AWS_ACCESS_KEY_ID: ENC[AES256_GCM,data:wNnfGYBspJDyYw==,iv:d9fK8E5ujPu9vXV1yRR5AgO3+j69EWfbWEoMt7Lq06w=,tag:0uuRSadwtodLol2Sw2ih9Q==,type:str]
+    AWS_SECRET_ACCESS_KEY: ENC[AES256_GCM,data:IywofASX3m8sznW3StMf/SvCPBcvxCUlT4cIQvu4gF9RglHlFCujwi8amnm7ASfc,iv:JnyMBQQ7oLg8TAVx7RveZHYDo6a/bl3t1IEPRxZufI8=,tag:5rUUqd6ZByuLIlZxQV1dPQ==,type:str]
+    #ENC[AES256_GCM,data:J5fQZE/UbhqpWo5OXzvGxCNTb5f6r+sh55wUHf3M0H4oRU2dyna7nJqFQzP1EQXfJTlxjZOpBkzZyppn,iv:bDGljEMmMiieCvfmVO8uhfw/O1rUslNvNAY6LfFFA4E=,tag:D9Gw78gtCigsWJnlmFzMdA==,type:comment]
+    ACCESS_KEY_ID: ENC[AES256_GCM,data:TYn3AVIkjU4QhA==,iv:09W2JkJywD5aadOr+XkbZlR2sSz3glT+yknI4Z46C9o=,tag:QGwIp3seKuemzi4jysDRcg==,type:str]
+    ACCESS_SECRET_KEY: ENC[AES256_GCM,data:lZ9fAqwPGwiKbTJYyvNGMOS9L37IB3/GcfpuZH3G2s0of3YII6vyPlAKZeGI6i2P,iv:Y0p8EDzZS3uUZ+BBHjjBgDmkyphwZ43X7VfZJboGS1s=,tag:rQ0EyRBEPNG41PGrqHKkhw==,type:str]
+    #ENC[AES256_GCM,data:EbjHG+jG6OnhZzWUAamIomBYYa5jwogBWlAgOPyIdFivuNZGDSbkyZ0dsUNJJ8yhl7lkIjvi1VvGA/B2QXdCw0pP4xxCs84uLSgpHDWB9v4G5cjF+kThANvaMomKoQ==,iv:1JpFJPSTXVxKVkLBkaYZAYzRHCmqSAgNYGmcW4ajZxo=,tag:9vWPXJ3COQC4m2AHCWPOlg==,type:comment]
+    cloud: ENC[AES256_GCM,data:RmDGXXK5ct97WO18YAYIUF3zPE7nb1zkuJV8pBZPkNEHV6y1Iup/RXAMg+l1rXcI7xdPSiMfSJ6FMudfxOzCzAJPRGxoHMWhzO5N9H+O30KOhHwYpeud2mL4xNZPzOvQgpszqJQzk82D5tnjGeE=,iv:wz16XKNzAWrOZv+Rx68MArqRJHZmRVX/613w7/3WXH4=,tag:5cxblz07Mepy9xOZ3pko2w==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA5Z1hkZTlrWmpsdzBuODlM
+            NXQvTXhmb00rMWNqY1l4Ly8yVjVrNXFSMXdRCmxMNXlBTEJyVncrcHVrWXFoZHBI
+            L3JLM3lLVHBJNFVHY3k0MVBjYk5USTgKLS0tIFJ4WWxFL0Flc2J5V3ZSU1JhSVE1
+            TDFyWmhqQUdCOTUyZGVnRDQ4d2lSRnMKPBexDlO9fZK6K6Rpcs2RzYE4drYtivwk
+            QsnSigjEPY35ixsPTMHTdTP0Almxc4U6lhT4gsIEqk3hqmJ3fcmzzA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-08-29T03:09:53Z"
+    mac: ENC[AES256_GCM,data:y1Uxrj9yAuZCryfCviU0pPplcQ0ccxcJSuNov4CvZKwnIimHVIsr7sU1mw3ZqHV/arQ2SjXkfDHxAEhvL6WSAjEjnDNaCMm++mPrfKXoGZhDU3gjaeyI7W6Iexo/bfc/oR6hALRjp6g7nTK6Yqdi6UMRoZZ+1YGSkFkRJjygLF4=,iv:9scdNtJQk6TkB/SN2yeFdhmiVCtGd2qRsD3N+wv0L5o=,tag:dYFy6oKEUIr7BLstFrhsEA==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

platform/backup/minio-credentials.sops.yaml.example

@@ -0,0 +1,51 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
+# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the scoped (non-root) MinIO S3 access key for the off-cluster backup target that
+# roles/minio_backup_target stands up on the Proxmox host (playbooks/backup-minio.yml). It is the
+# credential Velero (R-STATE-03) and CloudNativePG / Barman (R-IDP-05) use to write backups. The
+# SAME key/secret live (ansible-vault) in infra/ansible/inventory/group_vars/proxmox_hosts/vault.yml
+# as minio_backup_target_access_key/_secret_key — the two MUST stay in sync.
+#
+# ENDPOINT (not secret):  http://192.168.1.9:9000   (path-style; region us-east-1 is a placeholder,
+#                         MinIO ignores it but S3 clients require one). Buckets: velero, cnpg.
+#
+# HAND-OFF STEPS (operator):
+#  (a) Copy to:      platform/backup/minio-credentials.sops.yaml
+#  (b) Fill the access key + secret (== the vaulted scoped key on the Proxmox host).
+#  (c) Encrypt:      sops -e -i platform/backup/minio-credentials.sops.yaml
+#  (d) Wire per consumer (WS-STORAGE / WS-SEC):
+#        - Velero  → credentials.existingSecret pointing at a Secret with the `cloud` INI key,
+#                    backupStorageLocation.config.s3Url=S3_ENDPOINT, s3ForcePathStyle="true",
+#                    bucket=velero (namespace: velero).
+#        - CNPG    → Barman objectStore endpointURL=S3_ENDPOINT, destinationPath=s3://cnpg,
+#                    s3Credentials.accessKeyId/secretAccessKey from ACCESS_KEY_ID/ACCESS_SECRET_KEY
+#                    (namespace: cnpg-system).
+#      A Secret is namespace-scoped — replicate/rename this one into each consumer namespace as
+#      that wiring requires.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+    name: minio-backup-credentials
+    namespace: backup
+type: Opaque
+stringData:
+    # ── endpoint + buckets (not secret; single source of truth) ──
+    S3_ENDPOINT: "http://192.168.1.9:9000"
+    S3_REGION: "us-east-1"
+    S3_FORCE_PATH_STYLE: "true"
+    VELERO_BUCKET: "velero"
+    CNPG_BUCKET: "cnpg"
+    # ── scoped access key (velero + cnpg buckets only) ──
+    AWS_ACCESS_KEY_ID: "REPLACE-ME"
+    AWS_SECRET_ACCESS_KEY: "REPLACE-ME"
+    # CNPG Barman object store consumes these discrete key names.
+    ACCESS_KEY_ID: "REPLACE-ME"
+    ACCESS_SECRET_KEY: "REPLACE-ME"
+    # Velero's aws plugin reads an INI credentials file from the `cloud` key of its existingSecret.
+    cloud: |
+        [default]
+        aws_access_key_id=REPLACE-ME
+        aws_secret_access_key=REPLACE-ME