Replace MinIO backup target with versitygw (s3_backup_target role)

MinIO's open-source edition is de-prioritised (console removed, commercial
pivot), so swap the off-cluster S3 backup target on the Proxmox host to
versitygw v1.7.0 (github.com/versity/versitygw) — a maintained Go S3 gateway
serving a POSIX filesystem as S3.

Rename role minio_backup_target -> s3_backup_target and playbook backup-minio
-> backup-s3. The role now:
  - decommissions any pre-existing MinIO (service/binary/env/mc-client/user/
    object-layout), KEEPING the ZFS dataset Storage/k3s-backups and its mount
  - installs the pinned, checksummed versitygw binary (fetched+extracted on the
    controller, host has no internet), a dedicated versitygw system user, and a
    Type=simple hardened systemd unit running the posix backend rooted at
    /mnt/Storage/k3s-backups/data, bound to 192.168.1.9:9000 (same endpoint),
    plain HTTP / path-style
  - ensures the dataset has xattr=sa (versitygw stores object metadata in xattrs)
  - pre-creates the velero + cnpg buckets (bucket == top-level subdir)

Credentials are REUSED: versitygw's ROOT key is set to the same scoped
access_key/secret the cluster already uses, so endpoint + key + secret + bucket
names are byte-identical to the old MinIO target. The GitOps SOPS secret
(platform/backup/minio-credentials.sops.yaml) and all cluster-side Velero/CNPG
config need ZERO change. The vaulted group_vars now expose
s3_backup_target_access_key/_secret_key (same values, MinIO root creds retired).

Verified live on the Proxmox host: versitygw active+enabled on 192.168.1.9:9000,
MinIO fully removed, dataset intact; SigV4 ListBuckets with the reused key
returns velero+cnpg; a throwaway in-cluster pod reaches http://192.168.1.9:9000
(versitygw S3 response). Re-run is idempotent (no host changes).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-28 22:48:38 -05:00
1 parent 1bee2554f8
commit 6739339f53
17 files changed
+499 -491

No files matched your search

@@ -1,46 +0,0 @@
# roles/minio_backup_target
Stands up a **single-node, single-drive MinIO** S3 endpoint on the **Proxmox host** to serve as the
off-cluster **backup target** for the k3s platform's **Velero** (volume/resource DR, R-STATE-03) and
**CloudNativePG** (Keycloak DB Barman object store, R-IDP-05). A backup target — not primary storage —
so a single drive is intentional.
## What it does (all additive + idempotent)
1. **ZFS dataset** `Storage/k3s-backups` — `compression=zstd`, `quota=100G`,
`mountpoint=/mnt/Storage/k3s-backups`, **no sharenfs**. Uses `community.general.zfs` so re-runs are
no-ops and existing datasets are never touched.
2. **`minio` system user/group** + a data dir (`…/k3s-backups/data`) it owns.
3. **MinIO + `mc` binaries** (pinned, checksummed). The Proxmox host has **no outbound internet**, so
the binaries are downloaded on the **Ansible controller** and copied to the host.
4. **systemd unit** `minio.service` (Type=notify, hardened), bound to the host's
**cluster-reachable `192.168.1.x` address** on port **9000** (console 9001) — auto-discovered from
facts, assert-guarded.
5. **Buckets** `velero` + `cnpg`, a **scoped non-root access key** limited by an IAM policy to just
those two buckets, plus the retained root creds for host-side admin.
## Credentials
Supplied via the vaulted `inventory/group_vars/proxmox_hosts/vault.yml` (ansible-vault). Nothing is
defaulted — a missing vault fails the play loudly:
| var | purpose |
|---|---|
| `minio_backup_target_root_user` / `_root_password` | MinIO root (host admin only) |
| `minio_backup_target_access_key` / `_secret_key` | scoped key handed to Velero + CNPG |
The **same scoped key** is staged for the GitOps side, SOPS-encrypted, at
`platform/backup/minio-credentials.sops.yaml`.
## Run
```bash
cd infra/ansible
ansible-galaxy collection install -r collections/requirements.yml # community.general (zfs)
ansible-playbook playbooks/backup-minio.yml
```
## Endpoint
`http://<host-192.168.1.x>:9000` — the S3 URL the cluster uses (path-style). Health:
`GET /minio/health/ready` ⇒ 200. Verified reachable from an in-cluster pod at bring-up.
@@ -1,62 +0,0 @@
---
# roles/minio_backup_target — a single-node, single-drive MinIO S3 target on the Proxmox
# host, used ONLY as an off-cluster backup bucket store for Velero + CloudNativePG (ADR-0003,
# R-STATE-03 / R-IDP-05). This is a BACKUP TARGET, not primary storage — single drive is fine.
#
# SAFETY: every task in this role is additive and idempotent. It creates ONE new ZFS dataset,
# ONE system user, ONE systemd service and two buckets. It never touches existing datasets,
# sharenfs, pvesm storage, VMs or firewall. See tasks/main.yml for the guards.
# ── MinIO / mc release pins (Renovate-friendly; both are AGPL community builds) ───────────────
# The Proxmox host has NO outbound internet (PiHole-only DNS, no default route to WAN), so the
# binaries are fetched on the ANSIBLE CONTROLLER and copied to the host (tasks/main.yml).
minio_backup_target_minio_version: "RELEASE.2025-09-07T16-13-09Z"
minio_backup_target_minio_sha256: "7c5bd8512c6e966455b1d198209358b2d191c77a83ab377c4073281065fb855f"
minio_backup_target_minio_url: "https://dl.min.io/server/minio/release/linux-amd64/archive/minio.{{ minio_backup_target_minio_version }}"
minio_backup_target_mc_version: "RELEASE.2025-08-13T08-35-41Z"
minio_backup_target_mc_sha256: "01f866e9c5f9b87c2b09116fa5d7c06695b106242d829a8bb32990c00312e891"
minio_backup_target_mc_url: "https://dl.min.io/client/mc/release/linux-amd64/archive/mc.{{ minio_backup_target_mc_version }}"
# Controller-side cache the binaries land in before being copied to the host (delegate_to localhost).
minio_backup_target_controller_cache: "/tmp/minio-backup-target-cache"
# ── ZFS dataset (backup drive) ────────────────────────────────────────────────────────────────
# Host convention is Storage/<purpose> mounted at /mnt/Storage/<purpose>. NO sharenfs.
minio_backup_target_zfs_dataset: "Storage/k3s-backups"
minio_backup_target_mountpoint: "/mnt/Storage/k3s-backups"
minio_backup_target_zfs_compression: "zstd"
minio_backup_target_zfs_quota: "100G"
# MinIO writes its drive (incl. .minio.sys) into a subdir of the dataset, keeping the dataset
# root clean and owned by root.
minio_backup_target_data_dir: "{{ minio_backup_target_mountpoint }}/data"
# ── Service identity ────────────────────────────────────────────────────────────────────────
minio_backup_target_user: "minio"
minio_backup_target_group: "minio"
minio_backup_target_minio_bin: "/usr/local/bin/minio"
minio_backup_target_mc_bin: "/usr/local/bin/mc"
minio_backup_target_env_file: "/etc/default/minio"
# ── Network binding ──────────────────────────────────────────────────────────────────────────
# MinIO's S3 API must bind the host address the k3s pods can reach (the 192.168.1.0/24 LAN;
# nodes are .50/.51/.52). tasks/main.yml auto-discovers the host's address in this CIDR prefix
# and asserts it exists; override minio_backup_target_bind_address to pin it explicitly.
minio_backup_target_lan_prefix: "192.168.1."
minio_backup_target_bind_address: "" # "" ⇒ auto-discover from facts (see tasks)
minio_backup_target_api_port: 9000
minio_backup_target_console_port: 9001
# ── Buckets + scoped (non-root) access key ────────────────────────────────────────────────────
# Velero and CNPG each get a bucket; a single non-root user scoped to ONLY these two buckets is
# handed to the cluster (SOPS). Root creds stay on the host for admin.
minio_backup_target_buckets:
- velero
- cnpg
minio_backup_target_policy_name: "k3s-backup-rw"
# Credentials come from the vaulted group_vars (inventory/group_vars/proxmox_hosts/vault.yml):
# minio_backup_target_root_user, minio_backup_target_root_password,
# minio_backup_target_access_key, minio_backup_target_secret_key
# They are intentionally NOT defaulted here so a missing vault fails loudly rather than
# silently provisioning guessable creds.
@@ -1,263 +0,0 @@
---
# ── 0. Preconditions / safety asserts ─────────────────────────────────────────────────────────
- name: Assert MinIO credentials are supplied (from vaulted group_vars)
ansible.builtin.assert:
that:
- minio_backup_target_root_user | default('') | length > 0
- minio_backup_target_root_password | default('') | length >= 8
- minio_backup_target_access_key | default('') | length > 0
- minio_backup_target_secret_key | default('') | length >= 8
fail_msg: >-
MinIO credentials are missing. Provide minio_backup_target_root_user/_password and
minio_backup_target_access_key/_secret_key via inventory/group_vars/proxmox_hosts/vault.yml.
quiet: true
# Determine the host's 192.168.1.0/24 address (the S3 endpoint the cluster reaches). If the
# operator pinned minio_backup_target_bind_address, honour it; else auto-discover and assert.
- name: Discover the host LAN (cluster-reachable) IPv4 address
ansible.builtin.set_fact:
minio_backup_target_bind_address: >-
{{ minio_backup_target_bind_address
if (minio_backup_target_bind_address | length > 0)
else (ansible_facts.all_ipv4_addresses
| select('match', '^' ~ (minio_backup_target_lan_prefix | regex_escape))
| list | first | default('')) }}
- name: Assert a cluster-reachable LAN address was found
ansible.builtin.assert:
that:
- minio_backup_target_bind_address | length > 0
fail_msg: >-
Could not find a {{ minio_backup_target_lan_prefix }}x address on this host
(interfaces: {{ ansible_facts.all_ipv4_addresses | default([]) | join(', ') }}).
The cluster egresses onto that LAN — refusing to bind MinIO to an unreachable address.
Set minio_backup_target_bind_address explicitly if discovery is wrong.
quiet: true
# ── 1. ZFS dataset (backup drive) — idempotent, never touches other datasets ───────────────────
- name: Create/ensure the k3s-backups ZFS dataset (compression, quota, mountpoint; NO sharenfs)
community.general.zfs:
name: "{{ minio_backup_target_zfs_dataset }}"
state: present
extra_zfs_properties:
compression: "{{ minio_backup_target_zfs_compression }}"
quota: "{{ minio_backup_target_zfs_quota }}"
mountpoint: "{{ minio_backup_target_mountpoint }}"
# ── 2. Service identity ───────────────────────────────────────────────────────────────────────
- name: Create the minio system group
ansible.builtin.group:
name: "{{ minio_backup_target_group }}"
system: true
- name: Create the minio system user (no login shell, home = data dir)
ansible.builtin.user:
name: "{{ minio_backup_target_user }}"
group: "{{ minio_backup_target_group }}"
system: true
shell: /usr/sbin/nologin
home: "{{ minio_backup_target_data_dir }}"
create_home: false
- name: Ensure the MinIO data directory exists (owned by minio, inside the dataset)
ansible.builtin.file:
path: "{{ minio_backup_target_data_dir }}"
state: directory
owner: "{{ minio_backup_target_user }}"
group: "{{ minio_backup_target_group }}"
mode: "0750"
# ── 3. Binaries — fetched on the CONTROLLER (host has no internet), copied to the host ──────────
- name: Ensure controller-side download cache exists
ansible.builtin.file:
path: "{{ minio_backup_target_controller_cache }}"
state: directory
mode: "0700"
delegate_to: localhost
become: false
run_once: true
- name: Fetch MinIO server binary onto the controller (checksummed)
ansible.builtin.get_url:
url: "{{ minio_backup_target_minio_url }}"
dest: "{{ minio_backup_target_controller_cache }}/minio.{{ minio_backup_target_minio_version }}"
checksum: "sha256:{{ minio_backup_target_minio_sha256 }}"
mode: "0755"
delegate_to: localhost
become: false
run_once: true
- name: Fetch mc client binary onto the controller (checksummed)
ansible.builtin.get_url:
url: "{{ minio_backup_target_mc_url }}"
dest: "{{ minio_backup_target_controller_cache }}/mc.{{ minio_backup_target_mc_version }}"
checksum: "sha256:{{ minio_backup_target_mc_sha256 }}"
mode: "0755"
delegate_to: localhost
become: false
run_once: true
# Integrity is already guaranteed by the SHA256 verification get_url performed on the controller
# above; copy transfers that verified artifact faithfully. (The copy module's own `checksum` arg is
# SHA1, not SHA256 — deliberately not used here to avoid a false mismatch.)
- name: Install MinIO server binary on the host
ansible.builtin.copy:
src: "{{ minio_backup_target_controller_cache }}/minio.{{ minio_backup_target_minio_version }}"
dest: "{{ minio_backup_target_minio_bin }}"
owner: root
group: root
mode: "0755"
notify: Restart minio
- name: Install mc client binary on the host
ansible.builtin.copy:
src: "{{ minio_backup_target_controller_cache }}/mc.{{ minio_backup_target_mc_version }}"
dest: "{{ minio_backup_target_mc_bin }}"
owner: root
group: root
mode: "0755"
# ── 4. systemd service ────────────────────────────────────────────────────────────────────────
- name: Write the MinIO environment file (root creds, bind address, data dir)
ansible.builtin.template:
src: minio.env.j2
dest: "{{ minio_backup_target_env_file }}"
owner: root
group: "{{ minio_backup_target_group }}"
mode: "0640"
no_log: true
notify: Restart minio
- name: Install the MinIO systemd unit
ansible.builtin.template:
src: minio.service.j2
dest: /etc/systemd/system/minio.service
owner: root
group: root
mode: "0644"
notify: Restart minio
- name: Enable + start MinIO (apply any pending unit/env changes now)
ansible.builtin.systemd:
name: minio
enabled: true
state: started
daemon_reload: true
- name: Flush handlers so MinIO is (re)started before we configure buckets/users
ansible.builtin.meta: flush_handlers
# ── 5. Readiness gate ─────────────────────────────────────────────────────────────────────────
- name: Wait for the MinIO S3 health endpoint to report ready
ansible.builtin.uri:
url: "http://{{ minio_backup_target_bind_address }}:{{ minio_backup_target_api_port }}/minio/health/ready"
status_code: 200
register: minio_backup_target_health
retries: 30
delay: 2
until: minio_backup_target_health.status == 200
# ── 6. Buckets + scoped non-root user (via mc, idempotent) ─────────────────────────────────────
- name: Configure the local mc alias against the running MinIO
ansible.builtin.command:
argv:
- "{{ minio_backup_target_mc_bin }}"
- alias
- set
- k3sbackup
- "http://{{ minio_backup_target_bind_address }}:{{ minio_backup_target_api_port }}"
- "{{ minio_backup_target_root_user }}"
- "{{ minio_backup_target_root_password }}"
no_log: true
changed_when: false
- name: Create the Velero + CNPG buckets (idempotent)
ansible.builtin.command:
argv:
- "{{ minio_backup_target_mc_bin }}"
- mb
- --ignore-existing
- "k3sbackup/{{ item }}"
loop: "{{ minio_backup_target_buckets }}"
register: minio_backup_target_mb
changed_when: "'created successfully' in (minio_backup_target_mb.stdout | default(''))"
- name: Render the scoped bucket RW policy to the host
ansible.builtin.template:
src: bucket-policy.json.j2
dest: "/etc/minio-{{ minio_backup_target_policy_name }}.json"
owner: root
group: "{{ minio_backup_target_group }}"
mode: "0640"
- name: Check whether the scoped policy already exists
ansible.builtin.command:
argv:
- "{{ minio_backup_target_mc_bin }}"
- admin
- policy
- info
- k3sbackup
- "{{ minio_backup_target_policy_name }}"
register: minio_backup_target_policy_info
changed_when: false
failed_when: false
- name: Create the scoped bucket RW policy (velero + cnpg only)
ansible.builtin.command:
argv:
- "{{ minio_backup_target_mc_bin }}"
- admin
- policy
- create
- k3sbackup
- "{{ minio_backup_target_policy_name }}"
- "/etc/minio-{{ minio_backup_target_policy_name }}.json"
when: minio_backup_target_policy_info.rc != 0
changed_when: true
- name: Check whether the scoped access key already exists
ansible.builtin.command:
argv:
- "{{ minio_backup_target_mc_bin }}"
- admin
- user
- info
- k3sbackup
- "{{ minio_backup_target_access_key }}"
register: minio_backup_target_user_info
changed_when: false
failed_when: false
- name: Create the scoped non-root access key
ansible.builtin.command:
argv:
- "{{ minio_backup_target_mc_bin }}"
- admin
- user
- add
- k3sbackup
- "{{ minio_backup_target_access_key }}"
- "{{ minio_backup_target_secret_key }}"
when: minio_backup_target_user_info.rc != 0
no_log: true
changed_when: true
- name: Attach the scoped policy to the access key (idempotent)
ansible.builtin.command:
argv:
- "{{ minio_backup_target_mc_bin }}"
- admin
- policy
- attach
- k3sbackup
- "{{ minio_backup_target_policy_name }}"
- --user
- "{{ minio_backup_target_access_key }}"
register: minio_backup_target_attach
changed_when:
- minio_backup_target_attach.rc == 0
- "'already in effect' not in (minio_backup_target_attach.stderr | default(''))"
failed_when:
- minio_backup_target_attach.rc != 0
- "'already in effect' not in (minio_backup_target_attach.stderr | default(''))"
@@ -1,15 +0,0 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:*"],
"Resource": [
{% for b in minio_backup_target_buckets %}
"arn:aws:s3:::{{ b }}",
"arn:aws:s3:::{{ b }}/*"{{ "," if not loop.last else "" }}
{% endfor %}
]
}
]
}
@@ -1,11 +0,0 @@
# {{ ansible_managed }}
# MinIO server environment (single-node, single-drive backup target for k3s Velero + CNPG).
# Managed by roles/minio_backup_target — do not edit by hand.
MINIO_ROOT_USER={{ minio_backup_target_root_user }}
MINIO_ROOT_PASSWORD={{ minio_backup_target_root_password }}
# The S3 drive lives on the ZFS dataset {{ minio_backup_target_zfs_dataset }}.
MINIO_VOLUMES="{{ minio_backup_target_data_dir }}"
# Bind the S3 API (and console) to the cluster-reachable LAN address only.
MINIO_OPTS="--address {{ minio_backup_target_bind_address }}:{{ minio_backup_target_api_port }} --console-address {{ minio_backup_target_bind_address }}:{{ minio_backup_target_console_port }}"
@@ -1,28 +0,0 @@
{# {{ ansible_managed }} — MinIO systemd unit (roles/minio_backup_target) #}
[Unit]
Description=MinIO S3 backup target (k3s Velero + CNPG)
Documentation=https://min.io/docs/minio/linux/index.html
After=network-online.target
Wants=network-online.target
AssertFileIsExecutable={{ minio_backup_target_minio_bin }}
[Service]
Type=notify
User={{ minio_backup_target_user }}
Group={{ minio_backup_target_group }}
EnvironmentFile={{ minio_backup_target_env_file }}
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo 'MINIO_VOLUMES unset in {{ minio_backup_target_env_file }}'; exit 1; fi"
ExecStart={{ minio_backup_target_minio_bin }} server $MINIO_OPTS $MINIO_VOLUMES
Restart=always
RestartSec=5s
LimitNOFILE=65536
TimeoutStopSec=30s
# Hardening (backup target holds no other roles on this host).
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
ReadWritePaths={{ minio_backup_target_mountpoint }}
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,66 @@
# roles/s3_backup_target
Stands up a **single-node S3 endpoint** on the **Proxmox host** to serve as the off-cluster
**backup target** for the k3s platform's **Velero** (volume/resource DR, R-STATE-03) and
**CloudNativePG** (Keycloak DB Barman object store, R-IDP-05). A backup target — not primary
storage — so a single dataset is intentional.
Implemented with **[versitygw](https://github.com/versity/versitygw)** (Versity Gateway): a
maintained Go S3 gateway that serves a POSIX filesystem as S3. It **replaces the earlier MinIO
implementation** (MinIO's community edition was de-prioritised — console removed, commercial
pivot). The role also **decommissions any pre-existing MinIO** install on the host, so a re-run is
a clean, idempotent swap. Endpoint, access key, secret and bucket names are kept **byte-identical**
to the previous MinIO target, so nothing downstream (the SOPS secret, Velero, CNPG) changes.
## What it does (all additive + idempotent, surgical)
1. **Decommission MinIO** (tolerant no-op on a fresh host): stop/disable/remove `minio.service`,
the `minio` + `mc` binaries, `/etc/default/minio`, the IAM policy file, the `mc` client config,
the `minio` user/group, and MinIO's on-disk object layout (`.minio`, `.minio.sys`) inside the
gateway-root dir. **The ZFS dataset and mount are KEPT.**
2. **ZFS dataset** `Storage/k3s-backups` — `compression=zstd`, `quota=100G`, **`xattr=sa`**
(required — versitygw stores object metadata in POSIX extended attributes and refuses to start
without them), `mountpoint=/mnt/Storage/k3s-backups`, **no sharenfs**. Uses `community.general.zfs`
so re-runs are no-ops and existing datasets are never touched.
3. **`versitygw` system user/group** + the gateway-root dir (`…/k3s-backups/data`) it owns.
4. **versitygw binary** (pinned, checksummed). The Proxmox host has **no outbound internet**, so the
release tarball is downloaded + verified + extracted on the **Ansible controller** and only the
binary is copied to the host.
5. **systemd unit** `versitygw.service` (Type=simple, hardened), `posix` backend rooted at the
gateway root, bound to the host's **cluster-reachable `192.168.1.x` address** on port **9000**
(same endpoint as the old MinIO) — auto-discovered from facts, assert-guarded. TLS off (plain
HTTP, path-style).
6. **Buckets** `velero` + `cnpg` — on the posix backend a bucket is simply a top-level subdirectory
of the gateway root, so they are pre-created owned by the service user.
## Credentials
versitygw authenticates every S3 request against a single **ROOT** access key/secret
(`ROOT_ACCESS_KEY` / `ROOT_SECRET_KEY`). This role sets it to the **same** scoped key/secret the
cluster already uses, supplied via the vaulted `inventory/group_vars/proxmox_hosts/vault.yml`
(ansible-vault). Nothing is defaulted — a missing vault fails the play loudly:
| var | purpose |
|---|---|
| `s3_backup_target_access_key` / `_secret_key` | versitygw ROOT key = the key handed to Velero + CNPG |
The **same** key/secret is staged for the GitOps side, SOPS-encrypted, at
`platform/backup/minio-credentials.sops.yaml` — the two MUST stay in sync.
> Note: unlike MinIO, versitygw's single-binary posix backend has no separate "root admin" vs
> "scoped user" distinction without enabling an external IAM backend. For a dedicated backup target
> the scoped key **is** the root key — it only ever fronts the velero + cnpg buckets on this host.
## Run
```bash
cd infra/ansible
ansible-galaxy collection install -r collections/requirements.yml # community.general (zfs)
ansible-playbook playbooks/backup-s3.yml
```
## Endpoint
`http://<host-192.168.1.x>:9000` — the S3 URL the cluster uses (path-style). An unauthenticated
`GET /` returns `403 AccessDenied` once serving; a SigV4 `ListBuckets` with the root key returns
`velero` + `cnpg`. Verified reachable from an in-cluster pod at bring-up.
@@ -0,0 +1,94 @@
---
# roles/s3_backup_target — a single-node S3 backup endpoint on the Proxmox host, used ONLY as an
# off-cluster backup bucket store for Velero + CloudNativePG (ADR-0003, R-STATE-03 / R-IDP-05).
# This is a BACKUP TARGET, not primary storage — a single drive/dataset is intentional.
#
# Implemented with versitygw (Versity Gateway, github.com/versity/versitygw): a maintained Go S3
# gateway that serves a POSIX filesystem as S3. It REPLACES the earlier MinIO implementation (MinIO's
# community edition was de-prioritised); the role also decommissions any pre-existing MinIO install
# on the host so a re-run is a clean swap.
#
# SAFETY: every task in this role is additive/idempotent and SURGICAL. It touches ONLY the
# minio→versitygw swap: the ONE ZFS dataset it owns (Storage/k3s-backups), the versitygw system
# user, its systemd service + binary, and the two buckets inside that dataset. It never touches
# other datasets, sharenfs, pvesm storage, VMs or firewall. See tasks/main.yml for the guards.
# ── versitygw release pin (Renovate-friendly) ─────────────────────────────────────────────────
# The Proxmox host has NO outbound internet (PiHole-only DNS, no default route to WAN), so the
# release tarball is fetched + verified on the ANSIBLE CONTROLLER, the binary extracted there, and
# only the binary is copied to the host (tasks/main.yml).
# Release assets: https://github.com/versity/versitygw/releases
# tarball : versitygw_v{ver}_Linux_x86_64.tar.gz (extracts to versitygw_v{ver}_Linux_x86_64/versitygw)
# checksums: checksums.txt (sha256) in the same release
s3_backup_target_versitygw_version: "v1.7.0"
s3_backup_target_versitygw_sha256: "dacec530fecaa8776feec0536998dd4bd78036e0d4db0da9b9189d96bb790e03"
s3_backup_target_versitygw_url: "https://github.com/versity/versitygw/releases/download/{{ s3_backup_target_versitygw_version }}/versitygw_{{ s3_backup_target_versitygw_version }}_Linux_x86_64.tar.gz"
# Path of the binary INSIDE the extracted tarball (GoReleaser lays it out under a version dir).
s3_backup_target_versitygw_archive_member: "versitygw_{{ s3_backup_target_versitygw_version }}_Linux_x86_64/versitygw"
# Controller-side cache the tarball is fetched + extracted into (delegate_to localhost).
s3_backup_target_controller_cache: "/tmp/s3-backup-target-cache"
# ── ZFS dataset (backup drive) ────────────────────────────────────────────────────────────────
# Host convention is Storage/<purpose> mounted at /mnt/Storage/<purpose>. NO sharenfs.
# xattr=sa is REQUIRED: versitygw's posix backend stores object metadata (ETags, etc.) in POSIX
# extended attributes and refuses to start if the root FS can't hold them.
s3_backup_target_zfs_dataset: "Storage/k3s-backups"
s3_backup_target_mountpoint: "/mnt/Storage/k3s-backups"
s3_backup_target_zfs_compression: "zstd"
s3_backup_target_zfs_quota: "100G"
s3_backup_target_zfs_xattr: "sa"
# versitygw's gateway root — a subdir of the dataset, keeping the dataset root owned by root and
# clean. Its top-level subdirectories ARE the S3 buckets.
s3_backup_target_gateway_root: "{{ s3_backup_target_mountpoint }}/data"
# ── Service identity ────────────────────────────────────────────────────────────────────────
s3_backup_target_user: "versitygw"
s3_backup_target_group: "versitygw"
s3_backup_target_bin: "/usr/local/bin/versitygw"
s3_backup_target_env_file: "/etc/default/versitygw"
# ── Network binding ──────────────────────────────────────────────────────────────────────────
# The S3 API must bind the host address the k3s pods can reach (the 192.168.1.0/24 LAN; nodes are
# .50/.51/.52). tasks/main.yml auto-discovers the host's address in this CIDR prefix and asserts it
# exists; override s3_backup_target_bind_address to pin it explicitly. Port 9000 keeps the endpoint
# byte-identical to the previous MinIO install so nothing downstream (SOPS secret, Velero/CNPG)
# changes. (versitygw's own default is :7070 — deliberately overridden here.)
s3_backup_target_lan_prefix: "192.168.1."
s3_backup_target_bind_address: "" # "" ⇒ auto-discover from facts (see tasks)
s3_backup_target_api_port: 9000
s3_backup_target_region: "us-east-1"
# ── Buckets ────────────────────────────────────────────────────────────────────────────────────
# Velero and CNPG each get a bucket. On the posix backend a bucket == a top-level subdirectory of
# the gateway root, so they are simply pre-created (owned by the service user). The single root key
# below has full access to all buckets.
s3_backup_target_buckets:
- velero
- cnpg
# ── Credentials ──────────────────────────────────────────────────────────────────────────────────
# versitygw authenticates every S3 request against a single ROOT access key/secret (ROOT_ACCESS_KEY
# / ROOT_SECRET_KEY). We set it to the SAME scoped key/secret the cluster already uses, so the
# endpoint + access key + secret + buckets stay byte-identical to the previous MinIO target and the
# GitOps side (SOPS: platform/backup/minio-credentials.sops.yaml) needs ZERO change.
#
# Supplied via the vaulted group_vars (inventory/group_vars/proxmox_hosts/vault.yml):
# s3_backup_target_access_key, s3_backup_target_secret_key
# Intentionally NOT defaulted here so a missing vault fails loudly rather than silently
# provisioning guessable creds.
# ── MinIO decommission (of any pre-existing MinIO install this role is replacing) ──────────────
# Paths/identities the OLD minio_backup_target role created, removed by the decommission block.
s3_backup_target_minio_service: "minio.service"
s3_backup_target_minio_bin: "/usr/local/bin/minio"
s3_backup_target_minio_mc_bin: "/usr/local/bin/mc"
s3_backup_target_minio_env_file: "/etc/default/minio"
s3_backup_target_minio_policy_file: "/etc/minio-k3s-backup-rw.json"
s3_backup_target_minio_user: "minio"
s3_backup_target_minio_group: "minio"
# MinIO's on-disk object layout left inside the gateway-root dir (cleared so versitygw starts
# fresh; there are no real backups yet so this is safe). The dataset itself is KEPT.
s3_backup_target_minio_leftovers:
- "{{ s3_backup_target_gateway_root }}/.minio"
- "{{ s3_backup_target_gateway_root }}/.minio.sys"
@@ -1,6 +1,6 @@
---
- name: Restart minio
- name: Restart versitygw
ansible.builtin.systemd:
name: minio
name: versitygw
state: restarted
daemon_reload: true
@@ -0,0 +1,231 @@
---
# ── 0. Preconditions / safety asserts ─────────────────────────────────────────────────────────
- name: Assert S3 backup-target credentials are supplied (from vaulted group_vars)
ansible.builtin.assert:
that:
- s3_backup_target_access_key | default('') | length > 0
- s3_backup_target_secret_key | default('') | length >= 8
fail_msg: >-
versitygw credentials are missing. Provide s3_backup_target_access_key/_secret_key via
inventory/group_vars/proxmox_hosts/vault.yml (they MUST match the SOPS-staged scoped key at
platform/backup/minio-credentials.sops.yaml so nothing downstream changes).
quiet: true
# Determine the host's 192.168.1.0/24 address (the S3 endpoint the cluster reaches). If the
# operator pinned s3_backup_target_bind_address, honour it; else auto-discover and assert.
- name: Discover the host LAN (cluster-reachable) IPv4 address
ansible.builtin.set_fact:
s3_backup_target_bind_address: >-
{{ s3_backup_target_bind_address
if (s3_backup_target_bind_address | length > 0)
else (ansible_facts.all_ipv4_addresses
| select('match', '^' ~ (s3_backup_target_lan_prefix | regex_escape))
| list | first | default('')) }}
- name: Assert a cluster-reachable LAN address was found
ansible.builtin.assert:
that:
- s3_backup_target_bind_address | length > 0
fail_msg: >-
Could not find a {{ s3_backup_target_lan_prefix }}x address on this host
(interfaces: {{ ansible_facts.all_ipv4_addresses | default([]) | join(', ') }}).
The cluster egresses onto that LAN — refusing to bind the S3 endpoint to an unreachable
address. Set s3_backup_target_bind_address explicitly if discovery is wrong.
quiet: true
# ── 1. Decommission any pre-existing MinIO install this role replaces (idempotent + surgical) ───
# All steps tolerate absence, so on a fresh host (no MinIO) they are harmless no-ops. The ZFS
# dataset and mount are KEPT — only MinIO's service/binary/env/mc-client/object-layout go.
- name: Check whether the old MinIO systemd unit is present
ansible.builtin.stat:
path: "/etc/systemd/system/{{ s3_backup_target_minio_service }}"
register: s3_backup_target_minio_unit
- name: Stop + disable the old MinIO service (only if its unit exists)
ansible.builtin.systemd:
name: "{{ s3_backup_target_minio_service }}"
state: stopped
enabled: false
when: s3_backup_target_minio_unit.stat.exists
- name: Remove the old MinIO systemd unit
ansible.builtin.file:
path: "/etc/systemd/system/{{ s3_backup_target_minio_service }}"
state: absent
register: s3_backup_target_minio_unit_removed
- name: Reload systemd if the MinIO unit was removed
ansible.builtin.systemd:
daemon_reload: true
when: s3_backup_target_minio_unit_removed.changed
- name: Remove MinIO server + mc client binaries, env file and IAM policy file
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- "{{ s3_backup_target_minio_bin }}"
- "{{ s3_backup_target_minio_mc_bin }}"
- "{{ s3_backup_target_minio_env_file }}"
- "{{ s3_backup_target_minio_policy_file }}"
- name: Remove the mc client config left in root's home
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop:
- /root/.mc
- /root/.mcli
- name: Clear MinIO's on-disk object layout from the gateway-root dir (dataset itself KEPT)
ansible.builtin.file:
path: "{{ item }}"
state: absent
loop: "{{ s3_backup_target_minio_leftovers }}"
- name: Remove the old minio system user (its files are cleared / re-owned below)
ansible.builtin.user:
name: "{{ s3_backup_target_minio_user }}"
state: absent
remove: false
failed_when: false
- name: Remove the old minio system group
ansible.builtin.group:
name: "{{ s3_backup_target_minio_group }}"
state: absent
failed_when: false
# ── 2. ZFS dataset (backup drive) — idempotent, never touches other datasets ───────────────────
# xattr=sa is required by versitygw's posix backend (object metadata is stored in extended attrs).
- name: Create/ensure the k3s-backups ZFS dataset (compression, quota, xattr=sa, mountpoint; NO sharenfs)
community.general.zfs:
name: "{{ s3_backup_target_zfs_dataset }}"
state: present
extra_zfs_properties:
compression: "{{ s3_backup_target_zfs_compression }}"
quota: "{{ s3_backup_target_zfs_quota }}"
xattr: "{{ s3_backup_target_zfs_xattr }}"
mountpoint: "{{ s3_backup_target_mountpoint }}"
# ── 3. Service identity ───────────────────────────────────────────────────────────────────────
- name: Create the versitygw system group
ansible.builtin.group:
name: "{{ s3_backup_target_group }}"
system: true
- name: Create the versitygw system user (no login shell, home = gateway root)
ansible.builtin.user:
name: "{{ s3_backup_target_user }}"
group: "{{ s3_backup_target_group }}"
system: true
shell: /usr/sbin/nologin
home: "{{ s3_backup_target_gateway_root }}"
create_home: false
- name: Ensure the gateway root dir exists (owned by versitygw, inside the dataset)
ansible.builtin.file:
path: "{{ s3_backup_target_gateway_root }}"
state: directory
owner: "{{ s3_backup_target_user }}"
group: "{{ s3_backup_target_group }}"
mode: "0750"
- name: Pre-create the S3 buckets (a bucket == a top-level subdir of the gateway root)
ansible.builtin.file:
path: "{{ s3_backup_target_gateway_root }}/{{ item }}"
state: directory
owner: "{{ s3_backup_target_user }}"
group: "{{ s3_backup_target_group }}"
mode: "0750"
loop: "{{ s3_backup_target_buckets }}"
# ── 4. Binary — fetched + extracted on the CONTROLLER (host has no internet), copied to the host ─
- name: Ensure controller-side download cache exists
ansible.builtin.file:
path: "{{ s3_backup_target_controller_cache }}"
state: directory
mode: "0700"
delegate_to: localhost
become: false
run_once: true
- name: Fetch the versitygw release tarball onto the controller (checksummed)
ansible.builtin.get_url:
url: "{{ s3_backup_target_versitygw_url }}"
dest: "{{ s3_backup_target_controller_cache }}/versitygw_{{ s3_backup_target_versitygw_version }}.tar.gz"
checksum: "sha256:{{ s3_backup_target_versitygw_sha256 }}"
mode: "0600"
delegate_to: localhost
become: false
run_once: true
- name: Extract the versitygw binary on the controller
ansible.builtin.unarchive:
src: "{{ s3_backup_target_controller_cache }}/versitygw_{{ s3_backup_target_versitygw_version }}.tar.gz"
dest: "{{ s3_backup_target_controller_cache }}"
remote_src: true
creates: "{{ s3_backup_target_controller_cache }}/{{ s3_backup_target_versitygw_archive_member }}"
delegate_to: localhost
become: false
run_once: true
# Integrity is already guaranteed by the SHA256 get_url performed on the controller; copy transfers
# that verified artifact faithfully. (The copy module's own `checksum` arg is SHA1 — not used here.)
- name: Install the versitygw binary on the host
ansible.builtin.copy:
src: "{{ s3_backup_target_controller_cache }}/{{ s3_backup_target_versitygw_archive_member }}"
dest: "{{ s3_backup_target_bin }}"
owner: root
group: root
mode: "0755"
notify: Restart versitygw
# ── 5. systemd service ────────────────────────────────────────────────────────────────────────
- name: Write the versitygw environment file (root access key/secret)
ansible.builtin.template:
src: versitygw.env.j2
dest: "{{ s3_backup_target_env_file }}"
owner: root
group: "{{ s3_backup_target_group }}"
mode: "0640"
no_log: true
notify: Restart versitygw
- name: Install the versitygw systemd unit
ansible.builtin.template:
src: versitygw.service.j2
dest: /etc/systemd/system/versitygw.service
owner: root
group: root
mode: "0644"
notify: Restart versitygw
- name: Enable + start versitygw (apply any pending unit/env changes now)
ansible.builtin.systemd:
name: versitygw
enabled: true
state: started
daemon_reload: true
- name: Flush handlers so versitygw is (re)started before we verify it
ansible.builtin.meta: flush_handlers
# ── 6. Readiness gate ─────────────────────────────────────────────────────────────────────────
# versitygw has no health endpoint; an UNAUTHENTICATED GET / returns 403 (AccessDenied) once the
# S3 server is up and enforcing SigV4 — a reliable "listening + serving" signal.
- name: Wait for the versitygw S3 port to accept connections
ansible.builtin.wait_for:
host: "{{ s3_backup_target_bind_address }}"
port: "{{ s3_backup_target_api_port }}"
timeout: 60
- name: Confirm versitygw is serving S3 (unauthenticated GET / ⇒ 403 AccessDenied)
ansible.builtin.uri:
url: "http://{{ s3_backup_target_bind_address }}:{{ s3_backup_target_api_port }}/"
method: GET
status_code: 403
register: s3_backup_target_probe
retries: 15
delay: 2
until: s3_backup_target_probe.status == 403
@@ -0,0 +1,10 @@
# {{ ansible_managed }}
# versitygw root credentials (single-node posix backup target for k3s Velero + CNPG).
# Managed by roles/s3_backup_target — do not edit by hand.
#
# versitygw authenticates every S3 request against this ROOT key. It is set to the SAME scoped
# access key/secret the cluster already uses (mirrored via SOPS in
# platform/backup/minio-credentials.sops.yaml) so the endpoint + key + secret stay byte-identical
# to the previous MinIO target — nothing downstream changes.
ROOT_ACCESS_KEY={{ s3_backup_target_access_key }}
ROOT_SECRET_KEY={{ s3_backup_target_secret_key }}
@@ -0,0 +1,35 @@
{# {{ ansible_managed }} — versitygw systemd unit (roles/s3_backup_target) #}
[Unit]
Description=versitygw S3 backup target (k3s Velero + CNPG)
Documentation=https://github.com/versity/versitygw
After=network-online.target
Wants=network-online.target
AssertFileIsExecutable={{ s3_backup_target_bin }}
AssertPathIsDirectory={{ s3_backup_target_gateway_root }}
[Service]
Type=simple
User={{ s3_backup_target_user }}
Group={{ s3_backup_target_group }}
# ROOT_ACCESS_KEY / ROOT_SECRET_KEY are read from here (kept out of the process args).
EnvironmentFile={{ s3_backup_target_env_file }}
# Global options (--port/--region) come BEFORE the `posix` backend subcommand; posix's single
# positional arg is the gateway root, whose top-level subdirs are the S3 buckets. TLS off (no
# --cert/--key) ⇒ plain HTTP, path-style.
ExecStart={{ s3_backup_target_bin }} \
--port {{ s3_backup_target_bind_address }}:{{ s3_backup_target_api_port }} \
--region {{ s3_backup_target_region }} \
posix {{ s3_backup_target_gateway_root }}
Restart=always
RestartSec=5s
LimitNOFILE=65536
TimeoutStopSec=30s
# Hardening (backup target holds no other roles on this host).
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths={{ s3_backup_target_mountpoint }}
[Install]
WantedBy=multi-user.target