home-network-k3s-impl/infra/ansible/roles/s3_backup_target

roles/s3_backup_target

Stands up a single-node S3 endpoint on the Proxmox host to serve as the off-cluster backup target for the k3s platform's Velero (volume/resource DR, R-STATE-03) and CloudNativePG (Keycloak DB Barman object store, R-IDP-05). A backup target — not primary storage — so a single dataset is intentional.

Implemented with versitygw (Versity Gateway): a maintained Go S3 gateway that serves a POSIX filesystem as S3. It replaces the earlier MinIO implementation (MinIO's community edition was de-prioritised — console removed, commercial pivot). The role also decommissions any pre-existing MinIO install on the host, so a re-run is a clean, idempotent swap. Endpoint, access key, secret and bucket names are kept byte-identical to the previous MinIO target, so nothing downstream (the SOPS secret, Velero, CNPG) changes.

What it does (all additive + idempotent, surgical)

1. Decommission MinIO (tolerant no-op on a fresh host): stop/disable/remove minio.service, the minio + mc binaries, /etc/default/minio, the IAM policy file, the mc client config, the minio user/group, and MinIO's on-disk object layout (.minio, .minio.sys) inside the gateway-root dir. The ZFS dataset and mount are KEPT.
2. ZFS dataset Storage/k3s-backups — compression=zstd, quota=100G, xattr=sa (required — versitygw stores object metadata in POSIX extended attributes and refuses to start without them), mountpoint=/mnt/Storage/k3s-backups, no sharenfs. Uses community.general.zfs so re-runs are no-ops and existing datasets are never touched.
3. versitygw system user/group + the gateway-root dir (…/k3s-backups/data) it owns.
4. versitygw binary (pinned, checksummed). The Proxmox host has no outbound internet, so the release tarball is downloaded + verified + extracted on the Ansible controller and only the binary is copied to the host.
5. systemd unit versitygw.service (Type=simple, hardened), posix backend rooted at the gateway root, bound to the host's cluster-reachable 192.168.1.x address on port 9000 (same endpoint as the old MinIO) — auto-discovered from facts, assert-guarded. TLS off (plain HTTP, path-style).
6. Buckets velero + cnpg — on the posix backend a bucket is simply a top-level subdirectory of the gateway root, so they are pre-created owned by the service user.

Credentials

versitygw authenticates every S3 request against a single ROOT access key/secret (ROOT_ACCESS_KEY / ROOT_SECRET_KEY). This role sets it to the same scoped key/secret the cluster already uses, supplied via the vaulted inventory/group_vars/proxmox_hosts/vault.yml (ansible-vault). Nothing is defaulted — a missing vault fails the play loudly:

var	purpose
s3_backup_target_access_key / _secret_key	versitygw ROOT key = the key handed to Velero + CNPG

The same key/secret is staged for the GitOps side, SOPS-encrypted, at platform/backup/minio-credentials.sops.yaml — the two MUST stay in sync.

Note: unlike MinIO, versitygw's single-binary posix backend has no separate "root admin" vs "scoped user" distinction without enabling an external IAM backend. For a dedicated backup target the scoped key is the root key — it only ever fronts the velero + cnpg buckets on this host.

Run

cd infra/ansible
ansible-galaxy collection install -r collections/requirements.yml   # community.general (zfs)
ansible-playbook playbooks/backup-s3.yml

Endpoint

http://<host-192.168.1.x>:9000 — the S3 URL the cluster uses (path-style). An unauthenticated GET / returns 403 AccessDenied once serving; a SigV4 ListBuckets with the root key returns velero + cnpg. Verified reachable from an in-cluster pod at bring-up.