# roles/s3_backup_target

Stands up a **single-node S3 endpoint** on the **Proxmox host** to serve as the off-cluster
**backup target** for the k3s platform's **Velero** (volume/resource DR, R-STATE-03) and
**CloudNativePG** (Keycloak DB Barman object store, R-IDP-05). A backup target — not primary
storage — so a single dataset is intentional.

Implemented with **[versitygw](https://github.com/versity/versitygw)** (Versity Gateway): a
maintained Go S3 gateway that serves a POSIX filesystem as S3. It **replaces the earlier MinIO
implementation** (MinIO's community edition was de-prioritised — console removed, commercial
pivot). The role also **decommissions any pre-existing MinIO** install on the host, so a re-run is
a clean, idempotent swap. Endpoint, access key, secret and bucket names are kept **byte-identical**
to the previous MinIO target, so nothing downstream (the SOPS secret, Velero, CNPG) changes.

## What it does (all additive + idempotent, surgical)

1. **Decommission MinIO** (tolerant no-op on a fresh host): stop/disable/remove `minio.service`,
   the `minio` + `mc` binaries, `/etc/default/minio`, the IAM policy file, the `mc` client config,
   the `minio` user/group, and MinIO's on-disk object layout (`.minio`, `.minio.sys`) inside the
   gateway-root dir. **The ZFS dataset and mount are KEPT.**
2. **ZFS dataset** `Storage/k3s-backups` — `compression=zstd`, `quota=100G`, **`xattr=sa`**
   (required — versitygw stores object metadata in POSIX extended attributes and refuses to start
   without them), `mountpoint=/mnt/Storage/k3s-backups`, **no sharenfs**. Uses `community.general.zfs`
   so re-runs are no-ops and existing datasets are never touched.
3. **`versitygw` system user/group** + the gateway-root dir (`…/k3s-backups/data`) it owns.
4. **versitygw binary** (pinned, checksummed). The Proxmox host has **no outbound internet**, so the
   release tarball is downloaded + verified + extracted on the **Ansible controller** and only the
   binary is copied to the host.
5. **systemd unit** `versitygw.service` (Type=simple, hardened), `posix` backend rooted at the
   gateway root, bound to the host's **cluster-reachable `192.168.1.x` address** on port **9000**
   (same endpoint as the old MinIO) — auto-discovered from facts, assert-guarded. TLS off (plain
   HTTP, path-style).
6. **Buckets** `velero` + `cnpg` — on the posix backend a bucket is simply a top-level subdirectory
   of the gateway root, so they are pre-created owned by the service user.

## Credentials

versitygw authenticates every S3 request against a single **ROOT** access key/secret
(`ROOT_ACCESS_KEY` / `ROOT_SECRET_KEY`). This role sets it to the **same** scoped key/secret the
cluster already uses, supplied via the vaulted `inventory/group_vars/proxmox_hosts/vault.yml`
(ansible-vault). Nothing is defaulted — a missing vault fails the play loudly:

| var | purpose |
|---|---|
| `s3_backup_target_access_key` / `_secret_key` | versitygw ROOT key = the key handed to Velero + CNPG |

The **same** key/secret is staged for the GitOps side, SOPS-encrypted, at
`platform/backup/minio-credentials.sops.yaml` — the two MUST stay in sync.

> Note: unlike MinIO, versitygw's single-binary posix backend has no separate "root admin" vs
> "scoped user" distinction without enabling an external IAM backend. For a dedicated backup target
> the scoped key **is** the root key — it only ever fronts the velero + cnpg buckets on this host.

## Run

```bash
cd infra/ansible
ansible-galaxy collection install -r collections/requirements.yml   # community.general (zfs)
ansible-playbook playbooks/backup-s3.yml
```

## Endpoint

`http://<host-192.168.1.x>:9000` — the S3 URL the cluster uses (path-style). An unauthenticated
`GET /` returns `403 AccessDenied` once serving; a SigV4 `ListBuckets` with the root key returns
`velero` + `cnpg`. Verified reachable from an in-cluster pod at bring-up.